Files
agent-estate-wiki/current-state.md
T
Tony0410 0c4ed3350b wiki audit 2026-10-02: 2 new cron failures, cliprox/terra connection auth incident, delivery-failure semantics corrected, profile rewrite, broken wikilinks repaired
- new: incidents/2026-10-01-cliproxy-connection-auth-expired.md (HTTP 401, credential fault, cannot self-recover)
- systems/scheduled-tasks.md: 30 jobs / 27 enabled; failing 3 -> 6; streaks advanced; wiki-tree drift recorded
- systems/messaging-integrations.md: two distinct thread-failure variants (lost vs thread-less fallback)
- systems/current-profile.md: full rewrite from live config (no profiles/default/, smart-route, OmniRoute)
- current-state.md / index.md / log.md: 2026-10-02 entry
- repair: copied 5 vault-only pages referenced by index.md (ignis, nextcloud, 3 incidents) so wikilinks resolve
- include pending stocktake output: skills-index.md 2026-09-27, raw/stocktake/freshness-2026-09-2{6,7}.txt
2026-10-02 00:17:15 +08:00

20 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Current State concept active 2026-07-22 2026-10-02 2026-10-02 2026-10-30 high
system
live-config
live-cron-list
live-hermes-version

Current State

Updated: 2026-10-02

Recent Changes (2026-10-02)

  • Cron fleet re-verified: 30 jobs, 27 enabled / 3 disabled (live ~/.hermes/cron/jobs.json). Failing set 3 → 6: two new failures landed on 2026-10-01 —
    • 06a0a1c8704f (AI Newsletter Curation for Patrick, Thu 10:00) — HTTP 401 … All 1 connection(s) authentication expired — please reconnect in the dashboard on provider custom:Omniroute:20128, model cliprox/terra. Its failure notice reached topic 346; no digest was produced. Not self-recovering — needs the OmniRoute connection re-authenticated. Two other enabled jobs share that pin (1b23c042e17d, 79f9bb41ecf5) and will fail next run.
    • 0e753e9720fd (Facebook Birthday Monthly Scrape, 09:00) — HTTP 400, nvidia/deepseek-ai/deepseek-v4-flash absent from the active NVIDIA catalog, confirming the earlier diagnosis that this is a shared-pin problem (sibling fad5a86979d4 uses the same model).
    • 066bb5191280 (Uber Eats Monthly Spend Report, 02:11) — report generated correctly; delivery failed with configured thread_id 34795 for telegram:1793951355 was not found.
  • Carried-over failures, streaks advanced: dca8482e4f76 403 MODEL_NOT_IN_PLAN → streak 6; 1a8e90ea2271 HTTP 410 (retired qwen3.5:397b) → streak 8; fad5a86979d4 HTTP 400 → streak 3. All three still need a model repin.
  • Telegram delivery semantics corrected — a "failed" delivery is not always a lost message. Against ~/.hermes/cron/executions.db, the 2026-10-01 failures carry the fallback variant: configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id. Content reaches the DM, yet the run is recorded delivery_outcome: failed. The 2026-09-30 incident's Message thread not found variant is the one that truly loses the message. Both variants are now documented in systems/messaging-integrations.
  • The wiki has two trees, and they have drifted in both directions (new finding, 2026-10-02): the canonical source is /home/hermes/workspace/agent-estate-wiki — it holds the Gitea remote (Anthony/agent-estate-wiki), the Quartz build (serve.py, package.json, public/) and is what wiki-maintain.sh commits. /home/hermes/wiki/agent-estate-wiki is the Obsidian/Ignis vault copy (pages nested under wiki/, no git remote) that the iPad reads. The nightly audit is told to write to the vault path, so edits that skip the canonical tree never reach Gitea or the published site. Worse, the trees have diverged backwards too: systems/ignis.md, systems/nextcloud.md and three incidents/ pages exist only in the vault, while raw/stocktake/*, incidents/2026-09-02-sftpgo-outage.md, reports/communities-wa-social-media-audit-2026.md and agents/ exist only in canonical. index.md in canonical links [[systems/ignis]] and [[systems/nextcloud]], both of which are broken there. Any one-way sync will silently delete pages — reconcile before trusting either tree. See systems/scheduled-tasks.
  • Skills inventory: 344 SKILL.md under ~/.hermes/skills/ (live count 2026-10-02), against 341 in the last generated systems/skills-index (2026-09-27). The page cannot refresh while the stocktake job stays broken.

Recent Changes (2026-10-01)

  • Cron fleet re-verified: 30 jobs, 27 enabled / 3 disabled (live ~/.hermes/cron/jobs.json). Failing set 4 → 3: cbfe434a3b5e returned to ok as predicted on 2026-09-30. The residual 1a8e90ea2271 dispatch error also cleared — its current failure is a different, upstream problem (below).
  • Telegram delivery failures root-caused and fixed: the target, not the topic, was wrong — a bare deliver: "telegram" resolves to the home DM (1793951355), where group topic IDs cannot exist, so sends hard-failed and the message was lost. 1b23c042e17d and c8d52904e1d6 retargeted to telegram:-1004321904721:102 and verified. Three jobs still carry bare telegram targets (0e753e9720fd, fad5a86979d4, 066bb5191280). See systems/messaging-integrations.
  • Three jobs pinned to models that no longer exist upstream — cannot self-recover: 1a8e90ea2271 → qwen3.5:397b retired by Ollama 2026-09-25 (HTTP 410, streak 7; daily 03:00 wiki stocktake is consequently not running); dca8482e4f76 → gpt-5.6-terra plan-gated (403 MODEL_NOT_IN_PLAN, streak 5); fad5a86979d4 → nvidia/…/deepseek-v4-flash absent from the live catalog (HTTP 400, streak 3). Repin to the smart-route alias. See systems/model-providers.
  • GitHub Trending Radar — revoked Telegram bot token: c0a9fa314542 builds its digest then fails the push (getMe → 401). Token in /home/hermes/workspace/github-trending-radar/config.json needs replacing from @BotFather.
  • Memory hygiene pass (2026-09-30): 36 gateway background-process notifications had been written into Mnemosyne as [USER] [Anthony Martin] memories — i.e. machine status text surfacing as the user's own words. Rows purged; canonical facts hermes_cron_model_default and the job-radar model corrected. See systems/mnemosyne.
  • Provider stack rebuilt from live config: 5 custom providers (omni-101, Omniroute:20128, litellm, command-code, guacano). The provider key custom:omni no longer exists; two cron jobs (e530c89f2be0, d40085da631c) still name it but have not failed, because the default custom block resolves to the same base URL.
  • Vision routing — request recorded, not applied: Anthony asked on 2026-09-30 that vision calls use the local Ollama at http://anthonys-mac-mini:11434 (gemma4). Live config still points at nous / stepfun/step-3.7-flash:free as at 2026-10-01.

Recent Changes (2026-09-30)

  • Managed-runtime dependency failure — cron fleet down (2026-09-28 → 30), resolved 19:13 AWST: Jobs failed with Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1). Real cause: the uv-managed runtime venvs (10 under ~/.hermes/installs/019d0d114d22b7e6/environments/*/venv) were missing dotenv and mnemosyne, so the worker died on import before acknowledging the job (ModuleNotFoundError: No module named 'dotenv' — 459 occurrences in worker stderr). Mnemosyne Watchdog failed 176 consecutive runs. Fixed by hermes-runtime-deps-guard.sh v2 (checks/repairs all managed venvs via pip install --target), run every 15 min by job 96eb95ce12f2. weekly-save-restricted-bot-restart script restored 19:17; that job is ok again. See incidents/2026-09-28-hermes-managed-runtime-deps.
  • Hermes upgraded since last audit: v0.19.0 (2026.7.20) → v0.21.5+3837.gbfda74c (2026.9.24), upstream 27062c34, update available. Skills 330 → 341.
  • Provider routing drift: default provider is custom → http://100.78.63.20:20128 (OmniRoute) with default model smart-route; auxiliary curator openai-codex / gpt-5.6-terra. Verified live 2026-09-30.
  • Open — Daily Marketing Job Radar blocked: job dca8482e4f76 pinned to custom:omni-101 / gpt-5.6-terra; upstream command-code returns 403 MODEL_NOT_IN_PLAN. 5 consecutive failures (2026-09-26 → 30). See runbooks/job-radar.
  • Open — Telegram thread delivery failures (root cause found 2026-10-01): telegram:1793951355 thread 34795 "not found" for jobs fad5a86979d4 and 1b23c042e17d; c8d52904e1d6 reports "Message thread not found". Cause was the delivery target, not the topic — a bare telegram resolves to the home DM. Fixed for two of the three jobs on 2026-09-30; see the 2026-10-01 entry above.
  • Wiki tree structure confirmed (drift unresolved): /home/hermes/workspace/agent-estate-wiki is the Gitea/Quartz source (this repo); /home/hermes/wiki/agent-estate-wiki is the Obsidian/Ignis vault (pages under wiki/, no remote). Content is mirrored by hand — the nightly stocktake only maintains this tree.
  • Multi-bot group chat question raised: Anthony asked whether other estate Hermes instances can share a group chat. Preferred shape is a Telegram supergroup with one forum topic per bot (Mattermost rejected). Design question only — nothing deployed.

Recent Changes (2026-09-02)

  • SFTPGo outage resolved (02:18 AWST): Dashboard flagged "service down". CT299 swap fully exhausted (43Gi used, 100%); CT299's tailscaled had died, breaking tailnet reachability; PVE's pct start then failed because 5 stale lxc-attach probes from earlier diagnostics pinned CT299's cgroup (/sys/fs/cgroup/lxc/299/ns/.lxc). Killed those PIDs (3037951/3334928/3352203/3402511/4004148), pct stop 299 --skiplock + pct start 299, fresh PID 3471141, tailscaled reconnected direct. Verified end-to-end: tailscale ping sftpgo → direct 192.168.178.92:41641, SFTPGo WebAdmin HTTP 401, SFTP banner SSH-2.0-SFTPGo_2.7.0. See incidents/2026-09-02-sftpgo-outage.

Attested facts

  • Config version: _config_version: 46 — Verified, ~/.hermes/config.yaml, 2026-09-30 (previously recorded as 33 on 2026-07-25)
  • Default profile: default — Verified, WebUI prompt, 2026-07-22
  • Timezone: Australia/Perth — User-provided + Verified in config, 2026-07-22
  • Main provider: custom → http://100.78.63.20:20128 (OmniRoute) — Verified, ~/.hermes/config.yaml, 2026-09-30 (previously custom:litellm → litellm:4000, then custom:omni → omni.kangaroo-eel.ts.net)
  • Default model: smart-route — Verified, ~/.hermes/config.yaml, 2026-09-30 (previously freellmapi-auto)
  • Fallback providers: opencode-go / mimo-v2.5, opencode-zen / deepseek-v4-flash-free, nous / stepfun/step-3.7-flash:free, ollama-cloud / qwen3.5:397b, kilocode / poolside/laguna-m.1:free, custom / litellm:4000 — Verified, config, 2026-09-30
  • Auxiliary curator model: openai-codex / gpt-5.6-terra — Verified, config, 2026-09-30
  • Memory backend: Mnemosyne — Verified, config, 2026-07-22
  • Memory enabled: true — Verified, config, 2026-07-22
  • Streaming enabled: true — Verified, config, 2026-07-22
  • Dashboard public URL: https://hermes.martinwa.org/ — Verified, config, 2026-07-22
  • Pet config: enabled/slug null — Verified, config, 2026-07-22
  • WebUI connected platforms: local, api_server, discord, email, homeassistant, mattermost, ntfy, slack, telegram, webhook — Verified, WebUI session, 2026-07-22

Recent Changes (2026-08-02)

  • Browser backend reverted: Camoufox restored as primary on 2026-07-30. Steel Browser (CT201) experiment ended — CDP issues + DeepSeek 0731 tool name validation. browser.cloud_provider: camofox, camera_provider: camofox. Steel plugin disabled. See comparisons/steel-vs-camoufox and decisions/rollback-steel-to-camofox.
  • Mnemosyne capture fix (2026-08-01): Automatic conversational capture was broken by stringified YAML (sync_roles: "['user']"). Fixed by rewriting structured Mnemosyne settings with native YAML types. See incidents/2026-08-01-mnemosyne-capture-failure.
  • Mnemosyne vector coverage gap discovered: Only 159/1,476 working memories (10.8%) have vector embeddings. 0/458 episodic memories have embeddings. Semantic search is mostly FTS5 fallback.
  • DeepSeek V4 Flash 0731: Post-training bump, same model ID, same pricing. Terminal Bench 2.1: 82.7 (up from 61.8). Stricter tool name validation (^[a-zA-Z0-9_-]+$) — may cause issues with OmniRoute-transformed tool names.
  • Uber Eats report upgraded: Now uses Camofox authenticated session (not stale cookies.txt). HTML + wiki report generated. Cron job updated with camofox-browser-automation + browser-backend-verification skills.
  • GitHub Trending Radar — Daily Digest removed: Stopped by user on 2026-08-02. Weekly roundup still active.
  • Ops profile renamed: Tembo → Tusk. All SOUL.md, skill references, and memory entries updated.
  • Wiki audit cron job created: Runs nightly at 23:59 AWST — reviews past 24h of conversation for wiki changes.
  • Ops profile Mnemosyne enabled: Mnemosyne plugin enabled on ops profile with profile_isolation: true — Tusk gets its own memory bank, separate from default (Rhino). Zero cross-contamination. Verified end-to-end 2026-08-02. Config: ~/.hermes/profiles/ops/config.yaml (memory.mnemosyne.profile_isolation: true, allow_tool_override: false).
  • OmniRoute free-DeepSeek silent-stop mitigation (2026-08-04): Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances. Reworked both combos: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort. See incidents/2026-08-04-omniroute-free-deepseek-silent-stops.
  • OmniRoute three-layer fix (2026-08-07): npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall documented. See systems/omniroute.
  • Nanobot status degraded (2026-08-07): CT333 nanobot confirmed DOWN — connection refused on 8900. OmniRoute provider connection disabled. See systems/nanobot.
  • CT460 memory bumped (2026-08-07): 5096MB → 14GB RAM. Was causing swap thrashing at load 8+.

Recent Changes (2026-08-08)

  • Cron job model repinning: 3 jobs repinned from dead freeapi/minimax-m2.7 → smart-route (IDs: c8d52904e1d6, 1b23c042e17d, d40085da631c). Mosquito Tracker repinned from dead freeapi/auto → smart-route (ID: 1b31187397a5). 9 stale-named cron jobs (e.g. vertex/gemini-2.5-flash, gpt-5.6-terra, deepseek-v4-flash-free) repinned to smart-route. Daily Marketing Job Radar swapped to gpt-5.6-terra.
  • Hermes UI vite config revert: Vite config had been changed to HTTPS with self-signed certs (~16:00 Aug 8), breaking iPad access (plain HTTP → HTTPS-only without trusted cert). Reverted to pre-HTTPS config, restarted hermes-ui.service. Production build now serves /ui/ path correctly with relative asset paths. hermes-desktop.martinwa.org identified as separate Cloudflare infrastructure (likely on CT450), not fixable from CT460.
  • Phase One estate review completed: Manual cron run verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with last_status=ok, and CT333 Nanobot has a live jobs.json with 9 jobs. The cron review report contained two inaccuracies (wrong date, false claim about CT333 having no scheduler) — verified via live system probes instead.

Recent Changes (2026-08-09)

  • Portainer password reset (CT245): Admin user is anthony (not admin). Password was forgotten; reset via docker run --rm -v portainer_data:/data portainer/helper-reset-password after stopping the Portainer container. Verified end-to-end with POST /api/auth → HTTP 200 + valid JWT. Portainer now uses Tailscale/Let's Encrypt cert on :9443 (see systems/homepage-dashboard). Recovery procedure: stop container → run helper image → restart → verify with auth API.
  • SFTPGo security review (CT299): Anthony shared SFTPGo WebAdmin PDF settings for audit. Key findings: (1) remote SFTP root path // resolves to / — full filesystem access, (2) host fingerprints empty (low-risk on Tailscale but should be pinned for hygiene), (3) remote user is root, (4) quota value F1 on Cloudflare-R2 virtual folder is non-standard (may be display artifact). No action taken — flagged for Anthony's review.

Recent Changes (2026-08-10)

  • Cron job failures — session storage issue: Four cron jobs failed on 2026-08-10 with two distinct error patterns:
    • TimeoutError (idle for 601–602s): "Daily Marketing Job Radar" (dca8482e4f76) at 07:12, "Political News Digest (Mon/Thu)" (0e519874d280) at 08:12. Jobs stuck initializing, exceeded 600s idle limit.
    • RuntimeError (session storage): "Tech & AI Newsletter Digest (Analytical)" (85edf6772bca) at 07:12, "GitHub Trending Radar — Weekly Roundup" (c0a9fa314542) at 09:03. Error: "session storage could not be written (the transcript would have been lost on restart). This is often a full disk — free some space (or fix state.db permissions)."
    • Root cause: Disk at 83% (11GB free on 65GB). state.db is 593MB. Not critically full but may be triggering write failures. Sessions.json shows only 1 active session in past 24h — session DB likely needs cleanup or WAL checkpoint.
    • Impact: News digests and GitHub Trending did not deliver. Job Radar did not run.
    • See: systems/scheduled-tasks Known Issues section.
  • OpenCode WebSocket proxy errors: Repeated "OpenCode service unavailable" errors in hermes node journal (Aug 11 00:08). Process running (PID 809564, opencode serve), but WebSocket proxy failing to connect. Likely transient — service was available earlier in the day.

Recent Changes (2026-08-14)

  • OmniRoute factory reset on CT601: Database wiped clean (0 combos, 0 providers, 0 API keys, 115 built-in auto entries). Backup at /opt/omniroute/factory-reset-backup-20260814-225524. Providers re-added from Notion API vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter (all passed live tests). OpenAI keys tested but invalid — removed. See systems/omniroute.
  • Hermes default model changed: auto/best-free → resilient-free in ~/.hermes/config.yaml. auto/best-free was misconfigured (no combo definition, routed to paid gpt-5.6-sol).
  • Multiple free combos broken by upstream model ID changes: OpenRouter Poolside laguna-s-2-1:free → laguna-s-2.1:free (dash→dot); Groq llama-4-scout-17b-16e-instruct retired. Affected: resilient-free, free-IA, free-stack, good-free, deepseek-flash-free, mimo-free.
  • Guanaco router updated: CT205 ollama container — Guanaco v0.8.11 → v0.8.14. DSML tag leakage fixes (tool-call formatting no longer leaks raw syntax), dashboard performance fix. See systems/guanaco.
  • GPT-5.6 Luna research: Community consensus — Luna cleaner/polished; DeepSeek V4 Flash more persistent/cheaper. Provider route matters more than model name. Luna recommended for planning/review, DeepSeek for execution. See systems/model-providers.
  • Hermes UI restart: Port 8787 service restarted, now properly binding 0.0.0.0 for external access. Confirmed reachable from Tailscale.
  • LiteLLM routing issue identified: Hermes was previously pointed at LiteLLM (http://litellm:4000/v1, model free-auto) causing 6–26s per call. OmniRoute auto/best-free tested at 265ms. Config now points at OmniRoute.

Recent Changes (2026-08-13)

  • OmniRoute migrated CT600 → CT601: Provider renamed custom:omniroute → custom:omni; base URL http://omniroute:20128/v1 → http://omni:20128/v1. Host omni.kangaroo-eel.ts.net (100.93.204.4, active) replaces omniroute (100.88.81.19, offline). CT600 stopped, CT601 running. DB path /var/lib/omniroute/storage.sqlite → /opt/omniroute/storage.sqlite; API key unchanged. All config (14) + cron-job (22) custom:omniroute refs repointed to custom:omni. Backup: ~/.hermes/config.yaml.bak.omniroute-removal. See systems/omniroute.
  • Evening stand-down briefing weather bugfix: Weather collector returns a plain string (+16°C Sunny), but the three dashboard widgets (evening.html, evening-grid.html, evening-briefing.html) read object fields → undefined. Added string-vs-object guards; verified served HTML no longer contains broken raw field refs.