Files
agent-estate-wiki/index.md
T

53 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Anthony's Systems Wiki collection active 2026-07-22 2026-11-06 2026-11-06 high
index
infrastructure
systems
stream-failure
ct299
skills-index
generator-divergence
secrets-at-rest
incident
live-cron-ledger
live-deliveries-db
live-incident-ledger
live-session-store
raw/stocktake

Anthony's Systems Wiki

Operational knowledge for Anthony's agents, infrastructure, model routing and automation. Last updated: 2026-11-06.

2026-11-05 (stocktake) - the systems/skills-index.md flip-flop is fixed at the root: two different scripts were writing the page, and the nightly audit was pointed at the stale one; it is now a delegating wrapper -> incidents/2026-11-05-skills-index-generator-divergence. Docker inventory corrected against live probes on all five hosts - the worldmonitor-seeder-scheduled container recorded yesterday does not exist on any host. Tailscale node table re-verified programmatically - 52 rows, 52 live peers, every state matching. Freshness 0 stale / 122 fresh / 3 no-date of 125.)

2026-11-06 (nightly audit - window 2026-11-05) - a clean ledger day and an empty incident ledger: 2026-11-05 settled 175 rows - 173 completed / 2 running / 0 failed across 18 jobs, the first zero-failed day since 2026-11-02. The only two running rows are the Mnemosyne Health Monitor (12:00:45 slot, still in flight 12 h+ at audit time) and this audit itself. Unclosed incident rows 3 -> 0 - the Newsletter Digest row closed at 07:07:42 and both Job Radar - Discovery rows at 12:36:17, when dca8482e4f76 finally completed after 5 h 35 m 35 s; that closes out incidents/2026-11-04-upstream-stream-failure-0700-cohort and incidents/2026-11-03-job-radar-stream-active-timeout in full. Fleet non-clean last_status 3 -> 1 (only 066bb5191280, delivery_failed, streak 0); deliveries 10 rows / 10 delivered; the 03:00 stocktake overran 3 h 01 m 58 s (down from 7 h 40 m and 5 h 41 m) and committed 2f3fc99 at 05:55. Fleet unchanged at 29 jobs (26 / 3) - a fourth audit at the same count. ⚠ New and open: live .env values (TELEGRAM_BOT_TOKEN, MATTERMOST_TOKEN, HERMES_CUSTOM_OMNIROUTE_20128_API_KEY) sit verbatim in 17 rows of mnemosyne.db (memories, episodic_memory, working_memory, gists, memoria_facts.context_snippet and the FTS mirrors) - reproduced in full by each of the 7 uncompressed rotation snapshots - and all four values sit verbatim in the 1.7 GB session store state.db too; CAMOFOX_API_KEY sits verbatim in the prompts of two enabled Facebook Birthday jobs (0e753e9720fd, fad5a86979d4) and therefore in cron/deliveries.db -> incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.

2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze: the 11-04 fleet settled 170 rows - 167 completed / 2 failed / 1 running, non-clean last_status 2 -> 3, unclosed incident rows 1 -> 3, deliveries 8 rows / 8 delivered. Both failures are the same upstream fault: dca8482e4f76 (Daily Marketing Job Radar - Discovery) and 85edf6772bca (Tech & AI Newsletter Digest (Analytical)) died 36 seconds apart at 09:41-09:42, each after 2 h 41 m, on RuntimeError: Upstream stream failed before completion. - both pinned custom:omni-101 / smart-route. The day was not lost: the 11:30 Stage 2 slot reconstructed the missing handoff from Stage 1's own candidate files, re-verified every candidate live, and still wrote 5 roles (board 479 -> 484) + 1 canonical cover letter (MISSING_COVER_LETTER_COUNT 0) before consuming the handoff at 16:08:59 -> incidents/2026-11-04-upstream-stream-failure-0700-cohort. The cover-letter route survived only on an override: regenerate_cover_letter.py defaults to deepseek-paid-smart, which threw HTTP 402 mid-run, so the letter was pinned to gemini/gemini-3.5-flash via CL_MODEL - and that override was never written to .env, so it did not persist. And CT299 was frozen twice, not once: a second, manual vzdump 299 ... --storage ssd0 wedged from 11-04 08:30:04 until 11-05 ~00:30 (~16 h); the live probe at 00:54:21 AWST shows LOCK: 0, both cgroup.freeze flags 0, guest ALIVE, sftpgo: active -> incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage. ⚠ Corrected the same night: the first draft of this entry read the fleet as 158 jobs (144 / 14) and the 11-04 ledger as closing 1 item_suppressed; live reads give 29 jobs (26 / 3) and a final row of 1 running (this audit). Keyed probe 2026-11-05 01:33: deepseek-paid-smart is undispatchable on omniroute:20128 (HTTP 503, all targets skipped) but serves normally on omniroute-1 (HTTP 200 -> deepseek/deepseek-v4-flash).

2026-11-04 (nightly audit) — the ledger raised its own red row: the **11-03 fleet settled 170 completed / 1 failed / 1 running / 0 unknown (172 rows) — unclosed incident rows 0 → 1, non-clean last_status 1 → 2, deliveries 7 rows / 7 delivered. The single failure is the Job Radar Stage 1 (dca8482e4f76), which ran 07:00:33 → 13:28:52 (6 h 28 m 18 s) and died on RuntimeError: stream active timeout — a new terminal signature, not the iteration cap — after going silent for 65 m 21 s; its incident dca848_dfd4bea84f2a (timeout) is alerted and open and the notice was delivered (13:28:51.741724) → incidents/2026-11-03-job-radar-stream-active-timeout. That answers the 11-02 page's open question: output is suppressed when the run ends completed, delivered when it ends failed — which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so 2026-11-02's and 2026-11-03's candidate sets are both unrecovered. ⚠ The 03:00 stocktake overran for a second day (1a8e90ea2271, 03:00:30 → 10:41:13 = 7 h 40 m 43 s, suppressed, nothing delivered, artefacts uncommitted again). New page: systems/hermes-hq-push (v0.9.27, enabled; manifest says it replaces dispatch-push, and no trace of the old plugin survives).

⚠ Two Proxmox hosts exist — always qualify a guest with its host. A second unclustered hypervisor, m93p (100.104.51.98, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. Guest IDs collide between the two hosts.

Start here

Systems

Core

Integrations

  • systems/conduit — Conduit: Hermes on Anthony's phone (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree ~/.hermes/plugins/conduit_push, v0.122.2 (HEAD 9eac857, 2026-10-31 12:04 — "Every profile that turns Conduit on runs the main install (#482)"; +148 commits on since the 0.69.2 / 942ddb8 baseline, working tree clean), upstream kaishi00/hermes-conduit-notifier. Gateway on :9119 is now pid 176771, started 2026-10-31 19:35:02 (superseded: restarted 2026-10-25 18:42:33, which is when the upstream commits went live). One registered device (iPhone app 2.2.0). ✅ The first two real calls in the product's history are now on record — 2026-10-29 08:38:04 and 08:49:36 — after three faults in series were repaired (truncated speech-server URL → …/v1/realtime; 12 h dashboard session TTL → 30 d; dead groq STT → local, dead openai TTS → ollama_local) → incidents/2026-10-29-conduit-voice-silence. So the memory fix is no longer evidence-free. ✅ That patch is gone — the tree is clean and MEMORY_TIMEOUT_S ships upstream in dashboard/api/memory.py. ⚠ New fault: "local" voice mode never applies — the Mac mini speech server's LLM proxy is disabled (LLM endpoint → 501, /v1/usage reads llm_proxy.requests: 0, realtime socket rejects session.update), while the app's "test connection" checks socket reachability only → incidents/2026-10-31-conduit-voice-local-mode-not-applying. The call store was re-shaped: history is empty and last_request_at is null, so the two 10-29 calls are no longer in it. The 10-24 refusal fault has not recurred → incidents/2026-10-24-conduit-call-request-refused.
  • systems/messaging-integrations — Telegram, Discord, email, Slack, etc. Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26 (peer offline 36 d, tx 312 / rx 0; host answers neither :8065 nor ICMP). 268 connect timeouts on 10-25 vs 264 on 10-24; the reconnect attempt counter reset to 54 at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost.
  • systems/push-notifications — Cadu/Vory push bridge (hermes-push, vory-push): gateway on :9119, state in ~/.hermes/push/. Broken — connected: false (the bridge dials a name that resolves to 127.0.1.1 where nothing serves 443); chronic, not a blip.
  • systems/hermes-hq-push — Hermes HQ iPhone app plugin (standalone, v0.9.27, kind: standalone, author Hermes HQ; tree mtime 2026-11-03 11:06). hermes plugins list → enabled (state user); config.yaml plugins.enabled line 7556. Host side of the Hermes HQ app: Apple push (finished turns, lock-screen approvals, "waiting for a password/login/code" alerts, live Lock Screen cards sealed to the phone), phone device access (contacts, location, reminders, alarms, photos, Health, motion, music), the desktop-app Hermes HQ theme, the app's Terminal page, fast Sessions list on a busy host, cross-posting answers back to Telegram/Discord, a restart guard, and Grok Live voice-token minting (XAI_API_KEY or subscription sign-in; bearer never reaches the phone). Tools device_contacts_search / device_request_result; 7 hooks. Its manifest states it replaces dispatch-push — and the replacement is clean: no dispatch-push reference survives in config.yaml and no such directory remains. Different stack from the Cadu/Vory bridge on systems/push-notifications. ⚠ Its pre_tool_call guard also blocks any tool call whose arguments contain the literal text of a launchd one-off-restart command — describe it, never quote it.
  • systems/scheduled-tasks — As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean last_status 4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), and ea79a2107e36 is ENABLED again (last_status: ok, streak 0) after its 00:01:08 slot closed completed at 10:02:48; deliveries 8/8 delivered. Prior window — as of 2026-10-30 03:3x: 29 jobs (25 enabled, 4 disabled); non-clean last_status 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all delivered. ⚠ ea79a2107e36 (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed unknown on a gateway-connection failure — and its failure notice was NEVER DELIVERED (deliveries.db has zero rows for 2026-10-30), so the incident row is the only trace → incidents/2026-10-30-mnemosyne-health-monitor-paused. Every other previously-red job has cleared to ok/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean last_status 5 → 3 (all three are stale re-runs of old faults and every failure_streak read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all delivered. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with missing_cover_letter_count 0 (Notion 455 → 475). Both stage-1 incident rows closed resolved at 10:16:12; the lone survivor is the Facebook monthly's 0e753e_e7f8194a7d06.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** dca8482e4f76 ran 07:01:08 → 11:03:54 (4 h 2 m 46 s), closed failed, and delivered "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery finished, no Notion writes and no cover letters. Incident row dca848_e93a538ade6b opened 11:03:53 and is still open. The same day the single radar job was split into two: stage 1 dca8482e4f76 (Discovery, 07:00, deliver: local) + new stage 2 5fc292ff0f3b (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two completed test fires that evening). The two incident rows the 2026-10-26 planned restart raised auto-closed 00:38:32, and the 2026-10-26 orphan class did not recur. → incidents/2026-10-27-job-radar-budget-exhausted, runbooks/job-radar. Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean last_status, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault. ea79a2107e36 (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a planned 14:33:35 SIGTERM restart that timed the drain out after 1.35 s, killed after 2 h 34 m on a new string (lost its durable fire claim ownership), and raised two cron_incidents rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → incidents/2026-10-26-planned-restart-orphaned-cron-run. New fleet precedent: a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). Unclosed incident rows 2 → 3; one closed: c0a9fa_764558b5ae83 auto-closed 09:13:00 when the radar's 09:00 slot ran clean, so every row of the 2026-10-16 503 resource-pressure incident is now closed → incidents/2026-10-16-omniroute-503-resource-pressure. Non-clean last_status 5 → 4: c0a9fa314542 and 08eb86f02d37 cleared to ok; ea79a2107e36 joins as error, streak 2. Pins unchanged. ⚠ fad5a86979d4's 03:08 slot completed but delivered delivery_failed — Thread 34795 not found (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered “RUN INCOMPLETE — tool-call budget exhausted” and its cover-letter pipeline changed that morning → runbooks/job-radar. Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean last_status, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0. ⚠ Both 10-25 failures are 08eb86f02d37 (Gmail Action Digest), which also fired six times that day — and neither raised an incident row → incidents/2026-10-25-gmail-digest-runaway-and-refire. ⚠ The global default reverted to custom:omni-101 / smart-route on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again → systems/current-profile. ⚠ Nous Portal credentials still invalid — all 8 auxiliary slots moved to no-think/freeapi/auto at 19:13 → incidents/2026-10-22-nous-portal-refresh-token-invalid. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean last_status, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third consecutive clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0. Zero incident rows opened or closed in-window; pins unchanged. ⚠ Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls → incidents/2026-10-24-conduit-call-request-refused. Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean last_status, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four. The 2026-10-09 provider timeout storm is now fully closed: 1b23c042e17d ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so all 12 are closed and unclosed incident rows fell 4 → 2. ⚠ New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (CONDUIT_MEMORY_TIMEOUT_S) + degraded-not-aborted on TokenError; dashboard restarted 18:54:43, no abort since. Patch uncommitted → incidents/2026-10-22-conduit-memory-start-timeout, systems/conduit. The 10-22 stocktake overrun did not repeat (10-23 03:07 committed 250c735). Prior window: As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean last_status, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen. dca8482e4f76 (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s → incidents/2026-10-21-omniroute-pre-dispatch-filter-skip. 08eb86f02d37 cleared (08:30 slot clean; incident resolved 08:37:21) and 2026-10-20 settled at 169 / 0 + 1 unknown. ⚠ Hermes Agent was updated to 0.22.3 at 22:47 → 22:57 with its pre-update snapshot skipped → systems/hermes-agent. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The omni-101 503 fault did not recur (back to monitoring) and 4 incident rows closed → 9 unclosed. ⚠ Two new findings: the Gitea config backup has not pushed since 2026-08-08 (75 unpushed commits) while its row reads ok → incidents/2026-08-08-gitea-config-backup-push-dead; and a gateway unclean exit on 2026-10-20 orphaned a digest run — no delivery row, no incident → incidents/2026-10-20-gateway-unclean-exit-orphaned-run. Also: the global model default moved to nous direct (meituan/longcat-2.5-preview:free) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while 20 of 28 still pin it → systems/current-profile. Prior window: 28 cron jobs (25 enabled, 3 disabled); 12 carry a non-clean last_status as of 2026-10-20 00:09 — up from 7, because the omniroute-1 503 resource-pressure fault returned on 2026-10-19 (second burst in four days: 9 model calls across 8 jobs, 06:01 and 07:01 → 12:01, all model: smart-route on custom:omni-101, while /api/health stayed 200 ok) → incidents/2026-10-16-omniroute-503-resource-pressure. 2026-10-19 closed 159 / 13 against 10-18's 168 / 1. The Mosquito Tracker was removed (29 → 28 jobs), and the 2026-10-18 degenerate-stream placeholder-delivery mode remains an open question → incidents/2026-10-18-degenerate-stream-canned-substitute. Earlier: the connection-refused storm (2026-10-14 → 10-15) — 24 model-call failures across 12 jobs, ~34 h of digests blanked. All 18 model-calling jobs still sit on custom:omni-101 with no failover.
  • systems/skills-index — Installed skills inventory (302 skills (68 categories) as of the 2026-11-05 stocktake; regenerated daily by the 03:00 job). The generator excludes .archive/ (46 archived skills) — the earlier 346/347 counts included it. Freshness at the 2026-11-05 run: 0 stale / 122 fresh / 3 no-date of 125 pages (freshness-20261105.txt). Note: SKILL.md files on disk still exceed the index by one (303 vs 302) — the generator de-dups one nested duplicate, so 302 is the correct unique-skill figure; see systems/scheduled-tasks. ✅ 2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root: two different scripts were writing this page (the repo script → 302/68; the stale ~/.hermes/scripts/wiki-maintain.sh copy → 349/26, because it used Path.rglob() and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a delegating wrapper with no generator logic of its own → incidents/2026-11-05-skills-index-generator-divergence. ⚠ The 2026-10-22 slot ran 97 m 58 s (vs 16 m 01 s on 10-21), finished completed with delivery suppressed and committed nothing; the 10-23 slot closed that hole.

Incidents

  • incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts - OPEN. Live .env values were found stored verbatim in 17 rows of mnemosyne.db (memories 4, episodic_memory 3, working_memory 1, gists 4, memoria_facts.context_snippet 1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the 7 uncompressed rotation snapshots, and all four values also sit verbatim in the 1.7 GB session store state.db, which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two enabled Facebook Birthday jobs (0e753e9720fd, fad5a86979d4) and hence in cron/deliveries.db. Nothing rotated, deleted or scrubbed; no value recorded.

  • incidents/2026-11-04-upstream-stream-failure-0700-cohort - dca8482e4f76 + 85edf6772bca died 36 s apart on RuntimeError: Upstream stream failed before completion. after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter).

  • incidents/2026-11-05-skills-index-generator-divergence — RESOLVED — the skills index flip-flop had two writers. The repo wiki-maintain.sh (302 skills / 68 categories) and the stale ~/.hermes/scripts/wiki-maintain.sh copy (349 / 26, because it used Path.rglob() and counted the parked .archive/) were different scripts, and SKILL.md pointed the 23:59 nightly audit at the stale one — so each job overwrote the other's page and the wiki oscillated nightly with no error raised by either. The stale copy is now a delegating wrapper; the repo script gained --no-build so the wrapper honours the old interface without a second generator.

  • incidents/2026-10-31-conduit-voice-local-mode-not-applying — OPEN — the app's "local" voice mode never applies, and the connection test lies. The test checks socket reachability only, while every local turn is still impossible: the Mac mini speech server's LLM proxy is disabled (LLM endpoint → HTTP 501 "Start the server with --enable_llm_proxy"), /v1/usage reads llm_proxy.requests: 0, and its realtime socket accepts the connection but rejects session.update as an unknown event. voice.mode is a phone-side selector — no mode key exists in config.yaml → voice or conduit-settings.json — so a "mode did not apply" cannot be diagnosed on host config. Found via session 20261031_193709_40b9c2 (19:37:09, 30 msgs / 17 tool calls); voice.local_live.front_model was repointed to mistral/voxtral-small-latest during it and nothing was changed on the Mac. → systems/conduit

  • incidents/2026-10-31-gateway-reload-force-drain-orphaned-run — A gateway reload waited its full 1,800 s drain cap, then force-drained and orphaned a live cron run. At 18:06:41 the gateway took the reload; the drain reported timed_out=True at the cap and force-drained at 18:37:38 (new gateway 18:38:43). ea79a2107e36 (Mnemosyne Health Monitor, 12:01:06 slot, 6 h 38 m 09 s in) closed failed at 18:39:15 on "lost its durable fire claim ownership". Same class as incidents/2026-10-26-planned-restart-orphaned-cron-run — second instance, and the first on a plain reload. A second, clean restart followed at 19:23:39 → 19:24:40; 2026-10-31 closed 160 / 4. → systems/scheduled-tasks, systems/hermes-agent

  • incidents/2026-10-31-asana-access-failures — Both Asana paths on CT460 are non-functional. The MCP server asana (https://mcp.asana.com/v2/mcp, auth: oauth, enabled: true) has never authenticated — ~/.hermes/mcp-tokens/ holds only asana.client.json (319 B, mtime 2026-10-26 01:07) and the logs carry "MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run hermes mcp login asana interactively first"; the hosted connector also rejects the call shape. The fix needs an interactive hermes mcp login asana, which a cron context cannot perform. → systems/hermes-agent

  • incidents/2026-10-30-ct460-host-outage-scheduler-gap — RESOLVED (root cause not established). CT460 went dark for 5 h 30 m on 2026-10-30: the PVE host pve rebooted (boot 16:24:00), CT460 returned 16:28:00, the previous gateway life (pid=3913841) was logged as exited UNCLEANLY at 16:35:50 ("no exit path ran — SIGKILL / OOM / VM death"), and the scheduler resumed at 16:37:21 with 8 catch-up runs. Zero execution rows exist between 10:58:20 and 16:37:21; the day still closed 138 / 0 against 10-29's 177 / 0 (≈39 slots lost, no failures). No monitor alerted — the two jobs that could have (cbfe434a3b5e, 1a8e90ea2271) are disabled.

  • incidents/2026-10-30-mnemosyne-health-monitor-paused — RESOLVED 2026-10-30T10:02:48 (its 00:01:08 slot was still running, not dead: it closed completed after 10 h 01 m 40 s and delivered, auto-closing ea79a2_d93c3acdc4f2; ea79a2107e36 is enabled again with last_status: ok / streak 0). The Mnemosyne Health Monitor cron job was auto-paused overnight and its failure notice never reached Telegram — the only trace is the incident row. ea79a2107e36's 2026-10-30 00:01:08 slot closed unknown at 01:04:22 on "Cron execution unverified; reconcile ~/.hermes/cron/admissions/ad897cbc9f…json: Gateway connection failed; no local fallback", opening incident ea79a2_d93c3acdc4f2 (failure_type: unknown, detected) — and the scheduler paused the job at that moment. Fleet is now 29 jobs (25 enabled / 4 disabled); live last_status: error, failure_streak: 1. ⚠ Two traps in one fault: the failure notice itself failed to send (send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed — deliveries.db has zero rows for 2026-10-30), and the run closed unknown rather than failed. The job was not re-enabled and the gateway was not restarted by the stocktake — root cause is unverified. → systems/scheduled-tasks, systems/mnemosyne

  • incidents/2026-10-29-conduit-voice-silence — Every Conduit voice call connected and then died silently; three faults sat in series, and the first two real calls in the product's history were placed on 2026-10-29 after all three were fixed. (A) voice.speech_server.url was the truncated wss://…:8443/v1/ — the plugin dials it verbatim and the bare path answers 403, reported as "the speech server refused the key." (B) dashboard.basic_auth.session_ttl_seconds was the 12 h default, so an expired phone session gets WS 4401 with no Hermes turn and no error on the device (the phone re-authenticated at 08:00:59, minutes after a silent call). (C) Both host providers were dead — tts.provider: openai → Mac /v1/speech 502, stt.provider: groq → 403/Cloudflare 1010 (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → …/v1/realtime, TTL → 2 592 000 s (30 d), stt.provider: local (faster-whisper base), tts.provider: ollama_local; dashboard restarted 08:03:26. Verified by driving the whole call path for real, then by calls at 08:38:04 and 08:49:36. ⚠ Two TTL keys exist — dashboard.basic_auth.session_ttl_seconds (raised) vs dashboard.session_ttl_seconds (still 43 200). status: resolved.

  • incidents/2026-10-28-job-radar-stage1-iteration-cap — The radar's two-stage split ran for real on 2026-10-28 and landed nothing: stage 1 stopped at the agent iteration cap before writing its handoff, so both stage-2 slots correctly no-oped (silent). Two consecutive stage-1 failures, two different ceilings (tool-call budget 10-27, iteration cap 10-28); no Notion writes for a second day. RESOLVED 2026-10-29 — the next stage-1 run (dca8482e4f76, 07:00:59 → 10:16:12, 3 h 15 m 13 s) closed completed and wrote handoff/2026-10-29.json, which stage 2 consumed at 10:40:59 and delivered as 20 pages + 2 cover letters (missing_cover_letter_count 0). status: resolved, review 2026-11-13.

  • incidents/2026-10-27-job-radar-budget-exhausted — The Daily Marketing Job Radar exhausted its tool-call budget for the second day running — and this time the run is recorded as failed, not completed. On 2026-10-27 the 07:00 slot ran 07:01:08 → 11:03:54 (4 h 2 m 46 s) and its delivered report opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)": discovery/verify/dedup/classify finished, Notion writes, cover letters and the final audit never ran (Notion DB held 451 entries, 0 added for the day). Incident row dca848_e93a538ade6b opened 11:03:53 (alerted 11:03:59, failure_type: auth — wrong for a budget fault) and was still open at the 2026-10-28 audit. Response the same day: the radar was split into stage 1 dca8482e4f76 (Discovery, 07:00, deliver: local) and new stage 2 5fc292ff0f3b (Write and Letters, 11:30/14:30, topic 1030), created 17:34:15, two completed test fires that evening → runbooks/job-radar. RESOLVED 2026-10-29: both of this incident's ledger rows (dca848_e93a538ade6b, dca848_0a2f03b5952b) closed resolved at 10:16:12 on the first successful post-split run — dca8482e4f76 restored delivery to local, wrote the handoff inside budget, and no cover letter went missing (missing_cover_letter_count 0). status: resolved, review 2026-11-13.

  • incidents/2026-10-26-planned-restart-orphaned-cron-run — A planned gateway restart orphaned a live cron run — the second instance of the orphan class, and the first that alerted. On 2026-10-26 at 14:33:35 AWST the gateway took a planned SIGTERM (under_systemd=yes); the shutdown drain reported timed_out=True after just 1.35 s of its 30 s budget and interrupted ea79a2107e36 (Mnemosyne Health Monitor, 12:01 slot) 2 h 34 m in. It closed failed on a new string — lost its durable fire claim ownership — raised two cron_incidents rows (14:33:37 unverified / reconcile admission + 14:35:59 fire claim), delivered the failure at 14:35:59, and left the job on failure_streak: 2 from one interrupted fire. Two Mnemosyne Watchdog reds followed at 14:46 and 15:01 and self-resolved at 15:16:12. Net = 169 / 3 for 10-26. Unresolved: why a planned stop bypasses the drain deadline, and whether failure_streak should count the unverified row. → also updates incidents/2026-10-20-gateway-unclean-exit-orphaned-run (the class now recurs without a crash).

  • incidents/2026-10-25-gmail-digest-runaway-and-refire — The Gmail Action Digest ran 1 h 55 m and fired six times on 2026-10-25. The 08:30 slot started 08:31:14 and closed 10:26:37 failed on RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed) — against 91 s (10-23) and 104 s (10-24). The same job then fired five more times (17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53); the 19:18:41 fire died on a new signature — Cron admission missing: the owner never admitted this fire (distinct from the 10-20 orphan) — and all six fires, failures included, carry delivery_outcome: delivered, so up to six digests reached Anthony. Neither failure raised a cron_incidents row; the job now reads last_status: delivery_queued with an empty last_error. Hypothesis (unconfirmed): the 08:31 fire adopted an unrelated long-lived session — its worker made 70 terminal calls on VM101 /dev/sdb backup work with context growing 66 k → 148 k tokens, while context compression was unavailable all day (invalid_grant). → systems/scheduled-tasks, incidents/2026-10-22-nous-portal-refresh-token-invalid

  • incidents/2026-10-22-nous-portal-refresh-token-invalid — Invalid refresh token — Nous Portal runtime credentials are dead, so context summarisation failed. Gateway-log counts: 1 line 2026-10-22, 3 on 10-24, 20 on 10-25 (last 14:53:04, agent.context_compressor: Failed to generate context summary), 0 since the 19:27:13 restart. ⚠ Masked, not fixed: at 2026-10-25 19:13 all eight auxiliary: slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off nous to no-think/freeapi/auto on custom:omni-101 (fallback no-think/freeapi/deepseek-v4-flash → no-think/kg/kilo-auto/free, free_only: true; compression timeout 90 s, vision 180 s). No hermes model re-authentication has been run and nous is still the first entry in fallback_providers, so failover tries a dead credential. → systems/model-providers, systems/current-profile

  • incidents/2026-10-24-conduit-call-request-refused — The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56). He asked to be called to test Conduit calls (Call me now to test calls. → Yes. You do → CAL ME); all three replies asserted the agent has no calling capability, with zero tool calls in the session. conduit_call_user (toolset conduit) exists, is registered for every session, is reachable via tool_search, and the plugin ships the calling-the-user skill; his call settings (enabled / when_asked / decides) were permissive and the device was paired. Root cause: capability asserted instead of discovered — the first reply came from kilo-auto/free, but the switch to smart-route repeated it. Same audit also corrects the 10-22 incident's closing evidence: conduit-calls.json → ends records turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne cron sessions — history (real calls) is empty and no call has been placed since (superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → incidents/2026-10-29-conduit-voice-silence) the memory fix. → systems/conduit, incidents/2026-10-22-conduit-memory-start-timeout

  • incidents/2026-10-22-conduit-memory-start-timeout — Conduit voice calls were silently running without memory, 2026-10-22 → 10-23. The phone plugin gave one request — including the Mnemosyne provider start — a hard-coded 10 s budget (MEMORY_TIMEOUT_S), while Mnemosyne's cold start for a conduit-voice session runs 10–45 s. Three calls were abandoned at the memory layer (Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with no recall of prior conversation. Fixed 10-23 in the plugin's working tree: default 60 s (CONDUIT_MEMORY_TIMEOUT_S override) plus a TokenError guard that logs and proceeds without recall instead of aborting; dashboard restarted 18:54:43 and no abort line exists after it. ⚠ Corrected 2026-10-25: the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a false positive — those were the two Mnemosyne cron turn-ends of that night, and conduit-calls.json → history (real calls) is empty, so the fix has no positive call evidence and no call has been placed since (superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → incidents/2026-10-29-conduit-voice-silence). The patch is uncommitted — a plugin update reverts it. The cold start itself was not optimised. → systems/conduit, systems/mnemosyne

  • incidents/2026-10-21-omniroute-pre-dispatch-filter-skip — A third failure signature from the custom:omni-101 / smart-route pin, 2026-10-21. The Daily Marketing Job Radar (dca8482e4f76, 07:00 weekday) died at 07:40:48 after 40 m 36 s on RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s — no 503, no connection error, no repetition loop, and zero 503 rows exist for that day (last 2026-10-19 09:02). Its 07:00 sibling 85edf6772bca completed normally, so the node was serving; the run wrote no output (cron_incomplete_no_output, 82 msgs / 47 tool calls). Incident dca848_014affeb6954 was closed 2026-10-22 07:59:22 when the next daily slot ran clean. Not caused by the 22:47 hermes update — that landed 15 h later. → systems/omniroute

  • incidents/2026-10-20-gateway-unclean-exit-orphaned-run — The gateway exited uncleanly on 2026-10-20 (09:09:01, suspected_oom=False) and orphaned a live cron run. 08eb86f02d37 (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed unknown — “Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.” No delivery row exists for the slot and no incident row was created; the job table keeps showing the previous slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a 27-day clean run. Open: orphan-recovery on restart, and treating a delivery-targeted unknown as a probable non-delivery.

  • incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence — Third iteration-cap in seven days, and the first that raised no incident at all. Job Radar Stage 1 (dca8482e4f76) ran 07:00:33 → 16:01:52 = 9 h 01 m 19 s on 2026-11-02, hit the agent iteration cap before the handoff was written, and closed status: completed / error: NULL / delivery_outcome: suppressed. Both Stage 2 slots declined correctly (92665337… 11:30 and the 05a3f00b… 14:30 safety net both exited [SILENT]), so nothing reached Notion, no cover letters were generated, and the day's 164 deduped candidates were never persisted — a state a status-only health check reports as green. Contrast 10-27 (dca848_e93a538ade6b) and 10-28 (dca848_0a2f03b5952b), where the identical cap did open rows whose error text is the run's own INCOMPLETE report: on 11-02 the ledger lost the red row it used to produce. Discovery itself worked — SEEK 51, LinkedIn 60 IDs, Indeed live, WA Government 35 (197 raw / 164 new after three-level dedup) — and a URL-canonicalisation bug was found and fixed mid-run (au.seek.com vs au.seek.com.au had been dropping SEEK and Indeed entries from the consolidated list). → systems/scheduled-tasks

  • incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure — The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered: its execution closed unknown at 18:05:13 with "Cron execution unverified … Gateway connection failed; no local fallback", its incident row is still alerted, the job sits at error / failure_streak: 1, and deliveries.db shows no row after 11:01:12 for the rest of the day. Its batch-mate b63ed5817212 took the identical error and self-healed in 68 s. Same page records the inverse silent failure: ecc1fc13655a reads error with last_run 11-01T12:01:00 while owning no execution row and no incident row at all. ✅ RESOLVED 2026-11-02T01:27:21 — the review was never lost. The same execution row (be6ec11a8e68428084af9ab18fb148db) closed completed / delivered at 01:27:22 after 7 h 26 m 43 s, the delivery lands at 01:27:21.802246 and incident 79f9bb_9c443b602eec auto-closed at 01:27:21.606691 — so the 18:05:13 unknown was a reconciler timing artefact on a still-running job, not a verdict (page now resolved). Batch-mate b63ed5 closed in 68 s and 79f9bb in 7 h 26 m, both completed; an unknown row on a long job is a timing artefact, not a failure. ecc1fc13655a's ghost error cleared too (12:00:33 → completed / delivered 12:29:26), taking non-clean last_status 2 → 1.

  • incidents/2026-11-01-cron-worker-sigterm-orphaned-claims — Three restart-safe cron workers killed by SIGTERM (-15) after adopting their claim, so the scheduler recorded "whether side effects ran is unknown": 85edf6772bca @07:30:46, 96eb95ce12f2 + 0e753e9720fd @09:02:18. A fourth variant of the orphan class — and the first with no gateway restart, reload or drain anywhere in the window (PID 175269 up since 10-31 19:24:40; relay counter monotonic through both signals), so it cannot be attributed to reload behaviour. Likely trigger: a 30 s worker-ack window smaller than this host's cold start (83 plugins / 69 enabled discovered in 70 s). All three rows now resolved; one execution left unknown.

  • incidents/2026-11-03-job-radar-stream-active-timeout — The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the ledger went red. dca8482e4f76 ran 07:00:33.819190 → 13:28:52.305995 = 6 h 28 m 18 s on 2026-11-03 and closed failed / RuntimeError: stream active timeout, with the failure notice delivered (13:28:51.741724) and incident dca848_dfd4bea84f2a (failure_type: timeout, alerted, open) — unclosed rows 0 → 1. The run's activity stops at 12:23:28; the harness does not declare the turn dead until 13:28:49 — a 65 m 21 s silence with no tool call, no model turn and nothing in the ledger, and no "maximum number of tool-calling iterations" message anywhere (this is a provider-stream timeout, not the cap that ended 10-27 / 10-28 / 11-02). It compacted five times (08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21) in 363 messages. Both Stage 2 slots declined correctly (5fc292ff0f3b 11:30:36 → 11:32:16; 14:30:32 → 14:33:11) because no handoff was written, and radar-ref/handoff/ still holds only 10-27 / 10-29 / 10-30 — so 11-02's 164 deduped candidates and 11-03's discovery are both unrecovered. Pulse carried it to Anthony in the desktop session. Answers the 11-02 page's open question 3: suppression tracks the terminal state — completed → nothing delivered, failed → notice delivered. → incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence, systems/scheduled-tasks

  • incidents/2026-08-08-gitea-config-backup-push-dead — Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47. e27da6c34bbb reads last_status: ok and its executions read completed every night while 75 commits sit unpushed (fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net' — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (message_id 8447, 2026-10-20 11:00); AgentMail failed. No off-box ~/.hermes backup in 2.5 months. Second instance of the pattern after the GitHub Trending Radar token revocation. → infrastructure/backups

  • incidents/2026-10-18-degenerate-stream-canned-substitute — The repetition-loop failure mode, 2026-10-18: the FreshRSS Midday run (ecc1fc13655a) wrote "Your request was not processed…", delivered it to topic :346 at 12:08:11, then died at 12:08:25 on RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop. A delivered message is not evidence of delivered content — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? Second surface (2026-10-20 18:37): an interactive desktop session received the same canned line after Rebuilt-message restart limit (3) exceeded → reason=rebuilt_restart_limit_exceeded … response_len=0, so the substitute is emitted by the harness's failure path generally, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — was never answered.

  • incidents/2026-10-16-omniroute-503-resource-pressure — omniroute-1 load-sheds model calls with HTTP 503 … resource pressure — two bursts: 2026-10-16 → 10-17, and again 2026-10-19. Burst 1: 9 model calls across 7 jobs over ~19 h, self-cleared (reported then as 10-17 = 162 / 5; the DB reads 163 / 5 — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: 9 model calls across 8 jobs on 2026-10-19 (06:01 + 07:01 → 12:01, whole day 159 / 13). /api/health read 200 ok throughout both. Status: active → back to monitoring on 2026-10-21 — 2026-10-20 ran 168 / 0 with zero rows carrying the resource-pressure string, so the second burst is over; it stays a standing risk until the single-node topology changes. 4 rows closed on 10-20 (dca848_867d9caa96a8, 85edf6_256c5e150bc4, ecc1fc_c9976ea24553, ecc1fc_21a72aa11cc4); unclosed rows 13 → 9 (8 alerted / 1 detected); the ledger still tags every 503 row failure_type: unknown, so triage on error text. Single-node topology still open — and every job that failed reads model: smart-route, provider: custom:omni-101.

  • incidents/2026-10-14-omniroute-node-connection-refused-storm — Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15: omniroute-1 (100.78.63.20:20128, provider custom:omni-101) refused TCP connections, so every model-calling cron job failed on every slot — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. All 18 enabled model-calling jobs ride this one node with no failover. Recovered 10-15 22:01 (resolved)

  • incidents/2026-10-09-cron-provider-timeout-storm — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all RuntimeError: Request timed out. on the aliases behind custom:omni-101, 00:00 → 15:00 AWST. CLOSED — all 12 incident rows are now resolved; the last two (1b23c0_c35703f637b8, 1b23c0_103cbe04a79b, both on the weekly Friday job 1b23c042e17d) closed 2026-10-23 08:04 when that slot ran clean, and the job reads ok again

  • incidents/2026-10-09-memory-recall-relevance — Recall relevance: a stale conversational episode outranked current-state facts on an open question. ✅ RESOLVED 2026-11-01 — the ranking weights were persisted, in ~/.hermes/mnemosyne/config.yaml rather than the file the 2026-10-10 probe read; the original regression query now returns only current-state facts, and conflict 152 was auto-resolved on 2026-10-29

  • incidents/2026-10-05-dead-model-pin-repin — Seven cron jobs on dead model pins (cliprox/terra, qwen3.5:397b, deepseek-v4-flash, gpt-5.6-terra) repinned to routing aliases on custom:omni-101; failing set 7 → 0 (resolved 2026-10-05)

  • incidents/2026-10-03-mnemosyne-vector-recall-fts-only — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic

  • incidents/2026-10-03-ct450-hermes-update — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented

  • incidents/2026-10-01-cliproxy-connection-auth-expired — OmniRoute cliprox/terra connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); closed 2026-10-05 when all three were repinned off the pin

  • incidents/2026-08-26-work-outlook-basic-auth-disabled — Work Outlook mailbox unreachable: tenant allows only XOAuth2, himalaya configured for password (open since 2026-08-26)

  • incidents/2026-09-28-hermes-managed-runtime-deps — Managed-runtime venvs missing dotenv/mnemosyne broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. Recurred 2026-10-10 inside the gateway runtime env — mnemosyne and elevenlabs missing from the committed uv env the gateway loads; both repaired and verified the same evening

  • incidents/2026-08-18-mnemosyne-capture-pipeline — Mnemosyne capture pipeline broken (Aug 2026, superseded): capture verified healthy 2026-10-12 — 786 working rows, +2 that day, 98.3 % embedded

  • incidents/2026-08-15-skills-index-drift — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration

Monitoring & Dashboards

Reports

Infrastructure

Services

Networking

  • infrastructure/proxmox — PVE hosts (two: pve 9.2.10 + m93p 9.2.10, unclustered, IDs collide), VMs, LXC containers, storage (/mnt/ssd0 76.43 % / ~169 GiB free, 2026-10-31)
  • infrastructure/tailscale — Tailscale serve routes (CT460 verified exactly) and the 50-node tailnet inventory (re-verified 2026-10-31: mattermost back online after ~40 days; 7 offline ages refreshed)
  • systems/cloudflare-tunnel — ct207-hosted tunnel serving Hermes publicly

Concepts

Decisions

Comparisons

Runbooks

Raw