BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic. Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)". A scan of all 117 pages found exactly one such line; it is now properly quoted. Quartz now builds cleanly: 690 files emitted and deployed. RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page). The page claimed the recall-ranking fix "was not present in live config" and that conflict 152 was "the only open conflict". Both claims were false: - The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5, fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 — not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the 2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks. - The original regression test now passes: "mnemosyne recall 'Are you up to date'" returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale Uber Eats episode no longer outranks anything. - Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise. All three open items closed; page moved to status: resolved. LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01): - infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping times out); m93p peer record reads false for the 4th consecutive time while the host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages refreshed; no IP changes, no missing peers. - infrastructure/docker-services.md: CT201's Tailscale IP was wrong (100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale worldmonitor-seeder-fast entry removed from the CT200 list, which is now regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to thirsty_payne. - infrastructure/proxmox.md: storage percentages refreshed on both hosts from live pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an elevated load average recorded. - systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md: re-probed and verified_on bumped. - systems/skills-index.md: regenerated — 301 unique skills in 68 categories, 46 archived. (301 is the unique (category, skill_name) count; find returns 302 files due to the known nested obsidian-git-sync alias duplicate.) Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
40 KiB
title, type, status, created, updated, verified_on, confidence, tags
| title | type | status | created | updated | verified_on | confidence | tags | |||
|---|---|---|---|---|---|---|---|---|---|---|
| Anthony's Systems Wiki | collection | active | 2026-07-22 | 2026-11-01 | 2026-11-01 | high |
|
Anthony's Systems Wiki
Operational knowledge for Anthony's agents, infrastructure, model routing and automation. Last updated: 2026-11-01 (nightly audit — Conduit's plugin self-updated to
0.122.2on 10-31 18:05, its uncommitted memory-timeout patch is now upstream and the tree is clean, and the call store was re-shaped (historyempty); a new fault is documented — the app's "local" voice mode never applies because the Mac mini speech server's LLM proxy is disabled (501,llm_proxy.requests: 0) and its realtime socket rejectssession.update; the same evening's 18:06:41 gateway reload force-drained after the full 1,800 s cap and orphaned a cron run at 18:37:38, leaving 10-31 at 160 / 4; both Asana paths on CT460 are still non-functional)⚠ Two Proxmox hosts exist — always qualify a guest with its host. A second unclustered hypervisor,
m93p(100.104.51.98, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. Guest IDs collide between the two hosts.
Start here
- current-state — Current systems status
- systems/hermes-agent — Hermes Agent overview and version
- systems/current-profile — Active Hermes profile details
Systems
Core
- systems/hermes-agent — Agent version, install, config paths
- systems/current-profile — Active profile details
- systems/nextcloud — Nextcloud file sharing (CT270)
- systems/guanaco — Guanaco LLM proxy on CT205 (Ollama stack)
- systems/model-providers — LiteLLM + OmniRoute providers, routing setup
- systems/memory-backend — Mnemosyne vector memory, settings
- systems/mnemosyne — Mnemosyne cron jobs and config evidence
- systems/omniroute — OmniRoute catalog, combos, and host path
- systems/openrouter — OpenRouter current state and history
- systems/opencode-go — OpenCode Go subscription, model limits, pricing
- systems/nanobot — Nanobot current state and history
- systems/browser-backend — Camofox (primary,
systemd --useron CT460, REST:9377) — the only backend; Steel was fully removed from the stack on 2026-10-29 (plugin deleted,cloud_providerback tocamofox, CT201 and CT333 deployments torn down, Serve routes deleted;STEEL_BASE_URLsurvives only in already-running process environments); memory-containment drop-in and the 2026-10-12 ceiling event documented - systems/hermes-ui — Browser Hermes Desktop (przbadu/hermes-ui)
- systems/paseo — Paseo coding-agent daemon + web terminal workspace (CT460, :6767)
- systems/ignis — Ignis browser-native Obsidian (wiki interface)
- systems/terminal-backend — Shell persistence, working directory
- systems/tool-search — Always-on lazy discovery for external tools
Integrations
- systems/conduit — Conduit: Hermes on Anthony's phone (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree
~/.hermes/plugins/conduit_push, v0.122.2(HEAD9eac857, 2026-10-31 12:04 — "Every profile that turns Conduit on runs the main install (#482)"; +148 commits on since the0.69.2/942ddb8baseline, working tree clean), upstreamkaishi00/hermes-conduit-notifier. Gateway on:9119is now pid 176771, started 2026-10-31 19:35:02 (superseded: restarted 2026-10-25 18:42:33, which is when the upstream commits went live). One registered device (iPhone app 2.2.0). ✅ The first two real calls in the product's history are now on record — 2026-10-29 08:38:04 and 08:49:36 — after three faults in series were repaired (truncated speech-server URL →…/v1/realtime; 12 h dashboard session TTL → 30 d; deadgroqSTT →local, deadopenaiTTS →ollama_local) → incidents/2026-10-29-conduit-voice-silence. So the memory fix is no longer evidence-free. ✅ That patch is gone — the tree is clean andMEMORY_TIMEOUT_Sships upstream indashboard/api/memory.py. ⚠ New fault: "local" voice mode never applies — the Mac mini speech server's LLM proxy is disabled (LLM endpoint → 501,/v1/usagereadsllm_proxy.requests: 0, realtime socket rejectssession.update), while the app's "test connection" checks socket reachability only → incidents/2026-10-31-conduit-voice-local-mode-not-applying. The call store was re-shaped:historyis empty andlast_request_atisnull, so the two 10-29 calls are no longer in it. The 10-24 refusal fault has not recurred → incidents/2026-10-24-conduit-call-request-refused. - systems/messaging-integrations — Telegram, Discord, email, Slack, etc. Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26 (peer offline 36 d,
tx 312 / rx 0; host answers neither:8065nor ICMP). 268 connect timeouts on 10-25 vs 264 on 10-24; the reconnect attempt counter reset to 54 at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost. - systems/push-notifications — Cadu/Vory push bridge (
hermes-push,vory-push): gateway on:9119, state in~/.hermes/push/. Broken —connected: false(the bridge dials a name that resolves to127.0.1.1where nothing serves 443); chronic, not a blip. - systems/scheduled-tasks — As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean
last_status4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), andea79a2107e36is ENABLED again (last_status: ok, streak 0) after its 00:01:08 slot closedcompletedat 10:02:48; deliveries 8/8delivered. Prior window — as of 2026-10-30 03:3x: 29 jobs (25 enabled, 4 disabled); non-cleanlast_status3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries alldelivered. ⚠ea79a2107e36(Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closedunknownon a gateway-connection failure — and its failure notice was NEVER DELIVERED (deliveries.dbhas zero rows for 2026-10-30), so the incident row is the only trace → incidents/2026-10-30-mnemosyne-health-monitor-paused. Every other previously-red job has cleared took/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-cleanlast_status5 → 3 (all three are stale re-runs of old faults and everyfailure_streakread 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries alldelivered. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters withmissing_cover_letter_count 0(Notion 455 → 475). Both stage-1 incident rows closedresolvedat 10:16:12; the lone survivor is the Facebook monthly's0e753e_e7f8194a7d06.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.**dca8482e4f76ran 07:01:08 → 11:03:54 (4 h 2 m 46 s), closedfailed, and delivered "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery finished, no Notion writes and no cover letters. Incident rowdca848_e93a538ade6bopened 11:03:53 and is still open. The same day the single radar job was split into two: stage 1dca8482e4f76(Discovery, 07:00,deliver: local) + new stage 25fc292ff0f3b(Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, twocompletedtest fires that evening). The two incident rows the 2026-10-26 planned restart raised auto-closed 00:38:32, and the 2026-10-26 orphan class did not recur. → incidents/2026-10-27-job-radar-budget-exhausted, runbooks/job-radar. Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-cleanlast_status, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.ea79a2107e36(Mnemosyne Health Monitor, 12:01 slot) was interrupted by a planned 14:33:35SIGTERMrestart that timed the drain out after 1.35 s, killed after 2 h 34 m on a new string (lost its durable fire claim ownership), and raised twocron_incidentsrows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → incidents/2026-10-26-planned-restart-orphaned-cron-run. New fleet precedent: a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). Unclosed incident rows 2 → 3; one closed:c0a9fa_764558b5ae83auto-closed 09:13:00 when the radar's 09:00 slot ran clean, so every row of the 2026-10-16 503 resource-pressure incident is now closed → incidents/2026-10-16-omniroute-503-resource-pressure. Non-cleanlast_status5 → 4:c0a9fa314542and08eb86f02d37cleared took;ea79a2107e36joins aserror, streak 2. Pins unchanged. ⚠fad5a86979d4's 03:08 slot completed but delivereddelivery_failed— Thread 34795 not found (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered “RUN INCOMPLETE — tool-call budget exhausted” and its cover-letter pipeline changed that morning → runbooks/job-radar. Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-cleanlast_status, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0. ⚠ Both 10-25 failures are08eb86f02d37(Gmail Action Digest), which also fired six times that day — and neither raised an incident row → incidents/2026-10-25-gmail-digest-runaway-and-refire. ⚠ The global default reverted tocustom:omni-101/smart-routeon 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again → systems/current-profile. ⚠ Nous Portal credentials still invalid — all 8auxiliaryslots moved tono-think/freeapi/autoat 19:13 → incidents/2026-10-22-nous-portal-refresh-token-invalid. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-cleanlast_status, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third consecutive clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0. Zero incident rows opened or closed in-window; pins unchanged. ⚠ Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls → incidents/2026-10-24-conduit-call-request-refused. Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-cleanlast_status, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four. The 2026-10-09 provider timeout storm is now fully closed:1b23c042e17dran clean on its 10-23 08:04 slot, closing the storm's last two rows, so all 12 are closed and unclosed incident rows fell 4 → 2. ⚠ New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (CONDUIT_MEMORY_TIMEOUT_S) + degraded-not-aborted onTokenError; dashboard restarted 18:54:43, no abort since. Patch uncommitted → incidents/2026-10-22-conduit-memory-start-timeout, systems/conduit. The 10-22 stocktake overrun did not repeat (10-23 03:07 committed250c735). Prior window: As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-cleanlast_status, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.dca8482e4f76(Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s onRuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s→ incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.08eb86f02d37cleared (08:30 slot clean; incident resolved 08:37:21) and 2026-10-20 settled at 169 / 0 + 1unknown. ⚠ Hermes Agent was updated to0.22.3at 22:47 → 22:57 with its pre-update snapshotskipped→ systems/hermes-agent. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. Theomni-101503 fault did not recur (back tomonitoring) and 4 incident rows closed → 9 unclosed. ⚠ Two new findings: the Gitea config backup has not pushed since 2026-08-08 (75 unpushed commits) while its row readsok→ incidents/2026-08-08-gitea-config-backup-push-dead; and a gateway unclean exit on 2026-10-20 orphaned a digest run — no delivery row, no incident → incidents/2026-10-20-gateway-unclean-exit-orphaned-run. Also: the global model default moved tonousdirect (meituan/longcat-2.5-preview:free) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while 20 of 28 still pin it → systems/current-profile. Prior window: 28 cron jobs (25 enabled, 3 disabled); 12 carry a non-cleanlast_statusas of 2026-10-20 00:09 — up from 7, because theomniroute-1503 resource-pressure fault returned on 2026-10-19 (second burst in four days: 9 model calls across 8 jobs, 06:01 and 07:01 → 12:01, allmodel: smart-routeoncustom:omni-101, while/api/healthstayed200 ok) → incidents/2026-10-16-omniroute-503-resource-pressure. 2026-10-19 closed 159 / 13 against 10-18's 168 / 1. The Mosquito Tracker was removed (29 → 28 jobs), and the 2026-10-18 degenerate-stream placeholder-delivery mode remains an open question → incidents/2026-10-18-degenerate-stream-canned-substitute. Earlier: the connection-refused storm (2026-10-14 → 10-15) — 24 model-call failures across 12 jobs, ~34 h of digests blanked. All 18 model-calling jobs still sit oncustom:omni-101with no failover. - systems/skills-index — Installed skills inventory (302 skills (69 categories) as of the 2026-10-30 stocktake; regenerated daily by the 03:00 job). The generator now excludes
_archive/— the earlier 346/347 counts included it. Freshness at the 2026-10-30 run: 0 stale / 109 fresh / 3 no-date of 112 pages (freshness-20261030.txt). Note:SKILL.mdfiles on disk still exceed the index by one (303 vs 302) — the generator de-dups one nested duplicate, so 302 is the correct unique-skill figure; see systems/scheduled-tasks. ⚠ The 2026-10-22 slot ran 97 m 58 s (vs 16 m 01 s on 10-21), finishedcompletedwith deliverysuppressedand committed nothing; the 10-23 slot closed that hole.
Incidents
-
incidents/2026-10-31-conduit-voice-local-mode-not-applying — OPEN — the app's "local" voice mode never applies, and the connection test lies. The test checks socket reachability only, while every local turn is still impossible: the Mac mini speech server's LLM proxy is disabled (LLM endpoint → HTTP 501 "Start the server with
--enable_llm_proxy"),/v1/usagereadsllm_proxy.requests: 0, and its realtime socket accepts the connection but rejectssession.updateas an unknown event.voice.modeis a phone-side selector — nomodekey exists inconfig.yaml → voiceorconduit-settings.json— so a "mode did not apply" cannot be diagnosed on host config. Found via session20261031_193709_40b9c2(19:37:09, 30 msgs / 17 tool calls);voice.local_live.front_modelwas repointed tomistral/voxtral-small-latestduring it and nothing was changed on the Mac. → systems/conduit -
incidents/2026-10-31-gateway-reload-force-drain-orphaned-run — A gateway reload waited its full 1,800 s drain cap, then force-drained and orphaned a live cron run. At 18:06:41 the gateway took the reload; the drain reported
timed_out=Trueat the cap and force-drained at 18:37:38 (new gateway 18:38:43).ea79a2107e36(Mnemosyne Health Monitor, 12:01:06 slot, 6 h 38 m 09 s in) closedfailedat 18:39:15 on "lost its durable fire claim ownership". Same class as incidents/2026-10-26-planned-restart-orphaned-cron-run — second instance, and the first on a plain reload. A second, clean restart followed at 19:23:39 → 19:24:40; 2026-10-31 closed 160 / 4. → systems/scheduled-tasks, systems/hermes-agent -
incidents/2026-10-31-asana-access-failures — Both Asana paths on CT460 are non-functional. The MCP server
asana(https://mcp.asana.com/v2/mcp,auth: oauth,enabled: true) has never authenticated —~/.hermes/mcp-tokens/holds onlyasana.client.json(319 B, mtime 2026-10-26 01:07) and the logs carry "MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Runhermes mcp login asanainteractively first"; the hosted connector also rejects the call shape. The fix needs an interactivehermes mcp login asana, which a cron context cannot perform. → systems/hermes-agent -
incidents/2026-10-30-ct460-host-outage-scheduler-gap — RESOLVED (root cause not established). CT460 went dark for 5 h 30 m on 2026-10-30: the PVE host
pverebooted (boot 16:24:00), CT460 returned 16:28:00, the previous gateway life (pid=3913841) was logged as exited UNCLEANLY at 16:35:50 ("no exit path ran — SIGKILL / OOM / VM death"), and the scheduler resumed at 16:37:21 with 8 catch-up runs. Zero execution rows exist between 10:58:20 and 16:37:21; the day still closed 138 / 0 against 10-29's 177 / 0 (≈39 slots lost, no failures). No monitor alerted — the two jobs that could have (cbfe434a3b5e,1a8e90ea2271) are disabled. -
incidents/2026-10-30-mnemosyne-health-monitor-paused — RESOLVED 2026-10-30T10:02:48 (its 00:01:08 slot was still running, not dead: it closed
completedafter 10 h 01 m 40 s and delivered, auto-closingea79a2_d93c3acdc4f2;ea79a2107e36is enabled again withlast_status: ok/ streak 0). The Mnemosyne Health Monitor cron job was auto-paused overnight and its failure notice never reached Telegram — the only trace is the incident row.ea79a2107e36's 2026-10-30 00:01:08 slot closedunknownat 01:04:22 on "Cron execution unverified; reconcile~/.hermes/cron/admissions/ad897cbc9f…json: Gateway connection failed; no local fallback", opening incidentea79a2_d93c3acdc4f2(failure_type: unknown,detected) — and the scheduler paused the job at that moment. Fleet is now 29 jobs (25 enabled / 4 disabled); livelast_status: error,failure_streak: 1. ⚠ Two traps in one fault: the failure notice itself failed to send (send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed—deliveries.dbhas zero rows for 2026-10-30), and the run closedunknownrather thanfailed. The job was not re-enabled and the gateway was not restarted by the stocktake — root cause is unverified. → systems/scheduled-tasks, systems/mnemosyne -
incidents/2026-10-29-conduit-voice-silence — Every Conduit voice call connected and then died silently; three faults sat in series, and the first two real calls in the product's history were placed on 2026-10-29 after all three were fixed. (A)
voice.speech_server.urlwas the truncatedwss://…:8443/v1/— the plugin dials it verbatim and the bare path answers 403, reported as "the speech server refused the key." (B)dashboard.basic_auth.session_ttl_secondswas the 12 h default, so an expired phone session gets WS 4401 with no Hermes turn and no error on the device (the phone re-authenticated at 08:00:59, minutes after a silent call). (C) Both host providers were dead —tts.provider: openai→ Mac/v1/speech502,stt.provider: groq→ 403/Cloudflare 1010 (blocked, not mis-keyed). Fixed 08:02–08:4x: URL →…/v1/realtime, TTL → 2 592 000 s (30 d),stt.provider: local(faster-whisperbase),tts.provider: ollama_local; dashboard restarted 08:03:26. Verified by driving the whole call path for real, then by calls at 08:38:04 and 08:49:36. ⚠ Two TTL keys exist —dashboard.basic_auth.session_ttl_seconds(raised) vsdashboard.session_ttl_seconds(still 43 200).status: resolved. -
incidents/2026-10-28-job-radar-stage1-iteration-cap — The radar's two-stage split ran for real on 2026-10-28 and landed nothing: stage 1 stopped at the agent iteration cap before writing its handoff, so both stage-2 slots correctly no-oped (silent). Two consecutive stage-1 failures, two different ceilings (tool-call budget 10-27, iteration cap 10-28); no Notion writes for a second day. RESOLVED 2026-10-29 — the next stage-1 run (
dca8482e4f76, 07:00:59 → 10:16:12, 3 h 15 m 13 s) closedcompletedand wrotehandoff/2026-10-29.json, which stage 2 consumed at 10:40:59 and delivered as 20 pages + 2 cover letters (missing_cover_letter_count 0).status: resolved, review 2026-11-13. -
incidents/2026-10-27-job-radar-budget-exhausted — The Daily Marketing Job Radar exhausted its tool-call budget for the second day running — and this time the run is recorded as
failed, notcompleted. On 2026-10-27 the 07:00 slot ran 07:01:08 → 11:03:54 (4 h 2 m 46 s) and its delivered report opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)": discovery/verify/dedup/classify finished, Notion writes, cover letters and the final audit never ran (Notion DB held 451 entries, 0 added for the day). Incident rowdca848_e93a538ade6bopened 11:03:53 (alerted11:03:59,failure_type: auth— wrong for a budget fault) and was still open at the 2026-10-28 audit. Response the same day: the radar was split into stage 1dca8482e4f76(Discovery, 07:00,deliver: local) and new stage 25fc292ff0f3b(Write and Letters, 11:30/14:30, topic 1030), created 17:34:15, twocompletedtest fires that evening → runbooks/job-radar. RESOLVED 2026-10-29: both of this incident's ledger rows (dca848_e93a538ade6b,dca848_0a2f03b5952b) closedresolvedat 10:16:12 on the first successful post-split run —dca8482e4f76restored delivery tolocal, wrote the handoff inside budget, and no cover letter went missing (missing_cover_letter_count 0).status: resolved, review 2026-11-13. -
incidents/2026-10-26-planned-restart-orphaned-cron-run — A planned gateway restart orphaned a live cron run — the second instance of the orphan class, and the first that alerted. On 2026-10-26 at 14:33:35 AWST the gateway took a planned
SIGTERM(under_systemd=yes); the shutdown drain reportedtimed_out=Trueafter just 1.35 s of its 30 s budget and interruptedea79a2107e36(Mnemosyne Health Monitor, 12:01 slot) 2 h 34 m in. It closedfailedon a new string —lost its durable fire claim ownership— raised twocron_incidentsrows (14:33:37 unverified / reconcile admission + 14:35:59 fire claim), delivered the failure at 14:35:59, and left the job onfailure_streak: 2from one interrupted fire. Two Mnemosyne Watchdog reds followed at 14:46 and 15:01 and self-resolved at 15:16:12. Net = 169 / 3 for 10-26. Unresolved: why a planned stop bypasses the drain deadline, and whetherfailure_streakshould count the unverified row. → also updates incidents/2026-10-20-gateway-unclean-exit-orphaned-run (the class now recurs without a crash). -
incidents/2026-10-25-gmail-digest-runaway-and-refire — The Gmail Action Digest ran 1 h 55 m and fired six times on 2026-10-25. The 08:30 slot started 08:31:14 and closed 10:26:37
failedonRuntimeError: Operation interrupted: waiting for model response (347.2s elapsed)— against 91 s (10-23) and 104 s (10-24). The same job then fired five more times (17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53); the 19:18:41 fire died on a new signature —Cron admission missing: the owner never admitted this fire(distinct from the 10-20 orphan) — and all six fires, failures included, carrydelivery_outcome: delivered, so up to six digests reached Anthony. Neither failure raised acron_incidentsrow; the job now readslast_status: delivery_queuedwith an emptylast_error. Hypothesis (unconfirmed): the 08:31 fire adopted an unrelated long-lived session — its worker made 70terminalcalls on VM101/dev/sdbbackup work with context growing 66 k → 148 k tokens, while context compression was unavailable all day (invalid_grant). → systems/scheduled-tasks, incidents/2026-10-22-nous-portal-refresh-token-invalid -
incidents/2026-10-22-nous-portal-refresh-token-invalid —
Invalid refresh token— Nous Portal runtime credentials are dead, so context summarisation failed. Gateway-log counts: 1 line 2026-10-22, 3 on 10-24, 20 on 10-25 (last 14:53:04,agent.context_compressor: Failed to generate context summary), 0 since the 19:27:13 restart. ⚠ Masked, not fixed: at 2026-10-25 19:13 all eightauxiliary:slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) moved offnoustono-think/freeapi/autooncustom:omni-101(fallbackno-think/freeapi/deepseek-v4-flash→no-think/kg/kilo-auto/free,free_only: true; compression timeout 90 s, vision 180 s). Nohermes modelre-authentication has been run andnousis still the first entry infallback_providers, so failover tries a dead credential. → systems/model-providers, systems/current-profile -
incidents/2026-10-24-conduit-call-request-refused — The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56). He asked to be called to test Conduit calls (
Call me now to test calls.→Yes. You do→CAL ME); all three replies asserted the agent has no calling capability, with zero tool calls in the session.conduit_call_user(toolsetconduit) exists, is registered for every session, is reachable viatool_search, and the plugin ships thecalling-the-userskill; his call settings (enabled/when_asked/decides) were permissive and the device was paired. Root cause: capability asserted instead of discovered — the first reply came fromkilo-auto/free, but the switch tosmart-routerepeated it. Same audit also corrects the 10-22 incident's closing evidence:conduit-calls.json → endsrecords turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne cron sessions —history(real calls) is empty and no call has been placed since (superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → incidents/2026-10-29-conduit-voice-silence) the memory fix. → systems/conduit, incidents/2026-10-22-conduit-memory-start-timeout -
incidents/2026-10-22-conduit-memory-start-timeout — Conduit voice calls were silently running without memory, 2026-10-22 → 10-23. The phone plugin gave one request — including the Mnemosyne provider start — a hard-coded 10 s budget (
MEMORY_TIMEOUT_S), while Mnemosyne's cold start for aconduit-voicesession runs 10–45 s. Three calls were abandoned at the memory layer (Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with no recall of prior conversation. Fixed 10-23 in the plugin's working tree: default 60 s (CONDUIT_MEMORY_TIMEOUT_Soverride) plus aTokenErrorguard that logs and proceeds without recall instead of aborting; dashboard restarted 18:54:43 and no abort line exists after it. ⚠ Corrected 2026-10-25: the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a false positive — those were the two Mnemosyne cron turn-ends of that night, andconduit-calls.json → history(real calls) is empty, so the fix has no positive call evidence and no call has been placed since (superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → incidents/2026-10-29-conduit-voice-silence). The patch is uncommitted — a plugin update reverts it. The cold start itself was not optimised. → systems/conduit, systems/mnemosyne -
incidents/2026-10-21-omniroute-pre-dispatch-filter-skip — A third failure signature from the
custom:omni-101/smart-routepin, 2026-10-21. The Daily Marketing Job Radar (dca8482e4f76, 07:00 weekday) died at 07:40:48 after 40 m 36 s onRuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s— no 503, no connection error, no repetition loop, and zero 503 rows exist for that day (last 2026-10-19 09:02). Its 07:00 sibling85edf6772bcacompleted normally, so the node was serving; the run wrote no output (cron_incomplete_no_output, 82 msgs / 47 tool calls). Incidentdca848_014affeb6954was closed 2026-10-22 07:59:22 when the next daily slot ran clean. Not caused by the 22:47hermes update— that landed 15 h later. → systems/omniroute -
incidents/2026-10-20-gateway-unclean-exit-orphaned-run — The gateway exited uncleanly on 2026-10-20 (09:09:01,
suspected_oom=False) and orphaned a live cron run.08eb86f02d37(Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closedunknown— “Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.” No delivery row exists for the slot and no incident row was created; the job table keeps showing the previous slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a 27-day clean run. Open: orphan-recovery on restart, and treating a delivery-targetedunknownas a probable non-delivery. -
incidents/2026-08-08-gitea-config-backup-push-dead — Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.
e27da6c34bbbreadslast_status: okand its executions readcompletedevery night while 75 commits sit unpushed (fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'— HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (message_id 8447, 2026-10-20 11:00); AgentMail failed. No off-box~/.hermesbackup in 2.5 months. Second instance of the pattern after the GitHub Trending Radar token revocation. → infrastructure/backups -
incidents/2026-10-18-degenerate-stream-canned-substitute — The repetition-loop failure mode, 2026-10-18: the FreshRSS Midday run (
ecc1fc13655a) wrote "Your request was not processed…", delivered it to topic:346at 12:08:11, then died at 12:08:25 onRuntimeError: Model output entered a repetition loop and the stream was cut mid-loop. A delivered message is not evidence of delivered content — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? Second surface (2026-10-20 18:37): an interactivedesktopsession received the same canned line afterRebuilt-message restart limit (3) exceeded→reason=rebuilt_restart_limit_exceeded … response_len=0, so the substitute is emitted by the harness's failure path generally, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — was never answered. -
incidents/2026-10-16-omniroute-503-resource-pressure —
omniroute-1load-sheds model calls withHTTP 503 … resource pressure— two bursts: 2026-10-16 → 10-17, and again 2026-10-19. Burst 1: 9 model calls across 7 jobs over ~19 h, self-cleared (reported then as10-17 = 162 / 5; the DB reads 163 / 5 — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: 9 model calls across 8 jobs on 2026-10-19 (06:01 + 07:01 → 12:01, whole day 159 / 13)./api/healthread200 okthroughout both. Status:active→ back tomonitoringon 2026-10-21 — 2026-10-20 ran 168 / 0 with zero rows carrying the resource-pressure string, so the second burst is over; it stays a standing risk until the single-node topology changes. 4 rows closed on 10-20 (dca848_867d9caa96a8,85edf6_256c5e150bc4,ecc1fc_c9976ea24553,ecc1fc_21a72aa11cc4); unclosed rows 13 → 9 (8alerted/ 1detected); the ledger still tags every 503 rowfailure_type: unknown, so triage on error text. Single-node topology still open — and every job that failed readsmodel: smart-route,provider: custom:omni-101. -
incidents/2026-10-14-omniroute-node-connection-refused-storm — Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:
omniroute-1(100.78.63.20:20128, providercustom:omni-101) refused TCP connections, so every model-calling cron job failed on every slot — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. All 18 enabled model-calling jobs ride this one node with no failover. Recovered 10-15 22:01 (resolved) -
incidents/2026-10-09-cron-provider-timeout-storm — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all
RuntimeError: Request timed out.on the aliases behindcustom:omni-101, 00:00 → 15:00 AWST. CLOSED — all 12 incident rows are now resolved; the last two (1b23c0_c35703f637b8,1b23c0_103cbe04a79b, both on the weekly Friday job1b23c042e17d) closed 2026-10-23 08:04 when that slot ran clean, and the job readsokagain -
incidents/2026-10-09-memory-recall-relevance — Recall relevance: a stale conversational episode outranked current-state facts on an open question. ✅ RESOLVED 2026-11-01 — the ranking weights were persisted, in
~/.hermes/mnemosyne/config.yamlrather than the file the 2026-10-10 probe read; the original regression query now returns only current-state facts, and conflict152was auto-resolved on 2026-10-29 -
incidents/2026-10-05-dead-model-pin-repin — Seven cron jobs on dead model pins (
cliprox/terra,qwen3.5:397b,deepseek-v4-flash,gpt-5.6-terra) repinned to routing aliases oncustom:omni-101; failing set 7 → 0 (resolved 2026-10-05) -
incidents/2026-10-03-mnemosyne-vector-recall-fts-only — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
-
incidents/2026-10-03-ct450-hermes-update — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
-
incidents/2026-10-01-cliproxy-connection-auth-expired — OmniRoute
cliprox/terraconnection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); closed 2026-10-05 when all three were repinned off the pin -
incidents/2026-08-26-work-outlook-basic-auth-disabled — Work Outlook mailbox unreachable: tenant allows only
XOAuth2, himalaya configured for password (open since 2026-08-26) -
incidents/2026-09-28-hermes-managed-runtime-deps — Managed-runtime venvs missing
dotenv/mnemosynebroke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. Recurred 2026-10-10 inside the gateway runtime env —mnemosyneandelevenlabsmissing from the committed uv env the gateway loads; both repaired and verified the same evening -
incidents/2026-08-18-mnemosyne-capture-pipeline — Mnemosyne capture pipeline broken (Aug 2026, superseded): capture verified healthy 2026-10-12 — 786 working rows, +2 that day, 98.3 % embedded
-
incidents/2026-08-15-skills-index-drift — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration
Monitoring & Dashboards
- systems/homepage-dashboard — Homepage dashboard (CT201)
- systems/home-assistant — Home Assistant integration
Reports
- reports/uber-eats/index — Uber Eats monthly spend reports
- reports/communities-wa-social-media-audit-2026 — Department of Communities WA social media audit and first-day brief
- systems/obsidian — Obsidian vault structure and conventions
- queries/retained-research-answers — Retained research answers (fastest VNC clients, 2026-10-11)
Infrastructure
Services
- infrastructure/hosts — Hosts, containers, important paths
- infrastructure/docker-services — Docker compose files
- infrastructure/credentials-map — Where credentials live (no values)
- infrastructure/network-map — Zones, hosts, and ports across LAN, Tailscale, and the public tunnel
- infrastructure/domains-and-tunnels — martinwa.org hostnames and Cloudflare Tunnel routing
- infrastructure/backups — hermes backup CLI, retention, and gaps
Networking
- infrastructure/proxmox — PVE hosts (two:
pve9.2.10 +m93p9.2.10, unclustered, IDs collide), VMs, LXC containers, storage (/mnt/ssd076.43 % / ~169 GiB free, 2026-10-31) - infrastructure/tailscale — Tailscale serve routes (CT460 verified exactly) and the 50-node tailnet inventory (re-verified 2026-10-31:
mattermostback online after ~40 days; 7 offline ages refreshed) - systems/cloudflare-tunnel — ct207-hosted tunnel serving Hermes publicly
Concepts
- concepts/agent-memory — Memory model
- concepts/model-routing — Routing concepts
- concepts/persistent-shells — Terminal persistence
- concepts/browser-automation — Browser automation concepts
- concepts/tool-calling — Tool calling patterns
Decisions
- decisions/index — Decision index
- decisions/keep-openrouter-as-fallback — Keep OpenRouter as fallback provider
- decisions/persistent-shell-enabled — Enable shell persistence
Comparisons
- comparisons/camoufox-vs-chrome — Camoufox vs Chrome
- comparisons/hermes-vs-nanobot — Hermes vs Nanobot
- comparisons/memory-backends — Memory backend comparison
- comparisons/model-routers — Model router comparison
Runbooks
- runbooks/multiplexer-setup — Multi-profile gateway multiplexer setup
- runbooks/diagnose-docker — Docker diagnosis
- runbooks/recover-docker-service — Docker service recovery
- runbooks/restart-hermes — Hermes restart procedure
- runbooks/restart-browser — Browser restart procedure
- runbooks/update-hermes-safely — Safe Hermes update
- runbooks/restore-openrouter — OpenRouter restoration
- runbooks/provider-health — Provider health check
- runbooks/troubleshoot-tool-loop — Tool loop troubleshooting
- runbooks/gateway-resource-resilience — Gateway resilience
- runbooks/job-radar — Job Radar search, Notion, dashboard and recovery procedure (two-stage since 2026-10-27:
dca8482e4f76discovery →5fc292ff0f3bwrite and letters)
Raw
- raw/configs/hermes-config-sanitized.txt — Sanitized config