- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo - Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved) - Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB - Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification, decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree, honest AgentMail/Telegram delivery reporting - Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed - Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated - Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with 'lost its durable fire claim ownership' and left the tree dirty)
56 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Anthony's Systems Wiki | collection | active | 2026-07-22 | 2026-11-07 | 2026-11-07 | high |
|
|
Anthony's Systems Wiki
Operational knowledge for Anthony's agents, infrastructure, model routing and automation. Last updated: 2026-11-07.
2026-11-05 (stocktake) - the
systems/skills-index.mdflip-flop is fixed at the root: two different scripts were writing the page, and the nightly audit was pointed at the stale one; it is now a delegating wrapper -> incidents/2026-11-05-skills-index-generator-divergence. Docker inventory corrected against live probes on all five hosts -the⚠ CORRECTED 2026-11-06: false - the seeders are transient systemd-timer-driven containers, absent only between runs, not removed. Tailscale node table re-verified programmatically - 52 rows, 52 live peers,worldmonitor-seeder-scheduledcontainer recorded yesterday does not exist on any hostevery state matching⚠ CORRECTED 2026-11-06:anthonys-mac-minihas since come back online. Freshness 0 stale / 122 fresh / 3 no-date of 125.)2026-11-07 (nightly audit - window 2026-11-06) - a red row with no alert: 2026-11-06 settled 172 execution rows across 16 jobs - 170
completed/ 1failed/ 1running(this audit). The single failure is a new, non-restart signature: the Job Radar's 11:30 slot (5fc292ff0f3b, 11:30:32 -> 13:26:44) was discarded as stale onFire claim ownership lost; stale result was discarded.because adirectfire of the same job (08:27:09 -> 13:26:22, 4 h 59 m 13 s) was still holding the claim and delivered first. Every earlier occurrence of this string came from a restart/reload orphaning a run; this one had no restart -> incidents/2026-11-06-job-radar-concurrent-fire-claim-discard. It raised nocron_incidentsrow (newest ledger activity remains the 2026-11-05T12:36:17 closures) and nothing alerted; the failed run left no output file. The day still landed in full via the 14:30 slot (12 m 27 s,SAME_DAY_STRONG_COUNT 4,MISSING_COVER_LETTER_COUNT 0, cache 494, Run status: COMPLETE). Deliveries 10 rows / 10delivered/ 0for_failure; fleet 29 jobs (26 / 3) - fifth consecutive audit at the same count, nothing added/retired/disabled/repinned, non-cleanlast_statusstill 1 (066bb5191280). The stocktake's overrun fully unwound (1a8e90ea2271, 03:00:32 -> 04:11:35 = 1 h 11 m 03 s against 7 h 40 m / 5 h 41 m / 3 h 02 m on 11-03 -> 11-05) and Stage 1 ran clean for a second day (dca8482e4f76, 1 h 28 m 37 s,suppressed), clearing the 11-04 Open entry in substance. The last audit's open question closed itself: the Mnemosyne Health Monitor's 11-05 12:00:45 slot (12 h+ in flight, undelivered at the last audit) completed at 2026-11-06T00:14:39 after 12 h 13 m 54 s and delivered; its 11-06 slot took 7 h 47 m 56 s and delivered too - ballooned on consecutive days, never lost, watching, not open. The one open page remains incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts (untouched in-window).2026-11-06 (nightly audit - window 2026-11-05) - a clean ledger day and an empty incident ledger: 2026-11-05 settled 175 rows - 173
completed/ 2running/ 0failedacross 18 jobs, the first zero-failedday since 2026-11-02. The only tworunningrows are the Mnemosyne Health Monitor (12:00:45 slot, still in flight 12 h+ at audit time) and this audit itself. Unclosed incident rows 3 -> 0 - the Newsletter Digest row closed at 07:07:42 and both Job Radar - Discovery rows at 12:36:17, whendca8482e4f76finally completed after 5 h 35 m 35 s; that closes out incidents/2026-11-04-upstream-stream-failure-0700-cohort and incidents/2026-11-03-job-radar-stream-active-timeout in full. Fleet non-cleanlast_status3 -> 1 (only066bb5191280,delivery_failed, streak 0); deliveries 10 rows / 10delivered; the 03:00 stocktake overran 3 h 01 m 58 s (down from 7 h 40 m and 5 h 41 m) and committed2f3fc99at 05:55. Fleet unchanged at 29 jobs (26 / 3) - a fourth audit at the same count. ⚠ New and open: live.envvalues (TELEGRAM_BOT_TOKEN,MATTERMOST_TOKEN,HERMES_CUSTOM_OMNIROUTE_20128_API_KEY) sit verbatim in 17 rows ofmnemosyne.db(memories,episodic_memory,working_memory,gists,memoria_facts.context_snippetand the FTS mirrors) - reproduced in full by each of the 7 uncompressed rotation snapshots - and all four values sit verbatim in the 1.7 GB session storestate.dbtoo;CAMOFOX_API_KEYsits verbatim in the prompts of two enabled Facebook Birthday jobs (0e753e9720fd,fad5a86979d4) and therefore incron/deliveries.db-> incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze: the 11-04 fleet settled 170 rows - 167
completed/ 2failed/ 1running, non-cleanlast_status2 -> 3, unclosed incident rows 1 -> 3, deliveries 8 rows / 8delivered. Both failures are the same upstream fault:dca8482e4f76(Daily Marketing Job Radar - Discovery) and85edf6772bca(Tech & AI Newsletter Digest (Analytical)) died 36 seconds apart at 09:41-09:42, each after 2 h 41 m, onRuntimeError: Upstream stream failed before completion.- both pinnedcustom:omni-101/smart-route. The day was not lost: the 11:30 Stage 2 slot reconstructed the missing handoff from Stage 1's own candidate files, re-verified every candidate live, and still wrote 5 roles (board 479 -> 484) + 1 canonical cover letter (MISSING_COVER_LETTER_COUNT 0) before consuming the handoff at 16:08:59 -> incidents/2026-11-04-upstream-stream-failure-0700-cohort. The cover-letter route survived only on an override:regenerate_cover_letter.pydefaults todeepseek-paid-smart, which threw HTTP 402 mid-run, so the letter was pinned togemini/gemini-3.5-flashviaCL_MODEL- and that override was never written to.env, so it did not persist. And CT299 was frozen twice, not once: a second, manualvzdump 299 ... --storage ssd0wedged from 11-04 08:30:04 until 11-05 ~00:30 (~16 h); the live probe at 00:54:21 AWST showsLOCK: 0, bothcgroup.freezeflags0, guestALIVE,sftpgo: active-> incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage. ⚠ Corrected the same night: the first draft of this entry read the fleet as 158 jobs (144 / 14) and the 11-04 ledger as closing1 item_suppressed; live reads give 29 jobs (26 / 3) and a final row of 1running(this audit). Keyed probe 2026-11-05 01:33:deepseek-paid-smartis undispatchable onomniroute:20128(HTTP 503, all targets skipped) but serves normally onomniroute-1(HTTP 200 ->deepseek/deepseek-v4-flash).
2026-11-04 (nightly audit) — the ledger raised its own red row: the **11-03 fleet settled 170
completed/ 1failed/ 1running/ 0unknown(172 rows) — unclosed incident rows 0 → 1, non-cleanlast_status1 → 2, deliveries 7 rows / 7delivered. The single failure is the Job Radar Stage 1 (dca8482e4f76), which ran 07:00:33 → 13:28:52 (6 h 28 m 18 s) and died onRuntimeError: stream active timeout— a new terminal signature, not the iteration cap — after going silent for 65 m 21 s; its incidentdca848_dfd4bea84f2a(timeout) isalertedand open and the notice was delivered (13:28:51.741724) → incidents/2026-11-03-job-radar-stream-active-timeout. That answers the 11-02 page's open question: output is suppressed when the run endscompleted, delivered when it endsfailed— which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so 2026-11-02's and 2026-11-03's candidate sets are both unrecovered. ⚠ The 03:00 stocktake overran for a second day (1a8e90ea2271, 03:00:30 → 10:41:13 = 7 h 40 m 43 s,suppressed, nothing delivered, artefacts uncommitted again). New page: systems/hermes-hq-push (v0.9.27, enabled; manifest says it replacesdispatch-push, and no trace of the old plugin survives).⚠ Two Proxmox hosts exist — always qualify a guest with its host. A second unclustered hypervisor,
m93p(100.104.51.98, PVE 9.2.10), was discovered during the 2026-10-14 stocktake. Guest IDs collide between the two hosts.
Start here
- current-state — Current systems status
- systems/hermes-agent — Hermes Agent overview and version
- systems/current-profile — Active Hermes profile details
Systems
Core
- systems/hermes-agent — Agent version, install, config paths
- systems/current-profile — Active profile details
- systems/nextcloud — Nextcloud file sharing (CT270)
- systems/guanaco — Guanaco LLM proxy on CT205 (Ollama stack)
- systems/model-providers — LiteLLM + OmniRoute providers, routing setup
- systems/memory-backend — Mnemosyne vector memory, settings
- systems/mnemosyne — Mnemosyne cron jobs and config evidence
- systems/omniroute — OmniRoute catalog, combos, and host path
- systems/openrouter — OpenRouter current state and history
- systems/opencode-go — OpenCode Go subscription, model limits, pricing
- systems/nanobot — Nanobot current state and history
- systems/browser-backend — Camofox (primary,
systemd --useron CT460, REST:9377) — the only backend; Steel was fully removed from the stack on 2026-10-29 (plugin deleted,cloud_providerback tocamofox, CT201 and CT333 deployments torn down, Serve routes deleted;STEEL_BASE_URLsurvives only in already-running process environments); memory-containment drop-in and the 2026-10-12 ceiling event documented - systems/hermes-ui — Browser Hermes Desktop (przbadu/hermes-ui)
- systems/paseo — Paseo coding-agent daemon + web terminal workspace (CT460, :6767)
- systems/ignis — Ignis browser-native Obsidian (wiki interface)
- systems/terminal-backend — Shell persistence, working directory
- systems/tool-search — Always-on lazy discovery for external tools
Integrations
- systems/conduit — Conduit: Hermes on Anthony's phone (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree
~/.hermes/plugins/conduit_push, v0.122.2(HEAD9eac857, 2026-10-31 12:04 — "Every profile that turns Conduit on runs the main install (#482)"; +148 commits on since the0.69.2/942ddb8baseline, working tree clean), upstreamkaishi00/hermes-conduit-notifier. Gateway on:9119is now pid 176771, started 2026-10-31 19:35:02 (superseded: restarted 2026-10-25 18:42:33, which is when the upstream commits went live). One registered device (iPhone app 2.2.0). ✅ The first two real calls in the product's history are now on record — 2026-10-29 08:38:04 and 08:49:36 — after three faults in series were repaired (truncated speech-server URL →…/v1/realtime; 12 h dashboard session TTL → 30 d; deadgroqSTT →local, deadopenaiTTS →ollama_local) → incidents/2026-10-29-conduit-voice-silence. So the memory fix is no longer evidence-free. ✅ That patch is gone — the tree is clean andMEMORY_TIMEOUT_Sships upstream indashboard/api/memory.py. ⚠ New fault: "local" voice mode never applies — the Mac mini speech server's LLM proxy is disabled (LLM endpoint → 501,/v1/usagereadsllm_proxy.requests: 0, realtime socket rejectssession.update), while the app's "test connection" checks socket reachability only → incidents/2026-10-31-conduit-voice-local-mode-not-applying. The call store was re-shaped:historyis empty andlast_request_atisnull, so the two 10-29 calls are no longer in it. The 10-24 refusal fault has not recurred → incidents/2026-10-24-conduit-call-request-refused. - systems/messaging-integrations — Telegram, Discord, email, Slack, etc. Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26 (peer offline 36 d,
tx 312 / rx 0; host answers neither:8065nor ICMP). 268 connect timeouts on 10-25 vs 264 on 10-24; the reconnect attempt counter reset to 54 at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost. - systems/push-notifications — Cadu/Vory push bridge (
hermes-push,vory-push): gateway on:9119, state in~/.hermes/push/. Broken —connected: false(the bridge dials a name that resolves to127.0.1.1where nothing serves 443); chronic, not a blip. - systems/hermes-hq-push — Hermes HQ iPhone app plugin (standalone, v0.9.27,
kind: standalone, author Hermes HQ; tree mtime 2026-11-03 11:06).hermes plugins list→ enabled (stateuser);config.yamlplugins.enabledline 7556. Host side of the Hermes HQ app: Apple push (finished turns, lock-screen approvals, "waiting for a password/login/code" alerts, live Lock Screen cards sealed to the phone), phone device access (contacts, location, reminders, alarms, photos, Health, motion, music), the desktop-app Hermes HQ theme, the app's Terminal page, fast Sessions list on a busy host, cross-posting answers back to Telegram/Discord, a restart guard, and Grok Live voice-token minting (XAI_API_KEYor subscription sign-in; bearer never reaches the phone). Toolsdevice_contacts_search/device_request_result; 7 hooks. Its manifest states it replacesdispatch-push— and the replacement is clean: nodispatch-pushreference survives inconfig.yamland no such directory remains. Different stack from the Cadu/Vory bridge on systems/push-notifications. ⚠ Itspre_tool_callguard also blocks any tool call whose arguments contain the literal text of a launchd one-off-restart command — describe it, never quote it. - systems/scheduled-tasks — As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean
last_status4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), andea79a2107e36is ENABLED again (last_status: ok, streak 0) after its 00:01:08 slot closedcompletedat 10:02:48; deliveries 8/8delivered. Prior window — as of 2026-10-30 03:3x: 29 jobs (25 enabled, 4 disabled); non-cleanlast_status3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries alldelivered. ⚠ea79a2107e36(Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closedunknownon a gateway-connection failure — and its failure notice was NEVER DELIVERED (deliveries.dbhas zero rows for 2026-10-30), so the incident row is the only trace → incidents/2026-10-30-mnemosyne-health-monitor-paused. Every other previously-red job has cleared took/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-cleanlast_status5 → 3 (all three are stale re-runs of old faults and everyfailure_streakread 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries alldelivered. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters withmissing_cover_letter_count 0(Notion 455 → 475). Both stage-1 incident rows closedresolvedat 10:16:12; the lone survivor is the Facebook monthly's0e753e_e7f8194a7d06.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.**dca8482e4f76ran 07:01:08 → 11:03:54 (4 h 2 m 46 s), closedfailed, and delivered "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery finished, no Notion writes and no cover letters. Incident rowdca848_e93a538ade6bopened 11:03:53 and is still open. The same day the single radar job was split into two: stage 1dca8482e4f76(Discovery, 07:00,deliver: local) + new stage 25fc292ff0f3b(Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, twocompletedtest fires that evening). The two incident rows the 2026-10-26 planned restart raised auto-closed 00:38:32, and the 2026-10-26 orphan class did not recur. → incidents/2026-10-27-job-radar-budget-exhausted, runbooks/job-radar. Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-cleanlast_status, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.ea79a2107e36(Mnemosyne Health Monitor, 12:01 slot) was interrupted by a planned 14:33:35SIGTERMrestart that timed the drain out after 1.35 s, killed after 2 h 34 m on a new string (lost its durable fire claim ownership), and raised twocron_incidentsrows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → incidents/2026-10-26-planned-restart-orphaned-cron-run. New fleet precedent: a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). Unclosed incident rows 2 → 3; one closed:c0a9fa_764558b5ae83auto-closed 09:13:00 when the radar's 09:00 slot ran clean, so every row of the 2026-10-16 503 resource-pressure incident is now closed → incidents/2026-10-16-omniroute-503-resource-pressure. Non-cleanlast_status5 → 4:c0a9fa314542and08eb86f02d37cleared took;ea79a2107e36joins aserror, streak 2. Pins unchanged. ⚠fad5a86979d4's 03:08 slot completed but delivereddelivery_failed— Thread 34795 not found (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered “RUN INCOMPLETE — tool-call budget exhausted” and its cover-letter pipeline changed that morning → runbooks/job-radar. Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-cleanlast_status, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0. ⚠ Both 10-25 failures are08eb86f02d37(Gmail Action Digest), which also fired six times that day — and neither raised an incident row → incidents/2026-10-25-gmail-digest-runaway-and-refire. ⚠ The global default reverted tocustom:omni-101/smart-routeon 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again → systems/current-profile. ⚠ Nous Portal credentials still invalid — all 8auxiliaryslots moved tono-think/freeapi/autoat 19:13 → incidents/2026-10-22-nous-portal-refresh-token-invalid. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-cleanlast_status, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third consecutive clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0. Zero incident rows opened or closed in-window; pins unchanged. ⚠ Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls → incidents/2026-10-24-conduit-call-request-refused. Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-cleanlast_status, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four. The 2026-10-09 provider timeout storm is now fully closed:1b23c042e17dran clean on its 10-23 08:04 slot, closing the storm's last two rows, so all 12 are closed and unclosed incident rows fell 4 → 2. ⚠ New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (CONDUIT_MEMORY_TIMEOUT_S) + degraded-not-aborted onTokenError; dashboard restarted 18:54:43, no abort since. Patch uncommitted → incidents/2026-10-22-conduit-memory-start-timeout, systems/conduit. The 10-22 stocktake overrun did not repeat (10-23 03:07 committed250c735). Prior window: As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-cleanlast_status, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.dca8482e4f76(Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s onRuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s→ incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.08eb86f02d37cleared (08:30 slot clean; incident resolved 08:37:21) and 2026-10-20 settled at 169 / 0 + 1unknown. ⚠ Hermes Agent was updated to0.22.3at 22:47 → 22:57 with its pre-update snapshotskipped→ systems/hermes-agent. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. Theomni-101503 fault did not recur (back tomonitoring) and 4 incident rows closed → 9 unclosed. ⚠ Two new findings: the Gitea config backup has not pushed since 2026-08-08 (75 unpushed commits) while its row readsok→ incidents/2026-08-08-gitea-config-backup-push-dead; and a gateway unclean exit on 2026-10-20 orphaned a digest run — no delivery row, no incident → incidents/2026-10-20-gateway-unclean-exit-orphaned-run. Also: the global model default moved tonousdirect (meituan/longcat-2.5-preview:free) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while 20 of 28 still pin it → systems/current-profile. Prior window: 28 cron jobs (25 enabled, 3 disabled); 12 carry a non-cleanlast_statusas of 2026-10-20 00:09 — up from 7, because theomniroute-1503 resource-pressure fault returned on 2026-10-19 (second burst in four days: 9 model calls across 8 jobs, 06:01 and 07:01 → 12:01, allmodel: smart-routeoncustom:omni-101, while/api/healthstayed200 ok) → incidents/2026-10-16-omniroute-503-resource-pressure. 2026-10-19 closed 159 / 13 against 10-18's 168 / 1. The Mosquito Tracker was removed (29 → 28 jobs), and the 2026-10-18 degenerate-stream placeholder-delivery mode remains an open question → incidents/2026-10-18-degenerate-stream-canned-substitute. Earlier: the connection-refused storm (2026-10-14 → 10-15) — 24 model-call failures across 12 jobs, ~34 h of digests blanked. All 18 model-calling jobs still sit oncustom:omni-101with no failover. - systems/skills-index — Installed skills inventory (302 skills (68 categories) as of the 2026-11-05 stocktake; regenerated daily by the 03:00 job). The generator excludes
.archive/(46 archived skills) — the earlier 346/347 counts included it. Freshness at the 2026-11-07 run: 0 stale / 125 fresh / 3 no-date of 128 pages (freshness-20261107.txt). Note:SKILL.mdfiles on disk still exceed the index by one (303 vs 302) — the generator de-dups one nested duplicate, so 302 is the correct unique-skill figure; see systems/scheduled-tasks. ✅ 2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root: two different scripts were writing this page (the repo script → 302/68; the stale~/.hermes/scripts/wiki-maintain.shcopy → 349/26, because it usedPath.rglob()and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a delegating wrapper with no generator logic of its own → incidents/2026-11-05-skills-index-generator-divergence. ⚠ The 2026-10-22 slot ran 97 m 58 s (vs 16 m 01 s on 10-21), finishedcompletedwith deliverysuppressedand committed nothing; the 10-23 slot closed that hole.
Incidents
-
incidents/2026-11-06-job-radar-concurrent-fire-claim-discard - OPEN. A
direct-source fire of5fc292ff0f3b(08:27:09 -> 13:26:22, 4 h 59 m 13 s, delivered) settled the claim while the scheduler's own 11:30:32 slot was still running, so that slot was interrupted at 13:26:38 and closedfailedat 13:26:44 onFire claim ownership lost; stale result was discarded.- a new string variant and the first claim-loss failure with no gateway restart involved. It raised nocron_incidentsrow, so afailedexecution with no alert was the day's only red row; the discarded run left no output file. The 14:30 slot recovered the day in a normal 12 m 27 s cycle. Candidate fixes (slot suppression when a claim is live, consuming the next slot for out-of-band fires, raising an incident row on discard) are listed on the page and none was applied. -
incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts - OPEN. Live
.envvalues were found stored verbatim in 17 rows ofmnemosyne.db(memories4,episodic_memory3,working_memory1,gists4,memoria_facts.context_snippet1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the 7 uncompressed rotation snapshots, and all four values also sit verbatim in the 1.7 GB session storestate.db, which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two enabled Facebook Birthday jobs (0e753e9720fd,fad5a86979d4) and hence incron/deliveries.db. Nothing rotated, deleted or scrubbed; no value recorded. -
incidents/2026-11-04-upstream-stream-failure-0700-cohort -
dca8482e4f76+85edf6772bcadied 36 s apart onRuntimeError: Upstream stream failed before completion.after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter). -
incidents/2026-11-05-skills-index-generator-divergence — RESOLVED — the skills index flip-flop had two writers. The repo
wiki-maintain.sh(302 skills / 68 categories) and the stale~/.hermes/scripts/wiki-maintain.shcopy (349 / 26, because it usedPath.rglob()and counted the parked.archive/) were different scripts, and SKILL.md pointed the 23:59 nightly audit at the stale one — so each job overwrote the other's page and the wiki oscillated nightly with no error raised by either. The stale copy is now a delegating wrapper; the repo script gained--no-buildso the wrapper honours the old interface without a second generator. -
incidents/2026-10-31-conduit-voice-local-mode-not-applying — OPEN — the app's "local" voice mode never applies, and the connection test lies. The test checks socket reachability only, while every local turn is still impossible: the Mac mini speech server's LLM proxy is disabled (LLM endpoint → HTTP 501 "Start the server with
--enable_llm_proxy"),/v1/usagereadsllm_proxy.requests: 0, and its realtime socket accepts the connection but rejectssession.updateas an unknown event.voice.modeis a phone-side selector — nomodekey exists inconfig.yaml → voiceorconduit-settings.json— so a "mode did not apply" cannot be diagnosed on host config. Found via session20261031_193709_40b9c2(19:37:09, 30 msgs / 17 tool calls);voice.local_live.front_modelwas repointed tomistral/voxtral-small-latestduring it and nothing was changed on the Mac. → systems/conduit -
incidents/2026-10-31-gateway-reload-force-drain-orphaned-run — A gateway reload waited its full 1,800 s drain cap, then force-drained and orphaned a live cron run. At 18:06:41 the gateway took the reload; the drain reported
timed_out=Trueat the cap and force-drained at 18:37:38 (new gateway 18:38:43).ea79a2107e36(Mnemosyne Health Monitor, 12:01:06 slot, 6 h 38 m 09 s in) closedfailedat 18:39:15 on "lost its durable fire claim ownership". Same class as incidents/2026-10-26-planned-restart-orphaned-cron-run — second instance, and the first on a plain reload. A second, clean restart followed at 19:23:39 → 19:24:40; 2026-10-31 closed 160 / 4. → systems/scheduled-tasks, systems/hermes-agent -
incidents/2026-10-31-asana-access-failures — Both Asana paths on CT460 are non-functional. The MCP server
asana(https://mcp.asana.com/v2/mcp,auth: oauth,enabled: true) has never authenticated —~/.hermes/mcp-tokens/holds onlyasana.client.json(319 B, mtime 2026-10-26 01:07) and the logs carry "MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Runhermes mcp login asanainteractively first"; the hosted connector also rejects the call shape. The fix needs an interactivehermes mcp login asana, which a cron context cannot perform. → systems/hermes-agent -
incidents/2026-10-30-ct460-host-outage-scheduler-gap — RESOLVED (root cause not established). CT460 went dark for 5 h 30 m on 2026-10-30: the PVE host
pverebooted (boot 16:24:00), CT460 returned 16:28:00, the previous gateway life (pid=3913841) was logged as exited UNCLEANLY at 16:35:50 ("no exit path ran — SIGKILL / OOM / VM death"), and the scheduler resumed at 16:37:21 with 8 catch-up runs. Zero execution rows exist between 10:58:20 and 16:37:21; the day still closed 138 / 0 against 10-29's 177 / 0 (≈39 slots lost, no failures). No monitor alerted — the two jobs that could have (cbfe434a3b5e,1a8e90ea2271) are disabled. -
incidents/2026-10-30-mnemosyne-health-monitor-paused — RESOLVED 2026-10-30T10:02:48 (its 00:01:08 slot was still running, not dead: it closed
completedafter 10 h 01 m 40 s and delivered, auto-closingea79a2_d93c3acdc4f2;ea79a2107e36is enabled again withlast_status: ok/ streak 0). The Mnemosyne Health Monitor cron job was auto-paused overnight and its failure notice never reached Telegram — the only trace is the incident row.ea79a2107e36's 2026-10-30 00:01:08 slot closedunknownat 01:04:22 on "Cron execution unverified; reconcile~/.hermes/cron/admissions/ad897cbc9f…json: Gateway connection failed; no local fallback", opening incidentea79a2_d93c3acdc4f2(failure_type: unknown,detected) — and the scheduler paused the job at that moment. Fleet is now 29 jobs (25 enabled / 4 disabled); livelast_status: error,failure_streak: 1. ⚠ Two traps in one fault: the failure notice itself failed to send (send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed—deliveries.dbhas zero rows for 2026-10-30), and the run closedunknownrather thanfailed. The job was not re-enabled and the gateway was not restarted by the stocktake — root cause is unverified. → systems/scheduled-tasks, systems/mnemosyne -
incidents/2026-10-29-conduit-voice-silence — Every Conduit voice call connected and then died silently; three faults sat in series, and the first two real calls in the product's history were placed on 2026-10-29 after all three were fixed. (A)
voice.speech_server.urlwas the truncatedwss://…:8443/v1/— the plugin dials it verbatim and the bare path answers 403, reported as "the speech server refused the key." (B)dashboard.basic_auth.session_ttl_secondswas the 12 h default, so an expired phone session gets WS 4401 with no Hermes turn and no error on the device (the phone re-authenticated at 08:00:59, minutes after a silent call). (C) Both host providers were dead —tts.provider: openai→ Mac/v1/speech502,stt.provider: groq→ 403/Cloudflare 1010 (blocked, not mis-keyed). Fixed 08:02–08:4x: URL →…/v1/realtime, TTL → 2 592 000 s (30 d),stt.provider: local(faster-whisperbase),tts.provider: ollama_local; dashboard restarted 08:03:26. Verified by driving the whole call path for real, then by calls at 08:38:04 and 08:49:36. ⚠ Two TTL keys exist —dashboard.basic_auth.session_ttl_seconds(raised) vsdashboard.session_ttl_seconds(still 43 200).status: resolved. -
incidents/2026-10-28-job-radar-stage1-iteration-cap — The radar's two-stage split ran for real on 2026-10-28 and landed nothing: stage 1 stopped at the agent iteration cap before writing its handoff, so both stage-2 slots correctly no-oped (silent). Two consecutive stage-1 failures, two different ceilings (tool-call budget 10-27, iteration cap 10-28); no Notion writes for a second day. RESOLVED 2026-10-29 — the next stage-1 run (
dca8482e4f76, 07:00:59 → 10:16:12, 3 h 15 m 13 s) closedcompletedand wrotehandoff/2026-10-29.json, which stage 2 consumed at 10:40:59 and delivered as 20 pages + 2 cover letters (missing_cover_letter_count 0).status: resolved, review 2026-11-13. -
incidents/2026-10-27-job-radar-budget-exhausted — The Daily Marketing Job Radar exhausted its tool-call budget for the second day running — and this time the run is recorded as
failed, notcompleted. On 2026-10-27 the 07:00 slot ran 07:01:08 → 11:03:54 (4 h 2 m 46 s) and its delivered report opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)": discovery/verify/dedup/classify finished, Notion writes, cover letters and the final audit never ran (Notion DB held 451 entries, 0 added for the day). Incident rowdca848_e93a538ade6bopened 11:03:53 (alerted11:03:59,failure_type: auth— wrong for a budget fault) and was still open at the 2026-10-28 audit. Response the same day: the radar was split into stage 1dca8482e4f76(Discovery, 07:00,deliver: local) and new stage 25fc292ff0f3b(Write and Letters, 11:30/14:30, topic 1030), created 17:34:15, twocompletedtest fires that evening → runbooks/job-radar. RESOLVED 2026-10-29: both of this incident's ledger rows (dca848_e93a538ade6b,dca848_0a2f03b5952b) closedresolvedat 10:16:12 on the first successful post-split run —dca8482e4f76restored delivery tolocal, wrote the handoff inside budget, and no cover letter went missing (missing_cover_letter_count 0).status: resolved, review 2026-11-13. -
incidents/2026-10-26-planned-restart-orphaned-cron-run — A planned gateway restart orphaned a live cron run — the second instance of the orphan class, and the first that alerted. On 2026-10-26 at 14:33:35 AWST the gateway took a planned
SIGTERM(under_systemd=yes); the shutdown drain reportedtimed_out=Trueafter just 1.35 s of its 30 s budget and interruptedea79a2107e36(Mnemosyne Health Monitor, 12:01 slot) 2 h 34 m in. It closedfailedon a new string —lost its durable fire claim ownership— raised twocron_incidentsrows (14:33:37 unverified / reconcile admission + 14:35:59 fire claim), delivered the failure at 14:35:59, and left the job onfailure_streak: 2from one interrupted fire. Two Mnemosyne Watchdog reds followed at 14:46 and 15:01 and self-resolved at 15:16:12. Net = 169 / 3 for 10-26. Unresolved: why a planned stop bypasses the drain deadline, and whetherfailure_streakshould count the unverified row. → also updates incidents/2026-10-20-gateway-unclean-exit-orphaned-run (the class now recurs without a crash). -
incidents/2026-10-25-gmail-digest-runaway-and-refire — The Gmail Action Digest ran 1 h 55 m and fired six times on 2026-10-25. The 08:30 slot started 08:31:14 and closed 10:26:37
failedonRuntimeError: Operation interrupted: waiting for model response (347.2s elapsed)— against 91 s (10-23) and 104 s (10-24). The same job then fired five more times (17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53); the 19:18:41 fire died on a new signature —Cron admission missing: the owner never admitted this fire(distinct from the 10-20 orphan) — and all six fires, failures included, carrydelivery_outcome: delivered, so up to six digests reached Anthony. Neither failure raised acron_incidentsrow; the job now readslast_status: delivery_queuedwith an emptylast_error. Hypothesis (unconfirmed): the 08:31 fire adopted an unrelated long-lived session — its worker made 70terminalcalls on VM101/dev/sdbbackup work with context growing 66 k → 148 k tokens, while context compression was unavailable all day (invalid_grant). → systems/scheduled-tasks, incidents/2026-10-22-nous-portal-refresh-token-invalid -
incidents/2026-10-22-nous-portal-refresh-token-invalid —
Invalid refresh token— Nous Portal runtime credentials are dead, so context summarisation failed. Gateway-log counts: 1 line 2026-10-22, 3 on 10-24, 20 on 10-25 (last 14:53:04,agent.context_compressor: Failed to generate context summary), 0 since the 19:27:13 restart. ⚠ Masked, not fixed: at 2026-10-25 19:13 all eightauxiliary:slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) moved offnoustono-think/freeapi/autooncustom:omni-101(fallbackno-think/freeapi/deepseek-v4-flash→no-think/kg/kilo-auto/free,free_only: true; compression timeout 90 s, vision 180 s). Nohermes modelre-authentication has been run andnousis still the first entry infallback_providers, so failover tries a dead credential. → systems/model-providers, systems/current-profile -
incidents/2026-10-24-conduit-call-request-refused — The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56). He asked to be called to test Conduit calls (
Call me now to test calls.→Yes. You do→CAL ME); all three replies asserted the agent has no calling capability, with zero tool calls in the session.conduit_call_user(toolsetconduit) exists, is registered for every session, is reachable viatool_search, and the plugin ships thecalling-the-userskill; his call settings (enabled/when_asked/decides) were permissive and the device was paired. Root cause: capability asserted instead of discovered — the first reply came fromkilo-auto/free, but the switch tosmart-routerepeated it. Same audit also corrects the 10-22 incident's closing evidence:conduit-calls.json → endsrecords turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne cron sessions —history(real calls) is empty and no call has been placed since (superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → incidents/2026-10-29-conduit-voice-silence) the memory fix. → systems/conduit, incidents/2026-10-22-conduit-memory-start-timeout -
incidents/2026-10-22-conduit-memory-start-timeout — Conduit voice calls were silently running without memory, 2026-10-22 → 10-23. The phone plugin gave one request — including the Mnemosyne provider start — a hard-coded 10 s budget (
MEMORY_TIMEOUT_S), while Mnemosyne's cold start for aconduit-voicesession runs 10–45 s. Three calls were abandoned at the memory layer (Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with no recall of prior conversation. Fixed 10-23 in the plugin's working tree: default 60 s (CONDUIT_MEMORY_TIMEOUT_Soverride) plus aTokenErrorguard that logs and proceeds without recall instead of aborting; dashboard restarted 18:54:43 and no abort line exists after it. ⚠ Corrected 2026-10-25: the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a false positive — those were the two Mnemosyne cron turn-ends of that night, andconduit-calls.json → history(real calls) is empty, so the fix has no positive call evidence and no call has been placed since (superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → incidents/2026-10-29-conduit-voice-silence). The patch is uncommitted — a plugin update reverts it. The cold start itself was not optimised. → systems/conduit, systems/mnemosyne -
incidents/2026-10-21-omniroute-pre-dispatch-filter-skip — A third failure signature from the
custom:omni-101/smart-routepin, 2026-10-21. The Daily Marketing Job Radar (dca8482e4f76, 07:00 weekday) died at 07:40:48 after 40 m 36 s onRuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s— no 503, no connection error, no repetition loop, and zero 503 rows exist for that day (last 2026-10-19 09:02). Its 07:00 sibling85edf6772bcacompleted normally, so the node was serving; the run wrote no output (cron_incomplete_no_output, 82 msgs / 47 tool calls). Incidentdca848_014affeb6954was closed 2026-10-22 07:59:22 when the next daily slot ran clean. Not caused by the 22:47hermes update— that landed 15 h later. → systems/omniroute -
incidents/2026-10-20-gateway-unclean-exit-orphaned-run — The gateway exited uncleanly on 2026-10-20 (09:09:01,
suspected_oom=False) and orphaned a live cron run.08eb86f02d37(Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closedunknown— “Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.” No delivery row exists for the slot and no incident row was created; the job table keeps showing the previous slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a 27-day clean run. Open: orphan-recovery on restart, and treating a delivery-targetedunknownas a probable non-delivery. -
incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence — Third iteration-cap in seven days, and the first that raised no incident at all. Job Radar Stage 1 (
dca8482e4f76) ran 07:00:33 → 16:01:52 = 9 h 01 m 19 s on 2026-11-02, hit the agent iteration cap before the handoff was written, and closedstatus: completed/error: NULL/delivery_outcome: suppressed. Both Stage 2 slots declined correctly (92665337…11:30 and the05a3f00b…14:30 safety net both exited[SILENT]), so nothing reached Notion, no cover letters were generated, and the day's 164 deduped candidates were never persisted — a state astatus-only health check reports as green. Contrast 10-27 (dca848_e93a538ade6b) and 10-28 (dca848_0a2f03b5952b), where the identical cap did open rows whoseerrortext is the run's own INCOMPLETE report: on 11-02 the ledger lost the red row it used to produce. Discovery itself worked — SEEK 51, LinkedIn 60 IDs, Indeed live, WA Government 35 (197 raw / 164 new after three-level dedup) — and a URL-canonicalisation bug was found and fixed mid-run (au.seek.comvsau.seek.com.auhad been dropping SEEK and Indeed entries from the consolidated list). → systems/scheduled-tasks -
incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure — The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered: its execution closed
unknownat 18:05:13 with "Cron execution unverified … Gateway connection failed; no local fallback", its incident row is stillalerted, the job sits aterror/failure_streak: 1, anddeliveries.dbshows no row after 11:01:12 for the rest of the day. Its batch-mateb63ed5817212took the identical error and self-healed in 68 s. Same page records the inverse silent failure:ecc1fc13655areadserrorwithlast_run 11-01T12:01:00while owning no execution row and no incident row at all. ✅ RESOLVED 2026-11-02T01:27:21 — the review was never lost. The same execution row (be6ec11a8e68428084af9ab18fb148db) closedcompleted/deliveredat 01:27:22 after 7 h 26 m 43 s, the delivery lands at 01:27:21.802246 and incident79f9bb_9c443b602eecauto-closed at 01:27:21.606691 — so the 18:05:13unknownwas a reconciler timing artefact on a still-running job, not a verdict (page nowresolved). Batch-mateb63ed5closed in 68 s and79f9bbin 7 h 26 m, bothcompleted; anunknownrow on a long job is a timing artefact, not a failure.ecc1fc13655a's ghosterrorcleared too (12:00:33 →completed/delivered12:29:26), taking non-cleanlast_status2 → 1. -
incidents/2026-11-01-cron-worker-sigterm-orphaned-claims — Three restart-safe cron workers killed by
SIGTERM(-15) after adopting their claim, so the scheduler recorded "whether side effects ran is unknown":85edf6772bca@07:30:46,96eb95ce12f2+0e753e9720fd@09:02:18. A fourth variant of the orphan class — and the first with no gateway restart, reload or drain anywhere in the window (PID175269up since 10-31 19:24:40; relay counter monotonic through both signals), so it cannot be attributed to reload behaviour. Likely trigger: a 30 s worker-ack window smaller than this host's cold start (83 plugins / 69 enabled discovered in 70 s). All three rows nowresolved; one execution leftunknown. -
incidents/2026-11-03-job-radar-stream-active-timeout — The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the ledger went red.
dca8482e4f76ran 07:00:33.819190 → 13:28:52.305995 = 6 h 28 m 18 s on 2026-11-03 and closedfailed/RuntimeError: stream active timeout, with the failure notice delivered (13:28:51.741724) and incidentdca848_dfd4bea84f2a(failure_type: timeout,alerted, open) — unclosed rows 0 → 1. The run's activity stops at 12:23:28; the harness does not declare the turn dead until 13:28:49 — a 65 m 21 s silence with no tool call, no model turn and nothing in the ledger, and no "maximum number of tool-calling iterations" message anywhere (this is a provider-stream timeout, not the cap that ended 10-27 / 10-28 / 11-02). It compacted five times (08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21) in 363 messages. Both Stage 2 slots declined correctly (5fc292ff0f3b11:30:36 → 11:32:16; 14:30:32 → 14:33:11) because no handoff was written, andradar-ref/handoff/still holds only 10-27 / 10-29 / 10-30 — so 11-02's 164 deduped candidates and 11-03's discovery are both unrecovered. Pulse carried it to Anthony in the desktop session. Answers the 11-02 page's open question 3: suppression tracks the terminal state —completed→ nothing delivered,failed→ notice delivered. → incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence, systems/scheduled-tasks -
incidents/2026-08-08-gitea-config-backup-push-dead — Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.
e27da6c34bbbreadslast_status: okand its executions readcompletedevery night while 75 commits sit unpushed (fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'— HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (message_id 8447, 2026-10-20 11:00); AgentMail failed. No off-box~/.hermesbackup in 2.5 months. Second instance of the pattern after the GitHub Trending Radar token revocation. → infrastructure/backups -
incidents/2026-10-18-degenerate-stream-canned-substitute — The repetition-loop failure mode, 2026-10-18: the FreshRSS Midday run (
ecc1fc13655a) wrote "Your request was not processed…", delivered it to topic:346at 12:08:11, then died at 12:08:25 onRuntimeError: Model output entered a repetition loop and the stream was cut mid-loop. A delivered message is not evidence of delivered content — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? Second surface (2026-10-20 18:37): an interactivedesktopsession received the same canned line afterRebuilt-message restart limit (3) exceeded→reason=rebuilt_restart_limit_exceeded … response_len=0, so the substitute is emitted by the harness's failure path generally, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — was never answered. -
incidents/2026-10-16-omniroute-503-resource-pressure —
omniroute-1load-sheds model calls withHTTP 503 … resource pressure— two bursts: 2026-10-16 → 10-17, and again 2026-10-19. Burst 1: 9 model calls across 7 jobs over ~19 h, self-cleared (reported then as10-17 = 162 / 5; the DB reads 163 / 5 — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: 9 model calls across 8 jobs on 2026-10-19 (06:01 + 07:01 → 12:01, whole day 159 / 13)./api/healthread200 okthroughout both. Status:active→ back tomonitoringon 2026-10-21 — 2026-10-20 ran 168 / 0 with zero rows carrying the resource-pressure string, so the second burst is over; it stays a standing risk until the single-node topology changes. 4 rows closed on 10-20 (dca848_867d9caa96a8,85edf6_256c5e150bc4,ecc1fc_c9976ea24553,ecc1fc_21a72aa11cc4); unclosed rows 13 → 9 (8alerted/ 1detected); the ledger still tags every 503 rowfailure_type: unknown, so triage on error text. Single-node topology still open — and every job that failed readsmodel: smart-route,provider: custom:omni-101. -
incidents/2026-10-14-omniroute-node-connection-refused-storm — Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:
omniroute-1(100.78.63.20:20128, providercustom:omni-101) refused TCP connections, so every model-calling cron job failed on every slot — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. All 18 enabled model-calling jobs ride this one node with no failover. Recovered 10-15 22:01 (resolved) -
incidents/2026-10-09-cron-provider-timeout-storm — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all
RuntimeError: Request timed out.on the aliases behindcustom:omni-101, 00:00 → 15:00 AWST. CLOSED — all 12 incident rows are now resolved; the last two (1b23c0_c35703f637b8,1b23c0_103cbe04a79b, both on the weekly Friday job1b23c042e17d) closed 2026-10-23 08:04 when that slot ran clean, and the job readsokagain -
incidents/2026-10-09-memory-recall-relevance — Recall relevance: a stale conversational episode outranked current-state facts on an open question. ✅ RESOLVED 2026-11-01 — the ranking weights were persisted, in
~/.hermes/mnemosyne/config.yamlrather than the file the 2026-10-10 probe read; the original regression query now returns only current-state facts, and conflict152was auto-resolved on 2026-10-29 -
incidents/2026-10-05-dead-model-pin-repin — Seven cron jobs on dead model pins (
cliprox/terra,qwen3.5:397b,deepseek-v4-flash,gpt-5.6-terra) repinned to routing aliases oncustom:omni-101; failing set 7 → 0 (resolved 2026-10-05) -
incidents/2026-10-03-mnemosyne-vector-recall-fts-only — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
-
incidents/2026-10-03-ct450-hermes-update — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
-
incidents/2026-10-01-cliproxy-connection-auth-expired — OmniRoute
cliprox/terraconnection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); closed 2026-10-05 when all three were repinned off the pin -
incidents/2026-08-26-work-outlook-basic-auth-disabled — Work Outlook mailbox unreachable: tenant allows only
XOAuth2, himalaya configured for password (open since 2026-08-26) -
incidents/2026-09-28-hermes-managed-runtime-deps — Managed-runtime venvs missing
dotenv/mnemosynebroke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. Recurred 2026-10-10 inside the gateway runtime env —mnemosyneandelevenlabsmissing from the committed uv env the gateway loads; both repaired and verified the same evening -
incidents/2026-08-18-mnemosyne-capture-pipeline — Mnemosyne capture pipeline broken (Aug 2026, superseded): capture verified healthy 2026-10-12 — 786 working rows, +2 that day, 98.3 % embedded
-
incidents/2026-08-15-skills-index-drift — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration
Monitoring & Dashboards
- systems/homepage-dashboard — Homepage dashboard (CT201)
- systems/home-assistant — Home Assistant integration
Reports
- reports/uber-eats/index — Uber Eats monthly spend reports
- reports/communities-wa-social-media-audit-2026 — Department of Communities WA social media audit and first-day brief
- systems/obsidian — Obsidian vault structure and conventions
- queries/retained-research-answers — Retained research answers (fastest VNC clients, 2026-10-11)
Infrastructure
Services
- infrastructure/hosts — Hosts, containers, important paths
- infrastructure/docker-services — Docker compose files
- infrastructure/credentials-map — Where credentials live (no values)
- infrastructure/network-map — Zones, hosts, and ports across LAN, Tailscale, and the public tunnel
- infrastructure/domains-and-tunnels — martinwa.org hostnames and Cloudflare Tunnel routing
- infrastructure/backups — hermes backup CLI, retention, and gaps
Networking
- infrastructure/proxmox — PVE hosts (two:
pve9.2.10 +m93p9.2.10, unclustered, IDs collide), VMs, LXC containers, storage (/mnt/ssd078.29 % / ~152 GiB free, 2026-11-07) - infrastructure/tailscale — Tailscale serve routes (CT460 verified exactly) and the 52-node tailnet inventory (re-verified 2026-11-07:
octopback online; 9 offline ages refreshed; CT107mem0added to the m93p guest list) - systems/cloudflare-tunnel — ct207-hosted tunnel serving Hermes publicly
Concepts
- concepts/agent-memory — Memory model
- concepts/model-routing — Routing concepts
- concepts/persistent-shells — Terminal persistence
- concepts/browser-automation — Browser automation concepts
- concepts/tool-calling — Tool calling patterns
Decisions
- decisions/index — Decision index
- decisions/keep-openrouter-as-fallback — Keep OpenRouter as fallback provider
- decisions/persistent-shell-enabled — Enable shell persistence
Comparisons
- comparisons/camoufox-vs-chrome — Camoufox vs Chrome
- comparisons/hermes-vs-nanobot — Hermes vs Nanobot
- comparisons/memory-backends — Memory backend comparison
- comparisons/model-routers — Model router comparison
Runbooks
- runbooks/multiplexer-setup — Multi-profile gateway multiplexer setup
- runbooks/diagnose-docker — Docker diagnosis
- runbooks/recover-docker-service — Docker service recovery
- runbooks/restart-hermes — Hermes restart procedure
- runbooks/restart-browser — Browser restart procedure
- runbooks/update-hermes-safely — Safe Hermes update
- runbooks/restore-openrouter — OpenRouter restoration
- runbooks/provider-health — Provider health check
- runbooks/troubleshoot-tool-loop — Tool loop troubleshooting
- runbooks/gateway-resource-resilience — Gateway resilience
- runbooks/job-radar — Job Radar search, Notion, dashboard and recovery procedure (two-stage since 2026-10-27:
dca8482e4f76discovery →5fc292ff0f3bwrite and letters)
Raw
- raw/configs/hermes-config-sanitized.txt — Sanitized config