Files
agent-estate-wiki/index.md
T

82 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Anthony's Systems Wiki collection active 2026-07-22 2026-11-14 2026-11-14 high
index
infrastructure
systems
stream-failure
ct299
skills-index
generator-divergence
secrets-at-rest
incident
fire-claim-discard
job-radar
hook-timeout
live-cron-ledger
live-deliveries-db
live-incident-ledger
live-session-store
raw/stocktake

Anthony's Systems Wiki

Operational knowledge for Anthony's agents, infrastructure, model routing and automation. Last updated: 2026-11-15.

2026-11-14 (nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) — a third TTS stack landed on CT200, and the memory pipeline went quiet. New page systems/phonon-tts: ptts-openai-server (:8881, OpenAI-compatible /v1/audio/speech, model gradium/phonon-40m-en, container created 2026-11-14 10:31:56 AWST), phonon-demo (:8882, WASM demo UI on nginx:alpine, 11:00:11), ptts-ws-server (:8883, WebSocket, locally-built ptts-ws-server:0.7.1, 11:43:59) — verified live from CT460: /health 200, /v1/models, and a POST /v1/audio/speech returning a valid MP3; /opt/ptts-voices holds 7 voices; both ptts containers were restarted together at 17:54 AWST. ⚠ Provenance gap: the retained transcripts cover only 08:35–09:03 AWST, so the 10:31 / 11:00 / 11:43 / 17:54 changes are not attributable to any session in this profile (working hypothesis: another estate agent — unproven). New open incident incidents/2026-11-14-mnemosyne-capture-stall: Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST — working_memory and episodic_memory stop in the same minute, last 8 working writes all 01:01–01:32 UTC — and the health monitor ea79a2107e36 escalated only at 2026-11-15 00:03:35 (exit 1, incident ea79a2_601037516abd, failure_type: auth — the scheduler's classifier, not a credential fault, state alerted, unclosed, alert text printing the raw UTC value so it reads as a 23 h gap when the true AWST gap was 14 h 31 m). ⚠ Correction, re-derived 2026-11-15 01:2x: the 12:00:52 run saw nothing — it measured capture delay 29.4 min (healthy, exit 0, suppressed), and the -589.5 min ℹ️ line belongs to the 00:03:35 run's own output block (monitor log: header 957 = 12:01:52 run, header 1030 = 00:03:35 run, -589.5 at 1046, ❌ at 1047). The earlier "~12 h of lead time" claim is withdrawn, as is the "exited 0 at 12:00 and 18:03" sentence (schedule 0 */12; there was no 18:03 slot). Detection is activity-aware — mnemosyne_freshness_gate.py fires on delay < −120 min (magnitude-based, no sign-only test) and can only fire once a conversational message postdates the last capture; the conversation was idle 09:03→19:22, so midnight was the first slot able to see the miss. What actually went unread: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST, printed as (0 items) every run yet tested only for age (> 36 h). ~15 h of desktop (last message 19:22 AWST) / Telegram / cron context is uncaptured, and the 23:30 sleep cycle consolidated 0 items. Ruled out: the known stringified-config killer (config.yaml mtime 2026-11-13 18:03; full YAML walk found no [/{ scalars). Root cause not established. Fleet otherwise clean — until the monitor's own 00:03 alert: 2026-11-14 settled 167 rows — 166 completed / 0 failed / 1 running (the running row is this audit), deliveries 4 / 4 delivered, no other incident row moved, so cron_incidents went 0 → 1 unclosed (the monitor's row, alerted_at 00:03:44). 2026-11-15 so far: 8 rows — 7 completed / 1 failed (the monitor), 1 delivery delivered. Fleet 29 (26 / 3) — a ninth window unchanged, but non-clean last_status 0 → 1 (ea79a2107e36, error, failure_streak 1), the first red row since the Uber Eats job cleared on 2026-11-13. infrastructure/docker-services re-verified: CT200 64 running non-seeder / 72 total excluding the 1 transient seeder (was 61 / 68 on 11-07) — the delta is exactly the three TTS containers, no removals. The 2026-11-08 hook cascade gained no new occurrence in the 11-14 body (the 00:05:40 timeout was already recorded by the 11-13 audit; errors.log 11-14 = 1 timeout + 10 suppression lines, otherwise only the benign Browser Use viewing unavailable plugin warning at ~2/hour). 2026-11-13 (nightly audit; live probes 2026-11-13 23:59 - 2026-11-14 00:2x AWST) — the fleet went fully clean, then the hook came back. 2026-11-13: 174 ledger rows — 173 completed / 0 failed / 1 running (this audit, started 23:59:41) — the third consecutive zero-failed day (11-11 172/172, 11-12 176/176); deliveries 8 / 8 delivered, 0 for_failure; cron_incidents 0 unclosed. Every job now reads last_status: ok with failure_streak 0 — the fleet has no non-clean row at all, because 066bb5191280 (Uber Eats Monthly Spend Report) was fixed end-to-end on 2026-11-13, clearing a delivery_failed that had stood since 2026-11-01. Two stacked causes: the job's runtime prompt had grown to 88 k chars (it carried two heavy browser skills) so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest, and the delivery_failed label was a stale TELEGRAM_HOME_CHANNEL_THREAD_ID=34795 env var that only clears on a gateway restart (2026-11-13 13:14). Fix: deterministic printer ~/.hermes/cron/scripts/uber-eats-digest.py, rewritten prompt with a hard output contract, skills cleared (88 k -> 1.8 k); verified on the 18:51 -> 18:58 slot (154 unique / 153 completed orders, Oct 9 orders / $301.04), lesson saved to cron-job-maintenance. Fleet 29 (26 / 3) — an eighth window unchanged. ⚠ But the window's fault is the pre_tool_call hook, and it returned: _pre_tool_call timed out after 120s at 2026-11-14 00:05:40 plus 10 suppression lines through 00:06:38, refusing this audit's own tool calls for ~1 minute (first-hand). errors.log by day: 11-12 0/0, 11-13 0/0, 11-14 1/10 — the first such timeout in the retained log since the 2026-11-08 cascade (errors.log now starts 2026-11-09 13:18:39, so the 11-08 lines have rotated out and the page's 58-count stands on the 2026-11-09 read). Load is the trigger and it has ~doubled: load average 12.08 / 13.02 / 12.33 on 2 vCPU (vs 5.89 at diagnosis), with state.db now 2.04 GB (2,977 sessions / 207,577 messages, was 1.89 GB / 2,901 / 197,682) and the root filesystem 88 % used. The 30 -> 120 s fix is proven insufficient; incidents/2026-11-08-pre-tool-call-hook-timeout-cascade is reopened, resolved -> active, and state.db checkpoint/retention — not a bigger timeout — is now the estate's most urgent open item. Freshness (03:00 stocktake): 0 stale / 131 fresh / 3 no-date of 134.

2026-11-11 → 2026-11-12 (nightly audit; live probes 2026-11-12 23:59 – 2026-11-13 00:4x AWST) — a clean day, and an upgrade that could not be installed. The 429 row that was the estate's only open incident closed resolved at 2026-11-12T00:25:36 (the next slot of the same job ran clean, carrying both windows); cron_incidents now reads 0 unclosed of 137. 2026-11-12: 176 ledger rows — 175 completed / 0 failed / 1 running (this audit, started 23:59:54), deliveries 9 / 9 delivered; 11-11: 172 / 172 completed / 0 failed, deliveries 7 / 7. Fleet 29 (26 / 3), a seventh window unchanged. The window's one new open item is not a failure: the attempt to take hermes-hq-push past its installed 0.9.42 was refused by the pre-install plugin security scan — "Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict." The critical is the plugin's own skills/dashboard-tile/SKILL.md one-liner; no per-verdict override exists (plugins.scan_on_install: false would blind the scan for every future install, and plugin_guard.py hardcodes trust_level="community"). The fix is PR #2 on mrcharlesiv/hermes-hq-plugins (author Tony0410, fix/tile-skill-heredoc, OPEN / MERGEABLE, mergedAt: null, +7/−2 in one file), pushed from a fork because gh is read-only upstream. Installed 0.9.42 keeps delivering. Also: the alleged "stored hold-and-ask instruction" for scan warnings does not exist on disk (184,716-file sweep, zero matches in editable text; the sentence lives only in state.db history) — nothing was removed or reverted, and the hunt is now the skill stored-instruction-search-and-removal (skills 299 → 300). config.yaml is byte-identical to its 2026-11-12 18:14:52 backup; the only drift since 11-09 is the model catalogue block (6 out, 9 in). New page: incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.

2026-11-12 (03:00 stocktake) — the freshness scan was clean, the live-vs-page diff was not. The age scan flagged 1 stale / 129 fresh / 3 no-date of 133 (the sftpgo incident at 22d, re-verified live and cleared). But infrastructure/tailscale.md — only 2 days old — had drifted: mailflow, qwenpaw and springroll all read Online: false in the peer record while all three ping fine (CT1262 and CT465 running in pct list; springroll answers via DERP(sin), relay not direct), and omarchymac was stale in the other direction (page said offline 2d / last seen 2026-11-08 00:45Z; live reads LastSeen 2026-11-07T06:41:48Z = offline 4d, ping timeout). Node table rebuilt by script, all IPs matching, no rows missing, 58 rows = 57 live IPv4 peers + hermes, 80 peers total (57 IPv4 / 23 IPv6-only) unchanged. Also re-probed: CT299 sftpgo running/online/healthz-401; both DEAD Serve targets (8769, 9123) still listener-less; CT201's Serve table unchanged; container counts 17 LXCs + 4 VMs on pve and 20 LXCs + 2 VMs on m93p; and new guest CT1007 homebridge on m93p — running but tailscaled inactive, so it does not join the tailnet.

2026-11-12 (03:00 stocktake) — a "router" failure mode turned out to be fleet-level. incidents/2026-10-21-omniroute-pre-dispatch-filter-skip recorded "all targets were skipped by pre-dispatch filters" as a Job Radar one-off and left three open questions. Live re-read of executions.db shows the same signature on a second job and a second pin: cbfe434a3b5e (Wiki Audit, custom:omni-101 / free-stack) failed on 2026-11-10 with HTTP 429 … (credits 100%), incident row cbfe43_ed01845310d8 (rate_limit) closed resolved 2026-11-12T00:25. The original subject has not repeated — dca8482e4f76 ran clean 11-06/09/10/11. So open question 1 (a job-specific mis-configured filter) now looks less likely than a router-side guard that blanks every target, and open question 3 (no failover pin) applies to the whole free-tier cohort, not just the radar. ⚠ Caveat recorded on the page: executions.db's retention floor is 2026-11-06T05:09:30+08:00, so the 2026-10-21 row is not re-readable today — the recurrence is an in-window read.

2026-11-09 (nightly audit — window 2026-11-08) — the day's fault was a hook, not a schedule, and it is fixed. The conduit_push plugin is the only pre_tool_call registrant on the host (__init__.py:119), and its callback overran Hermes's 30 s hook budget 58 times between 2026-11-07 14:14:03 and 2026-11-08 08:21:03 — errors.log holds 12 lines on 11-07 and 46 on 11-08, plus 89 "skipped after previous timeout or while still running" suppression lines. Every tool call degraded to slow-then-suppressed, which is what stretched the day. Fixed 2026-11-08 13:35 AWST: plugins.hook_callback_timeout 30 → 120 in ~/.hermes/config.yaml — verified live, and verified effective (the last timeout line is still 08:21:03, re-checked at audit time). ⚠ The obvious suspect was measured and acquitted: the plugin's caching is fine (session_titles.py:21 CACHE_SECONDS = 30.0; timed in-process at 3.1 s cold, 0.00 s warm), so the "re-opens a 1.87 GB state.db per call, ~9 s each" theory was wrong — the trigger is load on the 2-core CT460 (load average 5.89 at 13:27:50, state.db checkpointing a 40 MB WAL on the same spindle) → incidents/2026-11-08-pre-tool-call-hook-timeout-cascade. Cost of the fault: Mnemosyne Health Monitor ran 00:01:20 → 08:29:05 (8 h 27 m) and 12:00:51 → 18:00:42 (5 h 59 m 51 s) — its two slowest slots on record, both delivered; FreshRSS Midday Briefing took 2 h 09 m 53 s; and the stocktake failed a third consecutive day (1a8e90ea2271, 03:01:11 → 04:36:08, failure_streak: 3) on a new terminal signature — RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text. where the previous two red days read "lost its durable fire claim ownership". Working hypothesis, not proven: an open tool result is exactly what a suppressed pre_tool_call would produce, but the string does not name the hook. 11-08 settled: 169 rows — 168 completed / 1 failed / 1 running (this audit, 23:59:33); deliveries 8 / 8 delivered; fleet 29 jobs (26 / 3) — a sixth consecutive audit at the same count with nothing added, retired, disabled or repinned, though updated_at moved to 2026-11-09T00:06:50+08:00 as a run-state stamp. Unclosed incident rows 2 → 3, all three the stocktake's and all silent (detected, alerted_at: null) — but the alerting path is not the fault: the Newsletter Digest row 85edf6_0877ca1e0eac alerted 07:23:07 and resolved 08:06:54 in the same window. ✅ Two verified resolutions: the Mnemosyne Health Monitor's no-agent conversion works — its first script slot ran 2026-11-09 00:00:48 → 00:02:02 = 73 s, exit 0, silent, writing a 159 B output file against 105,982 B for the last agent-mode run (coverage live at working 1030/1030 = 100.0 %, episodic 1192/1192 = 100.0 %, so the < 95 % alert is correctly quiet; the script uses awk because bc is not installed on this host); and the 1.89 GB state.db was re-measured (1,890,656,256 B, 2,901 sessions / 197,682 messages, live 40 MB WAL) with the 2026-08-11 remnant state.db.corrupted.20260811_151052 (624.1 MB) deleted. New page: incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.

2026-11-08 (nightly audit — window 2026-11-07) — two runs orphaned mid-restart, and two incident rows that never alerted. The gateway restart that arrived at 05:01:00 (after a requested restart sat in after-turn mode for the full 1800 s) drained two in-flight cron executions; both closed failed on lost its durable fire claim ownership — Mnemosyne Health Monitor (ea79a2107e36, 00:00:42 → 05:02:22) and wiki-stocktake-maintenance (1a8e90ea2271, 03:00:31 → 05:02:22, second consecutive failed day, failure_streak: 2). The restart was requested at ~04:29:52 — inside the conduit_push self-update window (0.122.2 → 0.134.4, 04:27:55→04:32:32, its own ledger recording restarting_gateway = failed) — working hypothesis, not proven; the 0.22.5 → 0.23.0 updater is excluded (its receipt starts 05:05:13). Day counts: 163 rows — 159 completed / 3 failed / 1 running; the third failure is the agent-log watchdog correctly reporting the event 19 s later. ⚠ Both stocktake incident rows are still detected with alerted_at: null — closed 0 → 2, and neither alerted. Elsewhere in the window: Hermes 0.22.5 → 0.23.0 on CT460 (snapshot stage skipped again — still no rollback point), the camofox browser backend arrived as an enabled plugin (1.1.0), conduit_push self-updated to 0.134.4, hermes-hq-push reinstalled 0.9.27 → 0.9.42 (routes reloaded by the 11:11 restart), and the OpenCode Go credential is expired (HTTP 401 api_key_expired — configuration alive, credential dead, nothing consuming it yet). New page: incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.

2026-11-05 (stocktake) - the systems/skills-index.md flip-flop is fixed at the root: two different scripts were writing the page, and the nightly audit was pointed at the stale one; it is now a delegating wrapper -> incidents/2026-11-05-skills-index-generator-divergence. Docker inventory corrected against live probes on all five hosts - the worldmonitor-seeder-scheduled container recorded yesterday does not exist on any host ⚠ CORRECTED 2026-11-06: false - the seeders are transient systemd-timer-driven containers, absent only between runs, not removed. Tailscale node table re-verified programmatically - 52 rows, 52 live peers, every state matching ⚠ CORRECTED 2026-11-06: anthonys-mac-mini has since come back online. Freshness 0 stale / 122 fresh / 3 no-date of 125.)

2026-11-07 (nightly audit - window 2026-11-06) - a red row with no alert: 2026-11-06 settled 172 execution rows across 16 jobs - 170 completed / 1 failed / 1 running (this audit). The single failure is a new, non-restart signature: the Job Radar's 11:30 slot (5fc292ff0f3b, 11:30:32 -> 13:26:44) was discarded as stale on Fire claim ownership lost; stale result was discarded. because a direct fire of the same job (08:27:09 -> 13:26:22, 4 h 59 m 13 s) was still holding the claim and delivered first. Every earlier occurrence of this string came from a restart/reload orphaning a run; this one had no restart -> incidents/2026-11-06-job-radar-concurrent-fire-claim-discard. It raised no cron_incidents row (newest ledger activity remains the 2026-11-05T12:36:17 closures) and nothing alerted; the failed run left no output file. The day still landed in full via the 14:30 slot (12 m 27 s, SAME_DAY_STRONG_COUNT 4, MISSING_COVER_LETTER_COUNT 0, cache 494, Run status: COMPLETE). Deliveries 10 rows / 10 delivered / 0 for_failure; fleet 29 jobs (26 / 3) - fifth consecutive audit at the same count, nothing added/retired/disabled/repinned, non-clean last_status still 1 (066bb5191280). The stocktake's overrun fully unwound (1a8e90ea2271, 03:00:32 -> 04:11:35 = 1 h 11 m 03 s against 7 h 40 m / 5 h 41 m / 3 h 02 m on 11-03 -> 11-05) and Stage 1 ran clean for a second day (dca8482e4f76, 1 h 28 m 37 s, suppressed), clearing the 11-04 Open entry in substance. The last audit's open question closed itself: the Mnemosyne Health Monitor's 11-05 12:00:45 slot (12 h+ in flight, undelivered at the last audit) completed at 2026-11-06T00:14:39 after 12 h 13 m 54 s and delivered; its 11-06 slot took 7 h 47 m 56 s and delivered too - ballooned on consecutive days, never lost, watching, not open. The one open page remains incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts (untouched in-window).

2026-11-06 (nightly audit - window 2026-11-05) - a clean ledger day and an empty incident ledger: 2026-11-05 settled 175 rows - 173 completed / 2 running / 0 failed across 18 jobs, the first zero-failed day since 2026-11-02. The only two running rows are the Mnemosyne Health Monitor (12:00:45 slot, still in flight 12 h+ at audit time) and this audit itself. Unclosed incident rows 3 -> 0 - the Newsletter Digest row closed at 07:07:42 and both Job Radar - Discovery rows at 12:36:17, when dca8482e4f76 finally completed after 5 h 35 m 35 s; that closes out incidents/2026-11-04-upstream-stream-failure-0700-cohort and incidents/2026-11-03-job-radar-stream-active-timeout in full. Fleet non-clean last_status 3 -> 1 (only 066bb5191280, delivery_failed, streak 0); deliveries 10 rows / 10 delivered; the 03:00 stocktake overran 3 h 01 m 58 s (down from 7 h 40 m and 5 h 41 m) and committed 2f3fc99 at 05:55. Fleet unchanged at 29 jobs (26 / 3) - a fourth audit at the same count. ⚠ New and open: live .env values (TELEGRAM_BOT_TOKEN, MATTERMOST_TOKEN, HERMES_CUSTOM_OMNIROUTE_20128_API_KEY) sit verbatim in 17 rows of mnemosyne.db (memories, episodic_memory, working_memory, gists, memoria_facts.context_snippet and the FTS mirrors) - reproduced in full by each of the 7 uncompressed rotation snapshots - and all four values sit verbatim in the 1.7 GB session store state.db too; CAMOFOX_API_KEY sits verbatim in the prompts of two enabled Facebook Birthday jobs (0e753e9720fd, fad5a86979d4) and therefore in cron/deliveries.db -> incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.

2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze: the 11-04 fleet settled 170 rows - 167 completed / 2 failed / 1 running, non-clean last_status 2 -> 3, unclosed incident rows 1 -> 3, deliveries 8 rows / 8 delivered. Both failures are the same upstream fault: dca8482e4f76 (Daily Marketing Job Radar - Discovery) and 85edf6772bca (Tech & AI Newsletter Digest (Analytical)) died 36 seconds apart at 09:41-09:42, each after 2 h 41 m, on RuntimeError: Upstream stream failed before completion. - both pinned custom:omni-101 / smart-route. The day was not lost: the 11:30 Stage 2 slot reconstructed the missing handoff from Stage 1's own candidate files, re-verified every candidate live, and still wrote 5 roles (board 479 -> 484) + 1 canonical cover letter (MISSING_COVER_LETTER_COUNT 0) before consuming the handoff at 16:08:59 -> incidents/2026-11-04-upstream-stream-failure-0700-cohort. The cover-letter route survived only on an override: regenerate_cover_letter.py defaults to deepseek-paid-smart, which threw HTTP 402 mid-run, so the letter was pinned to gemini/gemini-3.5-flash via CL_MODEL - and that override was never written to .env, so it did not persist. And CT299 was frozen twice, not once: a second, manual vzdump 299 ... --storage ssd0 wedged from 11-04 08:30:04 until 11-05 ~00:30 (~16 h); the live probe at 00:54:21 AWST shows LOCK: 0, both cgroup.freeze flags 0, guest ALIVE, sftpgo: active -> incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage. ⚠ Corrected the same night: the first draft of this entry read the fleet as 158 jobs (144 / 14) and the 11-04 ledger as closing 1 item_suppressed; live reads give 29 jobs (26 / 3) and a final row of 1 running (this audit). Keyed probe 2026-11-05 01:33: deepseek-paid-smart is undispatchable on omniroute:20128 (HTTP 503, all targets skipped) but serves normally on omniroute-1 (HTTP 200 -> deepseek/deepseek-v4-flash).

2026-11-04 (nightly audit) — the ledger raised its own red row: the **11-03 fleet settled 170 completed / 1 failed / 1 running / 0 unknown (172 rows) — unclosed incident rows 0 → 1, non-clean last_status 1 → 2, deliveries 7 rows / 7 delivered. The single failure is the Job Radar Stage 1 (dca8482e4f76), which ran 07:00:33 → 13:28:52 (6 h 28 m 18 s) and died on RuntimeError: stream active timeout — a new terminal signature, not the iteration cap — after going silent for 65 m 21 s; its incident dca848_dfd4bea84f2a (timeout) is alerted and open and the notice was delivered (13:28:51.741724) → incidents/2026-11-03-job-radar-stream-active-timeout. That answers the 11-02 page's open question: output is suppressed when the run ends completed, delivered when it ends failed — which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so 2026-11-02's and 2026-11-03's candidate sets are both unrecovered. ⚠ The 03:00 stocktake overran for a second day (1a8e90ea2271, 03:00:30 → 10:41:13 = 7 h 40 m 43 s, suppressed, nothing delivered, artefacts uncommitted again). New page: systems/hermes-hq-push (v0.9.27, enabled; manifest says it replaces dispatch-push, and no trace of the old plugin survives).

⚠ Two Proxmox hosts exist — always qualify a guest with its host. A second unclustered hypervisor, m93p (100.104.51.98, PVE 9.2.10), was discovered during the 2026-10-14 stocktake. Guest IDs collide between the two hosts.

Start here

Systems

Core

Integrations

  • systems/conduit — Conduit: Hermes on Anthony's phone (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree ~/.hermes/plugins/conduit_push, v0.122.2 (HEAD 9eac857, 2026-10-31 12:04 — "Every profile that turns Conduit on runs the main install (#482)"; +148 commits on since the 0.69.2 / 942ddb8 baseline, working tree clean), upstream kaishi00/hermes-conduit-notifier. Gateway on :9119 is now pid 176771, started 2026-10-31 19:35:02 (superseded: restarted 2026-10-25 18:42:33, which is when the upstream commits went live). One registered device (iPhone app 2.2.0). ✅ The first two real calls in the product's history are now on record — 2026-10-29 08:38:04 and 08:49:36 — after three faults in series were repaired (truncated speech-server URL → …/v1/realtime; 12 h dashboard session TTL → 30 d; dead groq STT → local, dead openai TTS → ollama_local) → incidents/2026-10-29-conduit-voice-silence. So the memory fix is no longer evidence-free. ✅ That patch is gone — the tree is clean and MEMORY_TIMEOUT_S ships upstream in dashboard/api/memory.py. ⚠ New fault: "local" voice mode never applies — the Mac mini speech server's LLM proxy is disabled (LLM endpoint → 501, /v1/usage reads llm_proxy.requests: 0, realtime socket rejects session.update), while the app's "test connection" checks socket reachability only → incidents/2026-10-31-conduit-voice-local-mode-not-applying. The call store was re-shaped: history is empty and last_request_at is null, so the two 10-29 calls are no longer in it. The 10-24 refusal fault has not recurred → incidents/2026-10-24-conduit-call-request-refused.
  • systems/messaging-integrations — Telegram, Discord, email, Slack, etc. Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26 (peer offline 36 d, tx 312 / rx 0; host answers neither :8065 nor ICMP). 268 connect timeouts on 10-25 vs 264 on 10-24; the reconnect attempt counter reset to 54 at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost.
  • systems/push-notifications — Cadu/Vory push bridge (hermes-push, vory-push): gateway on :9119, state in ~/.hermes/push/. Broken — connected: false (the bridge dials a name that resolves to 127.0.1.1 where nothing serves 443); chronic, not a blip.
  • systems/hermes-hq-push — Hermes HQ iPhone app plugin (standalone, v0.9.42, kind: standalone, author Hermes HQ; tree mtime 2026-11-08 02:19). hermes plugins list → enabled (state user); config.yaml → plugins.enabled (line 13226 as of 2026-11-13; the number drifts with each catalogue write). ⚠ 2026-11-12: the upgrade past 0.9.42 is blocked by the pre-install security scan (dangerous verdict, one critical in the plugin's own tile skill) — see incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan. Host side of the Hermes HQ app: Apple push (finished turns, lock-screen approvals, "waiting for a password/login/code" alerts, live Lock Screen cards sealed to the phone), phone device access (contacts, location, reminders, alarms, photos, Health, motion, music), the desktop-app Hermes HQ theme, the app's Terminal page, fast Sessions list on a busy host, cross-posting answers back to Telegram/Discord, a restart guard, and Grok Live voice-token minting (XAI_API_KEY or subscription sign-in; bearer never reaches the phone). Tools device_contacts_search / device_request_result; 7 hooks. Its manifest states it replaces dispatch-push — and the replacement is clean: no dispatch-push reference survives in config.yaml and no such directory remains. Different stack from the Cadu/Vory bridge on systems/push-notifications. ⚠ Its pre_tool_call guard also blocks any tool call whose arguments contain the literal text of a launchd one-off-restart command — describe it, never quote it.
  • systems/scheduled-tasks — As of 2026-11-14 (window 11-14): 167 rows / 166 completed / 0 failed / 1 running (this audit); deliveries 4 / 4 delivered; cron_incidents 0 → 1 unclosed (138 total) — the Mnemosyne Health Monitor wrote the estate's first red row since 11-13 (last_status: error, streak 1); fleet 29 (26 / 3). Earlier — As of 2026-11-12 (window 11-12): 176 rows / 175 completed / 0 failed / 1 running (this audit); deliveries 9 / 9 delivered; unclosed incident rows 0 of 137 — the 429 row closed resolved at 2026-11-12T00:25:36; fleet 29 (26 / 3). Earlier — As of 2026-11-09 (window 11-09): 174 rows / 173 completed / 0 failed — the first failed-free day since 11-05; deliveries 8/8 delivered; unclosed incident rows 3 → 0; the stocktake closed completed after 9 h 10 m with zero edits landed; Job Radar Stage 1 overran 9 h 09 m and landed nothing (new incident). Earlier: As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean last_status 4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), and ea79a2107e36 is ENABLED again (last_status: ok, streak 0) after its 00:01:08 slot closed completed at 10:02:48; deliveries 8/8 delivered. Prior window — as of 2026-10-30 03:3x: 29 jobs (25 enabled, 4 disabled); non-clean last_status 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all delivered. ⚠ ea79a2107e36 (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed unknown on a gateway-connection failure — and its failure notice was NEVER DELIVERED (deliveries.db has zero rows for 2026-10-30), so the incident row is the only trace → incidents/2026-10-30-mnemosyne-health-monitor-paused. Every other previously-red job has cleared to ok/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean last_status 5 → 3 (all three are stale re-runs of old faults and every failure_streak read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all delivered. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with missing_cover_letter_count 0 (Notion 455 → 475). Both stage-1 incident rows closed resolved at 10:16:12; the lone survivor is the Facebook monthly's 0e753e_e7f8194a7d06.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** dca8482e4f76 ran 07:01:08 → 11:03:54 (4 h 2 m 46 s), closed failed, and delivered "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery finished, no Notion writes and no cover letters. Incident row dca848_e93a538ade6b opened 11:03:53 and is still open. The same day the single radar job was split into two: stage 1 dca8482e4f76 (Discovery, 07:00, deliver: local) + new stage 2 5fc292ff0f3b (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two completed test fires that evening). The two incident rows the 2026-10-26 planned restart raised auto-closed 00:38:32, and the 2026-10-26 orphan class did not recur. → incidents/2026-10-27-job-radar-budget-exhausted, runbooks/job-radar. Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean last_status, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault. ea79a2107e36 (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a planned 14:33:35 SIGTERM restart that timed the drain out after 1.35 s, killed after 2 h 34 m on a new string (lost its durable fire claim ownership), and raised two cron_incidents rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → incidents/2026-10-26-planned-restart-orphaned-cron-run. New fleet precedent: a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). Unclosed incident rows 2 → 3; one closed: c0a9fa_764558b5ae83 auto-closed 09:13:00 when the radar's 09:00 slot ran clean, so every row of the 2026-10-16 503 resource-pressure incident is now closed → incidents/2026-10-16-omniroute-503-resource-pressure. Non-clean last_status 5 → 4: c0a9fa314542 and 08eb86f02d37 cleared to ok; ea79a2107e36 joins as error, streak 2. Pins unchanged. ⚠ fad5a86979d4's 03:08 slot completed but delivered delivery_failed — Thread 34795 not found (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered “RUN INCOMPLETE — tool-call budget exhausted” and its cover-letter pipeline changed that morning → runbooks/job-radar. Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean last_status, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0. ⚠ Both 10-25 failures are 08eb86f02d37 (Gmail Action Digest), which also fired six times that day — and neither raised an incident row → incidents/2026-10-25-gmail-digest-runaway-and-refire. ⚠ The global default reverted to custom:omni-101 / smart-route on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again → systems/current-profile. ⚠ Nous Portal credentials still invalid — all 8 auxiliary slots moved to no-think/freeapi/auto at 19:13 → incidents/2026-10-22-nous-portal-refresh-token-invalid. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean last_status, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third consecutive clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0. Zero incident rows opened or closed in-window; pins unchanged. ⚠ Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls → incidents/2026-10-24-conduit-call-request-refused. Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean last_status, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four. The 2026-10-09 provider timeout storm is now fully closed: 1b23c042e17d ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so all 12 are closed and unclosed incident rows fell 4 → 2. ⚠ New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (CONDUIT_MEMORY_TIMEOUT_S) + degraded-not-aborted on TokenError; dashboard restarted 18:54:43, no abort since. Patch uncommitted → incidents/2026-10-22-conduit-memory-start-timeout, systems/conduit. The 10-22 stocktake overrun did not repeat (10-23 03:07 committed 250c735). Prior window: As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean last_status, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen. dca8482e4f76 (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s → incidents/2026-10-21-omniroute-pre-dispatch-filter-skip. 08eb86f02d37 cleared (08:30 slot clean; incident resolved 08:37:21) and 2026-10-20 settled at 169 / 0 + 1 unknown. ⚠ Hermes Agent was updated to 0.22.3 at 22:47 → 22:57 with its pre-update snapshot skipped → systems/hermes-agent. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The omni-101 503 fault did not recur (back to monitoring) and 4 incident rows closed → 9 unclosed. ⚠ Two new findings: the Gitea config backup has not pushed since 2026-08-08 (75 unpushed commits) while its row reads ok → incidents/2026-08-08-gitea-config-backup-push-dead; and a gateway unclean exit on 2026-10-20 orphaned a digest run — no delivery row, no incident → incidents/2026-10-20-gateway-unclean-exit-orphaned-run. Also: the global model default moved to nous direct (meituan/longcat-2.5-preview:free) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while 20 of 28 still pin it → systems/current-profile. Prior window: 28 cron jobs (25 enabled, 3 disabled); 12 carry a non-clean last_status as of 2026-10-20 00:09 — up from 7, because the omniroute-1 503 resource-pressure fault returned on 2026-10-19 (second burst in four days: 9 model calls across 8 jobs, 06:01 and 07:01 → 12:01, all model: smart-route on custom:omni-101, while /api/health stayed 200 ok) → incidents/2026-10-16-omniroute-503-resource-pressure. 2026-10-19 closed 159 / 13 against 10-18's 168 / 1. The Mosquito Tracker was removed (29 → 28 jobs), and the 2026-10-18 degenerate-stream placeholder-delivery mode remains an open question → incidents/2026-10-18-degenerate-stream-canned-substitute. Earlier: the connection-refused storm (2026-10-14 → 10-15) — 24 model-call failures across 12 jobs, ~34 h of digests blanked. All 18 model-calling jobs still sit on custom:omni-101 with no failover.
  • systems/skills-index — Installed skills inventory (300 skills (67 categories) as of the 2026-11-13 audit; regenerated daily by the 03:00 job). The generator excludes .archive/ (50 archived skills) — the earlier 346/347 counts included it. ⚠ 2026-11-07 07:43 — four skills moved into .archive/ in one pass (cron-model-repair, fleet-wide-failure-triage, hermes-instance-update, mail-lookup), taking the count 302 → 299 and the parked archive 46 → 50; the Cron Model Repair category went with its only member. No active SKILL.md references any of the four, so if this was a consolidation the "absorbed into X" pointer is missing. One skill appeared in the same window (microsoft-office, Productivity 25 → 26). Freshness at the 2026-11-08 run: 0 stale / 126 fresh / 3 no-date of 129 pages (freshness-20261108.txt). Note: SKILL.md files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so 299 is the correct unique-skill figure; see systems/scheduled-tasks. ✅ 2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root: two different scripts were writing this page (the repo script → 302/68; the stale ~/.hermes/scripts/wiki-maintain.sh copy → 349/26, because it used Path.rglob() and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a delegating wrapper with no generator logic of its own → incidents/2026-11-05-skills-index-generator-divergence. ⚠ The 2026-10-22 slot ran 97 m 58 s (vs 16 m 01 s on 10-21), finished completed with delivery suppressed and committed nothing; the 10-23 slot closed that hole.

Incidents

  • incidents/2026-11-14-mnemosyne-capture-stall - ⚠ OPEN — Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST (~15 h at this read), and the health monitor took 14 h 31 m to say so. working_memory and episodic_memory stop in the same minute (last 8 working writes all 01:01–01:32 UTC); the pre_llm_call capture hook logs Mnemosyne capture ok on every desktop turn while writing 0 rows, so the hook's own success is not evidence of capture. The monitor ea79a2107e36 escalated at 2026-11-15 00:03:35 (exit 1, incident ea79a2_601037516abd, failure_type: auth — the scheduler's classifier, not a credential fault; this job's twelve earlier rows read agent / timeout / unknown — state alerted, unclosed), and its alert text prints the raw UTC instant, so it reads as a 23 h gap when the true AWST gap was 14 h 31 m. ⚠ Correction, re-derived 2026-11-15 01:2x: the 12:00:52 AWST run saw nothing (capture delay 29.4 min, exit 0, suppressed) — the -589.5 min ℹ️ line belongs to the 00:03:35 run itself (monitor log lines 957 / 1030 / 1046). Detection is activity-aware (mnemosyne_freshness_gate.py fires on delay < −120 min, i.e. on magnitude) and the first message to postdate the stall arrived at 19:22 AWST, after that slot — so the ~12 h of quiet was the 0 */12 cadence, not a threshold bug. What went unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST, visible as (0 items) in every run but tested only for age (36 h). Consequences: ~15 h of desktop / Telegram / cron context uncaptured, and the 23:30 sleep cycle consolidated 0 items. Ruled out: the known stringified-config killer (config.yaml mtime 2026-11-13 18:03; full YAML walk found no [/{ scalars). Root cause not established → systems/mnemosyne, systems/phonon-tts.

  • incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan - OPEN — the hermes-hq-push upgrade past v0.9.42 was refused by the pre-install plugin security scan. hermes plugins install mrcharlesiv/hermes-hq-plugins/hermes-hq-push --enable --force returned "Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict." The sole critical is the plugin's own skills/dashboard-tile/SKILL.md:41 shell one-liner (echo '<json>' | python3 … = obfuscation); the other 38 findings are low/medium and alone would have installed as caution. --force lifts caution only; a dangerous verdict is an unconditional block (tools/plugin_guard.py:406-418), and no per-verdict override exists — the only key is plugins.scan_on_install (default true, currently unset), which disables scanning for every future install; the trust_level/allowlist machinery lives in tools/skills_guard.py (the skills path) and plugin_guard.py hardcodes trust_level="community". Installed state unchanged: v0.9.42, enabled, still delivering. Fix = PR #2 on mrcharlesiv/hermes-hq-plugins (author Tony0410, fix/tile-skill-heredoc → main, OPEN / MERGEABLE, mergedAt: null, +7/−2 in one file, created 2026-11-12T10:00:01Z) — pushed from a fork because this host's gh account has read-only access upstream; local clone /home/hermes/workspace/hq-plugins-fix (90c9d28).

  • incidents/2026-11-10-wiki-audit-429-all-targets-skipped - RESOLVED 2026-11-12T00:25:36 (self-cleared on the next slot's clean completion; cron_incidents now 0 unclosed of 137) — the auditor failed, so 2026-11-10 was never audited. cbfe434a3b5e (Wiki Audit — Nightly) ran 2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s and died on RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s (delivery suppressed) — 6 targets evaluated, attempted 0, tenant tn-5ud77abtl1 quota exceeded. Incident row cbfe43_ed01845310d8 (rate_limit) was raised at 00:12:28 and is still open with alerted_at: NULL — the estate's only open incident row. Fleet non-clean last_status 1 -> 2; pool reset ≈ 06:36:57. Third symptom of the same router behaviour (cf. incidents/2026-10-21-omniroute-pre-dispatch-filter-skip).

  • incidents/2026-11-08-pre-tool-call-hook-timeout-cascade - ⚠ REOPENED 2026-11-14 — the hook was the fault, and the 30 → 120 s fix did not hold at current load. The conduit_push plugin is the only pre_tool_call registrant on the host (__init__.py:119) and its callback overran Hermes's 30 s hook budget 58 times between 2026-11-07 14:14:03 and 2026-11-08 08:21:03 (errors.log: 12 lines on 11-07, 46 on 11-08) plus 89 "skipped after previous timeout or while still running" suppression lines — every tool call degraded to slow-then-suppressed. Fixed 2026-11-08 13:35 AWST: plugins.hook_callback_timeout 30 -> 120 in ~/.hermes/config.yaml; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ The plugin's caching was measured, not assumed, and was innocent — session_titles.py:21 CACHE_SECONDS = 30.0, timed in-process at 3.1 s cold / 0.00 s warm, so the "re-opens a 1.87 GB state.db per call, ~9 s each" theory was wrong; the trigger is load on the 2-core CT460 (load average 5.89, state.db checkpointing a 40 MB WAL). Cost: Mnemosyne Health Monitor 8 h 27 m and 5 h 59 m 51 s; FreshRSS Midday 2 h 09 m 53 s; stocktake failed a third day. Next lever is state.db checkpoint/retention, not more caching. ⚠ Recurred 2026-11-14 00:05:40 — 1 timeout + 10 skips, overrunning the 120 s budget at load average 12.08 on 2 vCPU (~2× the 5.89 at diagnosis), with state.db grown to 2.04 GB. Status resolved → active; the lever is now urgent. → systems/conduit, systems/hermes-agent

  • incidents/2026-11-09-stocktake-completes-with-zero-edits (resolved 2026-11-11) - RESOLVED BY ITSELF. The nightly stocktake closed completed after 9 h 10 m having landed zero edits and no commit — failure_streak 3 → 0 while the work stayed undone, and the top-priority item on its brief had been the monotonically worsening failure streak. Third consecutive day its findings never landed.

  • incidents/2026-11-09-job-radar-stage1-stall-silent-noop - NO INCIDENT RAISED. Job Radar Stage 1 overran to 9 h 09 m, skipped its own verification, wrote no handoff file, and both Stage-2 slots ran before it finished and declined silently. No incident row was created — the completion-time-only alerting gap leaves the failure invisible.

  • incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain - OPEN. A restart requested at ~04:29:52 sat in after-turn mode for the full 1800 s, then timed out and drained at 05:00:43/47 — the same #77184 behaviour as the 10-31 reload, but this time it killed two in-flight cron executions: Mnemosyne Health Monitor (ea79a2107e36, 00:00:42.918 -> 05:02:22.870) and wiki-stocktake-maintenance (1a8e90ea2271, 03:00:31.515 -> 05:02:22.174), both closed failed on lost its durable fire claim ownership; systemd restarted the unit at 05:01:00 (restart counter is at 3). A third failure that night was the agent-log ERROR watchdog correctly catching the event 19 s later. The request landed inside the conduit_push self-update window (04:27:55 -> 04:32:32, its own ledger recording restarting_gateway = failed) — the working hypothesis for the requester, not proven; the 0.22.5 -> 0.23.0 updater is excluded (receipt starts 05:05:13). ⚠ Closed 0 -> 2 incident rows and neither alerted: both 1a8e90_* rows sit detected with alerted_at: null. The stocktake slot has now failed two days running (failure_streak: 2). Job Radar ran clean. → systems/scheduled-tasks, systems/hermes-agent, systems/conduit, systems/mnemosyne

  • incidents/2026-11-06-job-radar-concurrent-fire-claim-discard - OPEN. A direct-source fire of 5fc292ff0f3b (08:27:09 -> 13:26:22, 4 h 59 m 13 s, delivered) settled the claim while the scheduler's own 11:30:32 slot was still running, so that slot was interrupted at 13:26:38 and closed failed at 13:26:44 on Fire claim ownership lost; stale result was discarded. - a new string variant and the first claim-loss failure with no gateway restart involved. It raised no cron_incidents row, so a failed execution with no alert was the day's only red row; the discarded run left no output file. The 14:30 slot recovered the day in a normal 12 m 27 s cycle. Candidate fixes (slot suppression when a claim is live, consuming the next slot for out-of-band fires, raising an incident row on discard) are listed on the page and none was applied.

  • incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts - OPEN. Live .env values were found stored verbatim in 17 rows of mnemosyne.db (memories 4, episodic_memory 3, working_memory 1, gists 4, memoria_facts.context_snippet 1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the 7 uncompressed rotation snapshots, and all four values also sit verbatim in the 1.89 GB session store state.db (1,890,656,256 B, 2,901 sessions / 197,682 messages, live 40 MB WAL as at 2026-11-09), which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two enabled Facebook Birthday jobs (0e753e9720fd, fad5a86979d4) and hence in cron/deliveries.db. Nothing rotated, deleted or scrubbed; no value recorded.

  • incidents/2026-11-04-upstream-stream-failure-0700-cohort - dca8482e4f76 + 85edf6772bca died 36 s apart on RuntimeError: Upstream stream failed before completion. after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter).

  • incidents/2026-11-05-skills-index-generator-divergence — RESOLVED — the skills index flip-flop had two writers. The repo wiki-maintain.sh (302 skills / 68 categories) and the stale ~/.hermes/scripts/wiki-maintain.sh copy (349 / 26, because it used Path.rglob() and counted the parked .archive/) were different scripts, and SKILL.md pointed the 23:59 nightly audit at the stale one — so each job overwrote the other's page and the wiki oscillated nightly with no error raised by either. The stale copy is now a delegating wrapper; the repo script gained --no-build so the wrapper honours the old interface without a second generator.

  • incidents/2026-10-31-conduit-voice-local-mode-not-applying — OPEN — the app's "local" voice mode never applies, and the connection test lies. The test checks socket reachability only, while every local turn is still impossible: the Mac mini speech server's LLM proxy is disabled (LLM endpoint → HTTP 501 "Start the server with --enable_llm_proxy"), /v1/usage reads llm_proxy.requests: 0, and its realtime socket accepts the connection but rejects session.update as an unknown event. voice.mode is a phone-side selector — no mode key exists in config.yaml → voice or conduit-settings.json — so a "mode did not apply" cannot be diagnosed on host config. Found via session 20261031_193709_40b9c2 (19:37:09, 30 msgs / 17 tool calls); voice.local_live.front_model was repointed to mistral/voxtral-small-latest during it and nothing was changed on the Mac. → systems/conduit

  • incidents/2026-10-31-gateway-reload-force-drain-orphaned-run — A gateway reload waited its full 1,800 s drain cap, then force-drained and orphaned a live cron run. At 18:06:41 the gateway took the reload; the drain reported timed_out=True at the cap and force-drained at 18:37:38 (new gateway 18:38:43). ea79a2107e36 (Mnemosyne Health Monitor, 12:01:06 slot, 6 h 38 m 09 s in) closed failed at 18:39:15 on "lost its durable fire claim ownership". Same class as incidents/2026-10-26-planned-restart-orphaned-cron-run — second instance, and the first on a plain reload. A second, clean restart followed at 19:23:39 → 19:24:40; 2026-10-31 closed 160 / 4. → systems/scheduled-tasks, systems/hermes-agent

  • incidents/2026-10-31-asana-access-failures — Both Asana paths on CT460 are non-functional. The MCP server asana (https://mcp.asana.com/v2/mcp, auth: oauth, enabled: true) has never authenticated — ~/.hermes/mcp-tokens/ holds only asana.client.json (319 B, mtime 2026-10-26 01:07) and the logs carry "MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run hermes mcp login asana interactively first"; the hosted connector also rejects the call shape. The fix needs an interactive hermes mcp login asana, which a cron context cannot perform. → systems/hermes-agent

  • incidents/2026-10-30-ct460-host-outage-scheduler-gap — RESOLVED (root cause not established). CT460 went dark for 5 h 30 m on 2026-10-30: the PVE host pve rebooted (boot 16:24:00), CT460 returned 16:28:00, the previous gateway life (pid=3913841) was logged as exited UNCLEANLY at 16:35:50 ("no exit path ran — SIGKILL / OOM / VM death"), and the scheduler resumed at 16:37:21 with 8 catch-up runs. Zero execution rows exist between 10:58:20 and 16:37:21; the day still closed 138 / 0 against 10-29's 177 / 0 (≈39 slots lost, no failures). No monitor alerted — the two jobs that could have (cbfe434a3b5e, 1a8e90ea2271) are disabled.

  • incidents/2026-10-30-mnemosyne-health-monitor-paused — RESOLVED 2026-10-30T10:02:48 (its 00:01:08 slot was still running, not dead: it closed completed after 10 h 01 m 40 s and delivered, auto-closing ea79a2_d93c3acdc4f2; ea79a2107e36 is enabled again with last_status: ok / streak 0). The Mnemosyne Health Monitor cron job was auto-paused overnight and its failure notice never reached Telegram — the only trace is the incident row. ea79a2107e36's 2026-10-30 00:01:08 slot closed unknown at 01:04:22 on "Cron execution unverified; reconcile ~/.hermes/cron/admissions/ad897cbc9f…json: Gateway connection failed; no local fallback", opening incident ea79a2_d93c3acdc4f2 (failure_type: unknown, detected) — and the scheduler paused the job at that moment. Fleet is now 29 jobs (25 enabled / 4 disabled); live last_status: error, failure_streak: 1. ⚠ Two traps in one fault: the failure notice itself failed to send (send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed — deliveries.db has zero rows for 2026-10-30), and the run closed unknown rather than failed. The job was not re-enabled and the gateway was not restarted by the stocktake — root cause is unverified. → systems/scheduled-tasks, systems/mnemosyne

  • incidents/2026-10-29-conduit-voice-silence — Every Conduit voice call connected and then died silently; three faults sat in series, and the first two real calls in the product's history were placed on 2026-10-29 after all three were fixed. (A) voice.speech_server.url was the truncated wss://…:8443/v1/ — the plugin dials it verbatim and the bare path answers 403, reported as "the speech server refused the key." (B) dashboard.basic_auth.session_ttl_seconds was the 12 h default, so an expired phone session gets WS 4401 with no Hermes turn and no error on the device (the phone re-authenticated at 08:00:59, minutes after a silent call). (C) Both host providers were dead — tts.provider: openai → Mac /v1/speech 502, stt.provider: groq → 403/Cloudflare 1010 (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → …/v1/realtime, TTL → 2 592 000 s (30 d), stt.provider: local (faster-whisper base), tts.provider: ollama_local; dashboard restarted 08:03:26. Verified by driving the whole call path for real, then by calls at 08:38:04 and 08:49:36. ⚠ Two TTL keys exist — dashboard.basic_auth.session_ttl_seconds (raised) vs dashboard.session_ttl_seconds (still 43 200). status: resolved.

  • incidents/2026-10-28-job-radar-stage1-iteration-cap — The radar's two-stage split ran for real on 2026-10-28 and landed nothing: stage 1 stopped at the agent iteration cap before writing its handoff, so both stage-2 slots correctly no-oped (silent). Two consecutive stage-1 failures, two different ceilings (tool-call budget 10-27, iteration cap 10-28); no Notion writes for a second day. RESOLVED 2026-10-29 — the next stage-1 run (dca8482e4f76, 07:00:59 → 10:16:12, 3 h 15 m 13 s) closed completed and wrote handoff/2026-10-29.json, which stage 2 consumed at 10:40:59 and delivered as 20 pages + 2 cover letters (missing_cover_letter_count 0). status: resolved, review 2026-11-13.

  • incidents/2026-10-27-job-radar-budget-exhausted — The Daily Marketing Job Radar exhausted its tool-call budget for the second day running — and this time the run is recorded as failed, not completed. On 2026-10-27 the 07:00 slot ran 07:01:08 → 11:03:54 (4 h 2 m 46 s) and its delivered report opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)": discovery/verify/dedup/classify finished, Notion writes, cover letters and the final audit never ran (Notion DB held 451 entries, 0 added for the day). Incident row dca848_e93a538ade6b opened 11:03:53 (alerted 11:03:59, failure_type: auth — wrong for a budget fault) and was still open at the 2026-10-28 audit. Response the same day: the radar was split into stage 1 dca8482e4f76 (Discovery, 07:00, deliver: local) and new stage 2 5fc292ff0f3b (Write and Letters, 11:30/14:30, topic 1030), created 17:34:15, two completed test fires that evening → runbooks/job-radar. RESOLVED 2026-10-29: both of this incident's ledger rows (dca848_e93a538ade6b, dca848_0a2f03b5952b) closed resolved at 10:16:12 on the first successful post-split run — dca8482e4f76 restored delivery to local, wrote the handoff inside budget, and no cover letter went missing (missing_cover_letter_count 0). status: resolved, review 2026-11-13.

  • incidents/2026-10-26-planned-restart-orphaned-cron-run — A planned gateway restart orphaned a live cron run — the second instance of the orphan class, and the first that alerted. On 2026-10-26 at 14:33:35 AWST the gateway took a planned SIGTERM (under_systemd=yes); the shutdown drain reported timed_out=True after just 1.35 s of its 30 s budget and interrupted ea79a2107e36 (Mnemosyne Health Monitor, 12:01 slot) 2 h 34 m in. It closed failed on a new string — lost its durable fire claim ownership — raised two cron_incidents rows (14:33:37 unverified / reconcile admission + 14:35:59 fire claim), delivered the failure at 14:35:59, and left the job on failure_streak: 2 from one interrupted fire. Two Mnemosyne Watchdog reds followed at 14:46 and 15:01 and self-resolved at 15:16:12. Net = 169 / 3 for 10-26. Unresolved: why a planned stop bypasses the drain deadline, and whether failure_streak should count the unverified row. → also updates incidents/2026-10-20-gateway-unclean-exit-orphaned-run (the class now recurs without a crash).

  • incidents/2026-10-25-gmail-digest-runaway-and-refire — The Gmail Action Digest ran 1 h 55 m and fired six times on 2026-10-25. The 08:30 slot started 08:31:14 and closed 10:26:37 failed on RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed) — against 91 s (10-23) and 104 s (10-24). The same job then fired five more times (17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53); the 19:18:41 fire died on a new signature — Cron admission missing: the owner never admitted this fire (distinct from the 10-20 orphan) — and all six fires, failures included, carry delivery_outcome: delivered, so up to six digests reached Anthony. Neither failure raised a cron_incidents row; the job now reads last_status: delivery_queued with an empty last_error. Hypothesis (unconfirmed): the 08:31 fire adopted an unrelated long-lived session — its worker made 70 terminal calls on VM101 /dev/sdb backup work with context growing 66 k → 148 k tokens, while context compression was unavailable all day (invalid_grant). → systems/scheduled-tasks, incidents/2026-10-22-nous-portal-refresh-token-invalid

  • incidents/2026-10-22-nous-portal-refresh-token-invalid — Invalid refresh token — Nous Portal runtime credentials are dead, so context summarisation failed. Gateway-log counts: 1 line 2026-10-22, 3 on 10-24, 20 on 10-25 (last 14:53:04, agent.context_compressor: Failed to generate context summary), 0 since the 19:27:13 restart. ⚠ Masked, not fixed: at 2026-10-25 19:13 all eight auxiliary: slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off nous to no-think/freeapi/auto on custom:omni-101 (fallback no-think/freeapi/deepseek-v4-flash → no-think/kg/kilo-auto/free, free_only: true; compression timeout 90 s, vision 180 s). No hermes model re-authentication has been run and nous is still the first entry in fallback_providers, so failover tries a dead credential. → systems/model-providers, systems/current-profile

  • incidents/2026-10-24-conduit-call-request-refused — The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56). He asked to be called to test Conduit calls (Call me now to test calls. → Yes. You do → CAL ME); all three replies asserted the agent has no calling capability, with zero tool calls in the session. conduit_call_user (toolset conduit) exists, is registered for every session, is reachable via tool_search, and the plugin ships the calling-the-user skill; his call settings (enabled / when_asked / decides) were permissive and the device was paired. Root cause: capability asserted instead of discovered — the first reply came from kilo-auto/free, but the switch to smart-route repeated it. Same audit also corrects the 10-22 incident's closing evidence: conduit-calls.json → ends records turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne cron sessions — history (real calls) is empty and no call has been placed since (superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → incidents/2026-10-29-conduit-voice-silence) the memory fix. → systems/conduit, incidents/2026-10-22-conduit-memory-start-timeout

  • incidents/2026-10-22-conduit-memory-start-timeout — Conduit voice calls were silently running without memory, 2026-10-22 → 10-23. The phone plugin gave one request — including the Mnemosyne provider start — a hard-coded 10 s budget (MEMORY_TIMEOUT_S), while Mnemosyne's cold start for a conduit-voice session runs 10–45 s. Three calls were abandoned at the memory layer (Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with no recall of prior conversation. Fixed 10-23 in the plugin's working tree: default 60 s (CONDUIT_MEMORY_TIMEOUT_S override) plus a TokenError guard that logs and proceeds without recall instead of aborting; dashboard restarted 18:54:43 and no abort line exists after it. ⚠ Corrected 2026-10-25: the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a false positive — those were the two Mnemosyne cron turn-ends of that night, and conduit-calls.json → history (real calls) is empty, so the fix has no positive call evidence and no call has been placed since (superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → incidents/2026-10-29-conduit-voice-silence). The patch is uncommitted — a plugin update reverts it. The cold start itself was not optimised. → systems/conduit, systems/mnemosyne

  • incidents/2026-10-21-omniroute-pre-dispatch-filter-skip — A third failure signature from the custom:omni-101 / smart-route pin, 2026-10-21. The Daily Marketing Job Radar (dca8482e4f76, 07:00 weekday) died at 07:40:48 after 40 m 36 s on RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s — no 503, no connection error, no repetition loop, and zero 503 rows exist for that day (last 2026-10-19 09:02). Its 07:00 sibling 85edf6772bca completed normally, so the node was serving; the run wrote no output (cron_incomplete_no_output, 82 msgs / 47 tool calls). Incident dca848_014affeb6954 was closed 2026-10-22 07:59:22 when the next daily slot ran clean. Not caused by the 22:47 hermes update — that landed 15 h later. → systems/omniroute

  • incidents/2026-10-20-gateway-unclean-exit-orphaned-run — The gateway exited uncleanly on 2026-10-20 (09:09:01, suspected_oom=False) and orphaned a live cron run. 08eb86f02d37 (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed unknown — “Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.” No delivery row exists for the slot and no incident row was created; the job table keeps showing the previous slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a 27-day clean run. Open: orphan-recovery on restart, and treating a delivery-targeted unknown as a probable non-delivery.

  • incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence — Third iteration-cap in seven days, and the first that raised no incident at all. Job Radar Stage 1 (dca8482e4f76) ran 07:00:33 → 16:01:52 = 9 h 01 m 19 s on 2026-11-02, hit the agent iteration cap before the handoff was written, and closed status: completed / error: NULL / delivery_outcome: suppressed. Both Stage 2 slots declined correctly (92665337… 11:30 and the 05a3f00b… 14:30 safety net both exited [SILENT]), so nothing reached Notion, no cover letters were generated, and the day's 164 deduped candidates were never persisted — a state a status-only health check reports as green. Contrast 10-27 (dca848_e93a538ade6b) and 10-28 (dca848_0a2f03b5952b), where the identical cap did open rows whose error text is the run's own INCOMPLETE report: on 11-02 the ledger lost the red row it used to produce. Discovery itself worked — SEEK 51, LinkedIn 60 IDs, Indeed live, WA Government 35 (197 raw / 164 new after three-level dedup) — and a URL-canonicalisation bug was found and fixed mid-run (au.seek.com vs au.seek.com.au had been dropping SEEK and Indeed entries from the consolidated list). → systems/scheduled-tasks

  • incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure — The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered: its execution closed unknown at 18:05:13 with "Cron execution unverified … Gateway connection failed; no local fallback", its incident row is still alerted, the job sits at error / failure_streak: 1, and deliveries.db shows no row after 11:01:12 for the rest of the day. Its batch-mate b63ed5817212 took the identical error and self-healed in 68 s. Same page records the inverse silent failure: ecc1fc13655a reads error with last_run 11-01T12:01:00 while owning no execution row and no incident row at all. ✅ RESOLVED 2026-11-02T01:27:21 — the review was never lost. The same execution row (be6ec11a8e68428084af9ab18fb148db) closed completed / delivered at 01:27:22 after 7 h 26 m 43 s, the delivery lands at 01:27:21.802246 and incident 79f9bb_9c443b602eec auto-closed at 01:27:21.606691 — so the 18:05:13 unknown was a reconciler timing artefact on a still-running job, not a verdict (page now resolved). Batch-mate b63ed5 closed in 68 s and 79f9bb in 7 h 26 m, both completed; an unknown row on a long job is a timing artefact, not a failure. ecc1fc13655a's ghost error cleared too (12:00:33 → completed / delivered 12:29:26), taking non-clean last_status 2 → 1.

  • incidents/2026-11-01-cron-worker-sigterm-orphaned-claims — Three restart-safe cron workers killed by SIGTERM (-15) after adopting their claim, so the scheduler recorded "whether side effects ran is unknown": 85edf6772bca @07:30:46, 96eb95ce12f2 + 0e753e9720fd @09:02:18. A fourth variant of the orphan class — and the first with no gateway restart, reload or drain anywhere in the window (PID 175269 up since 10-31 19:24:40; relay counter monotonic through both signals), so it cannot be attributed to reload behaviour. Likely trigger: a 30 s worker-ack window smaller than this host's cold start (83 plugins / 69 enabled discovered in 70 s). All three rows now resolved; one execution left unknown.

  • incidents/2026-11-03-job-radar-stream-active-timeout — The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the ledger went red. dca8482e4f76 ran 07:00:33.819190 → 13:28:52.305995 = 6 h 28 m 18 s on 2026-11-03 and closed failed / RuntimeError: stream active timeout, with the failure notice delivered (13:28:51.741724) and incident dca848_dfd4bea84f2a (failure_type: timeout, alerted, open) — unclosed rows 0 → 1. The run's activity stops at 12:23:28; the harness does not declare the turn dead until 13:28:49 — a 65 m 21 s silence with no tool call, no model turn and nothing in the ledger, and no "maximum number of tool-calling iterations" message anywhere (this is a provider-stream timeout, not the cap that ended 10-27 / 10-28 / 11-02). It compacted five times (08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21) in 363 messages. Both Stage 2 slots declined correctly (5fc292ff0f3b 11:30:36 → 11:32:16; 14:30:32 → 14:33:11) because no handoff was written, and radar-ref/handoff/ still holds only 10-27 / 10-29 / 10-30 — so 11-02's 164 deduped candidates and 11-03's discovery are both unrecovered. Pulse carried it to Anthony in the desktop session. Answers the 11-02 page's open question 3: suppression tracks the terminal state — completed → nothing delivered, failed → notice delivered. → incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence, systems/scheduled-tasks

  • incidents/2026-08-08-gitea-config-backup-push-dead — Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47. e27da6c34bbb reads last_status: ok and its executions read completed every night while 75 commits sit unpushed (fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net' — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (message_id 8447, 2026-10-20 11:00); AgentMail failed. No off-box ~/.hermes backup in 2.5 months. Second instance of the pattern after the GitHub Trending Radar token revocation. → infrastructure/backups

  • incidents/2026-10-18-degenerate-stream-canned-substitute — The repetition-loop failure mode, 2026-10-18: the FreshRSS Midday run (ecc1fc13655a) wrote "Your request was not processed…", delivered it to topic :346 at 12:08:11, then died at 12:08:25 on RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop. A delivered message is not evidence of delivered content — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? Second surface (2026-10-20 18:37): an interactive desktop session received the same canned line after Rebuilt-message restart limit (3) exceeded → reason=rebuilt_restart_limit_exceeded … response_len=0, so the substitute is emitted by the harness's failure path generally, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — was never answered.

  • incidents/2026-10-16-omniroute-503-resource-pressure — omniroute-1 load-sheds model calls with HTTP 503 … resource pressure — two bursts: 2026-10-16 → 10-17, and again 2026-10-19. Burst 1: 9 model calls across 7 jobs over ~19 h, self-cleared (reported then as 10-17 = 162 / 5; the DB reads 163 / 5 — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: 9 model calls across 8 jobs on 2026-10-19 (06:01 + 07:01 → 12:01, whole day 159 / 13). /api/health read 200 ok throughout both. Status: active → back to monitoring on 2026-10-21 — 2026-10-20 ran 168 / 0 with zero rows carrying the resource-pressure string, so the second burst is over; it stays a standing risk until the single-node topology changes. 4 rows closed on 10-20 (dca848_867d9caa96a8, 85edf6_256c5e150bc4, ecc1fc_c9976ea24553, ecc1fc_21a72aa11cc4); unclosed rows 13 → 9 (8 alerted / 1 detected); the ledger still tags every 503 row failure_type: unknown, so triage on error text. Single-node topology still open — and every job that failed reads model: smart-route, provider: custom:omni-101.

  • incidents/2026-10-14-omniroute-node-connection-refused-storm — Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15: omniroute-1 (100.78.63.20:20128, provider custom:omni-101) refused TCP connections, so every model-calling cron job failed on every slot — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. All 18 enabled model-calling jobs ride this one node with no failover. Recovered 10-15 22:01 (resolved)

  • incidents/2026-10-09-cron-provider-timeout-storm — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all RuntimeError: Request timed out. on the aliases behind custom:omni-101, 00:00 → 15:00 AWST. CLOSED — all 12 incident rows are now resolved; the last two (1b23c0_c35703f637b8, 1b23c0_103cbe04a79b, both on the weekly Friday job 1b23c042e17d) closed 2026-10-23 08:04 when that slot ran clean, and the job reads ok again

  • incidents/2026-10-09-memory-recall-relevance — Recall relevance: a stale conversational episode outranked current-state facts on an open question. ✅ RESOLVED 2026-11-01 — the ranking weights were persisted, in ~/.hermes/mnemosyne/config.yaml rather than the file the 2026-10-10 probe read; the original regression query now returns only current-state facts, and conflict 152 was auto-resolved on 2026-10-29

  • incidents/2026-10-05-dead-model-pin-repin — Seven cron jobs on dead model pins (cliprox/terra, qwen3.5:397b, deepseek-v4-flash, gpt-5.6-terra) repinned to routing aliases on custom:omni-101; failing set 7 → 0 (resolved 2026-10-05)

  • incidents/2026-10-03-mnemosyne-vector-recall-fts-only — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic

  • incidents/2026-10-03-ct450-hermes-update — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented

  • incidents/2026-10-01-cliproxy-connection-auth-expired — OmniRoute cliprox/terra connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); closed 2026-10-05 when all three were repinned off the pin

  • incidents/2026-08-26-work-outlook-basic-auth-disabled — Work Outlook mailbox unreachable: tenant allows only XOAuth2, himalaya configured for password (open since 2026-08-26)

  • incidents/2026-09-28-hermes-managed-runtime-deps — Managed-runtime venvs missing dotenv/mnemosyne broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. Recurred 2026-10-10 inside the gateway runtime env — mnemosyne and elevenlabs missing from the committed uv env the gateway loads; both repaired and verified the same evening

  • incidents/2026-08-18-mnemosyne-capture-pipeline — Mnemosyne capture pipeline broken (Aug 2026, superseded): capture verified healthy 2026-10-12 — 786 working rows, +2 that day, 98.3 % embedded

  • incidents/2026-08-15-skills-index-drift — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration

Monitoring & Dashboards

Reports

Infrastructure

Services

Networking

  • infrastructure/proxmox — PVE hosts (two: pve 9.2.10 + m93p 9.2.10, unclustered, IDs collide), VMs, LXC containers, storage (/mnt/ssd0 78.29 % / ~152 GiB free, 2026-11-07)
  • infrastructure/tailscale — Tailscale serve routes (CT460 verified exactly) and the 58-node tailnet inventory (re-verified 2026-11-10: 57 peers + hermes, 42 online; octop back online, 8 offline ages refreshed, 6 ephemeral codex-environment-* peers documented; CT107 mem0 added to the m93p guest list)
  • systems/cloudflare-tunnel — ct207-hosted tunnel serving Hermes publicly

Concepts

Decisions

Comparisons

Runbooks

Raw