Files
agent-estate-wiki/index.md
T
Tony0410 226767f3ec wiki audit 2026-11-20: three-window catch-up (11-17/11-18/11-19, both prior audit slots died)
- new: incidents/2026-11-18-wiki-audit-credit-outage (audit failed 11-18 + 11-19, deliveries suppressed, incident row alerted_at=None)
- new: incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm (115 consecutive slots, 59 unresolved conflicts vs threshold 50)
- corrected: incidents/2026-11-14-mnemosyne-capture-stall + systems/mnemosyne — freeze ENDED 2026-11-18 19:25 AWST (105h52m, no intervention); 'capture dead' reading was conversational silence (every interactive session captured)
- systems/scheduled-tasks: new Count bullet 11-17 173(171/2) / 11-18 172(153/19) / 11-19 176(78/98); fleet 29 (26/3); non-clean last_status 1->4; unclosed incidents 5->4; deliveries 24 (23 delivered / 1 failed)
- index.md feed + incidents + systems lines; log.md dated entry; current-state.md new section; frontmatter dates -> 2026-11-20
2026-11-20 01:02:44 +08:00

202 lines
90 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Anthony's Systems Wiki
type: collection
status: active
created: 2026-07-22
updated: 2026-11-20
verified_on: 2026-11-20
confidence: high
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
---
# Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. last updated: 2026-11-20
>
> **2026-11-20 (three-window audit; live probes 2026-11-20 00:0x – 00:5x AWST) — the memory freeze broke by itself, the alarm did not, and the audits that would have said so had died twice.**
> **The capture stall is over — silently.** `working_memory` **1130 → 1131** / `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST**: **105 h 52 m**, ended with **no intervention**. And the "capture is dead" reading was wrong — `state.db` holds only **three** non-cron sessions in that window (two `desktop` on 11-14, both *before* the 09:33 boundary, then the 11-18 **19:18** session whose turn is the captured row), so **every interactive session was captured**; the alert measures **conversational silence** (age of the newest non-`cron`/`subagent` message). Corrected on [[incidents/2026-11-14-mnemosyne-capture-stall]] and [[systems/mnemosyne]].
> **Two of the three jobs that guard the estate are themselves red — and nobody was told.** The **Wiki Audit failed on 11-18 and 11-19** (chained-provider credit/auth), so **neither night was ever audited** (this run covers three) → [[incidents/2026-11-18-wiki-audit-credit-outage]]; both slots `delivery_outcome: suppressed` and its incident row still reads **`alerted_at: None`**. The **Mnemosyne Watchdog has failed 115 consecutive 15-minute slots** since **11-18 20:01:06** — one durable cause, **59 unresolved conflicts against a threshold of 50** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. Ledger **11-19: 176 rows — 78 `completed` / 98 `failed`** (worst day in the retained window); 11-17 173 (171/2), 11-18 172 (153/19). Deliveries since 11-18 **24 — 23 `delivered` / 1 `failed`** (content, Telegram timeout `2026-11-19 08:34:13`).
> Fleet flat at **29 jobs (26 enabled / 3 disabled)** — a twelfth consecutive window — but non-clean `last_status` **1 → 4**; unclosed `cron_incidents` **5 → 4**. The 03:00 stocktake kept working throughout (11 m 35 s / 23 m 07 s / 24 m 38 s), and the consolidator is still consolidating **0 items — 199 of the last 200 cycles**.
>
> **2026-11-16 (nightly audit; live probes 2026-11-16 23:59 – 2026-11-17 00:27 AWST) — memory dead at 62 h 54 m, and a correction to the 11-15 record.**
> The capture stall is unchanged in every respect but the counters: **1130** / **1249** rows, newest write `2026-11-14 09:32:59` AWST, **five** unclosed `alerted` incidents, monitor `failure_streak` 5, consolidator zero-item streak **120** cycles. → [[incidents/2026-11-14-mnemosyne-capture-stall]]
> **New:** the 11-15 window also carried a **plugin load-timeout storm** — **52** failed loads, **519** discarded registrations (`conduit_push` 474, so its tools and hooks were absent for those processes), **11** `pre_tool_call` + **2** `pre_llm_call` hook timeouts and **10** refused tool calls, none of it logged by the 11-15 audit. → [[incidents/2026-11-15-plugin-load-timeout-storm]]
> Cron fleet flat at **29 jobs**; 11-16 ledger 175 rows (173/2), deliveries 11/11. **Zero** load failures or hook timeouts on 11-16 and 11-17.
>
> **2026-11-15 (nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST) — the memory pipeline is still dead, and the alarm has been understating it.** [[incidents/2026-11-14-mnemosyne-capture-stall]] is now **38 h 46 m** old and **unchanged on every re-read** — `working_memory` **1130** / `episodic_memory` **1249**, *identical to the 11-15 00:4x and 12:03 reads*, newest rows still the `09:32:58`/`09:32:59` `[MODEL_REFRESH_PROPOSAL]` maintenance writes. `cron_incidents` **1 → 3 unclosed** (138 → **141** rows), all this job, all `failure_type: auth`, each `first_seen_at == last_seen_at` — **new mechanism: the signature embeds the elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), so one durable fault mints a fresh row every 12 h and never escalates.** ⚠ **And that ≈4.64 h is not the stall:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so the estate's only alarm has **understated severity on every fire** (11-15 00:03 → 11-14 19:22:30; 11-15 12:03 → 11-15 07:23:25; 11-16 00:03 → 11-15 19:22:59); four 11-15 Pulse turns are uncaptured. The consolidator is alive and every cycle is a **deliberate no-op** (`4696` at `2026-11-15T16:14:14.934306+00:00`, `items_consolidated: 0`, `summary_preview: 'No-op consolidation cycle — timestamp updated for watchdog'`; **~74 consecutive zero-item cycles**) — it bumps its own timestamp, so an age-only gate never notices. **2026-11-15: 168 rows — 165 `completed` / 3 `failed`** (the monitor's `00:00:53` and `12:01:24` slots, plus the watchdog's `11:31:20` slot failing after **17 m 49.8 s** with `runtime deps guard … timed out after 600 seconds`, which skipped the `11:45` slot and self-recovered at `12:01:25`), deliveries **9 / 9 `delivered`** (3 `for_failure`); **2026-11-14 settled 167 / 167 `completed` / 0 `failed`**. Fleet **29 (26 / 3)** — a **tenth** window unchanged; non-clean `last_status` still 1 (`ea79a2107e36`, `error`, `failure_streak` **3**). Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**. No interactive session started on 11-15 — all 9 sessions are `source: cron`.
>
> **2026-11-14 (nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) — a third TTS stack landed on CT200, and the memory pipeline went quiet.** **New page [[systems/phonon-tts]]**: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model **`gradium/phonon-40m-en`**, container created 2026-11-14 10:31:56 AWST), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`, 11:00:11), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`, 11:43:59) — verified live from CT460: `/health` 200, `/v1/models`, and a `POST /v1/audio/speech` returning a valid MP3; `/opt/ptts-voices` holds **7** voices; both ptts containers were restarted together at **17:54 AWST**. ⚠ **Provenance gap**: the retained transcripts cover only 08:35–09:03 AWST, so the 10:31 / 11:00 / 11:43 / 17:54 changes are **not attributable to any session in this profile** (working hypothesis: another estate agent — unproven). **New open incident [[incidents/2026-11-14-mnemosyne-capture-stall]]**: Mnemosyne has captured **nothing since 2026-11-14 09:32:59 AWST** — `working_memory` and `episodic_memory` stop in the same minute, last 8 working writes all 01:01–01:32 UTC — and the health monitor `ea79a2107e36` **escalated only at 2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault, state **`alerted`**, unclosed, alert text printing the raw **UTC** value so it reads as a 23 h gap when the true AWST gap was 14 h 31 m). ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 run saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run's own output block** (monitor log: header 957 = 12:01:52 run, header 1030 = 00:03:35 run, `-589.5` at 1046, ❌ at 1047). The earlier "~12 h of lead time" claim is **withdrawn**, as is the "exited 0 at 12:00 and 18:03" sentence (schedule `0 */12`; there was no 18:03 slot). Detection is **activity-aware** — `mnemosyne_freshness_gate.py` fires on `delay < −120 min` (magnitude-based, no sign-only test) and can only fire once a conversational message postdates the last capture; the conversation was idle 09:03→19:22, so midnight was the first slot able to see the miss. **What actually went unread: 29 consecutive 0-item consolidation cycles** from **10:04:42 AWST**, printed as `(0 items)` every run yet tested only for age (`> 36 h`). ~15 h of desktop (last message 19:22 AWST) / Telegram / cron context is uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established**. **Fleet otherwise clean — until the monitor's own 00:03 alert: 2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**, no other incident row moved, so `cron_incidents` went **0 → 1 unclosed** (the monitor's row, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 (26 / 3)** — a **ninth** window unchanged, but **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13. [[infrastructure/docker-services]] re-verified: CT200 **64 running non-seeder / 72 total** excluding the 1 transient seeder (was 61 / 68 on 11-07) — the delta is **exactly** the three TTS containers, **no removals**. The **2026-11-08 hook cascade** gained **no** new occurrence in the 11-14 body (the 00:05:40 timeout was already recorded by the 11-13 audit; `errors.log` 11-14 = 1 timeout + 10 suppression lines, otherwise only the benign `Browser Use viewing unavailable` plugin warning at ~2/hour).
> **2026-11-13 (nightly audit; live probes 2026-11-13 23:59 - 2026-11-14 00:2x AWST) — the fleet went fully clean, then the hook came back.** **2026-11-13: 174 ledger rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:41) — the **third consecutive zero-`failed` day** (11-11 172/172, 11-12 176/176); deliveries **8 / 8 `delivered`**, 0 `for_failure`; `cron_incidents` **0 unclosed**. **Every job now reads `last_status: ok` with `failure_streak` 0 — the fleet has no non-clean row at all**, because **`066bb5191280` (Uber Eats Monthly Spend Report) was fixed end-to-end on 2026-11-13**, clearing a `delivery_failed` that had stood since 2026-11-01. Two stacked causes: the job's **runtime prompt had grown to 88 k chars** (it carried two heavy browser skills) so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest, and the `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart (**2026-11-13 13:14**). Fix: deterministic printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, rewritten prompt with a hard output contract, skills cleared (**88 k -> 1.8 k**); verified on the 18:51 -> 18:58 slot (**154 unique / 153 completed** orders, Oct **9 orders / $301.04**), lesson saved to `cron-job-maintenance`. Fleet **29 (26 / 3)** — an **eighth** window unchanged. ⚠ **But the window's fault is the `pre_tool_call` hook, and it returned:** `_pre_tool_call timed out after 120s` at **2026-11-14 00:05:40** plus **10** suppression lines through **00:06:38**, refusing this audit's own tool calls for ~1 minute (first-hand). `errors.log` by day: 11-12 **0/0**, 11-13 **0/0**, 11-14 **1/10** — the first such timeout in the retained log since the 2026-11-08 cascade (**`errors.log` now starts 2026-11-09 13:18:39**, so the 11-08 lines have rotated out and the page's 58-count stands on the 2026-11-09 read). **Load is the trigger and it has ~doubled: `load average 12.08 / 13.02 / 12.33` on 2 vCPU** (vs **5.89** at diagnosis), with **`state.db` now 2.04 GB** (**2,977 sessions / 207,577 messages**, was 1.89 GB / 2,901 / 197,682) and the root filesystem **88 % used**. The 30 -> 120 s fix is **proven insufficient**; [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] is **reopened, `resolved` -> `active`**, and `state.db` checkpoint/retention — not a bigger timeout — is now the estate's most urgent open item. Freshness (03:00 stocktake): **0 stale / 131 fresh / 3 no-date of 134**.
>
> **2026-11-11 → 2026-11-12 (nightly audit; live probes 2026-11-12 23:59 – 2026-11-13 00:4x AWST) — a clean day, and an upgrade that could not be installed.** The 429 row that was the estate's only open incident **closed `resolved` at 2026-11-12T00:25:36** (the next slot of the same job ran clean, carrying both windows); *cron_incidents* now reads **0 unclosed of 137**. **2026-11-12: 176 ledger rows — 175 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:54), deliveries **9 / 9 `delivered`**; **11-11: 172 / 172 `completed` / 0 `failed`**, deliveries 7 / 7. Fleet **29 (26 / 3)**, a seventh window unchanged. The window's one new open item is not a failure: the attempt to take **`hermes-hq-push`** past its installed **0.9.42** was **refused by the pre-install plugin security scan** — *"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."* The critical is the plugin's **own** `skills/dashboard-tile/SKILL.md` one-liner; **no per-verdict override exists** (`plugins.scan_on_install: false` would blind the scan for *every* future install, and `plugin_guard.py` hardcodes `trust_level="community"`). The fix is **PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file), pushed from a fork because `gh` is **read-only** upstream. Installed 0.9.42 keeps delivering. Also: the alleged *"stored hold-and-ask instruction"* for scan warnings **does not exist on disk** (184,716-file sweep, zero matches in editable text; the sentence lives only in `state.db` history) — nothing was removed or reverted, and the hunt is now the skill `stored-instruction-search-and-removal` (**skills 299 → 300**). `config.yaml` is **byte-identical to its 2026-11-12 18:14:52 backup**; the only drift since 11-09 is the **model catalogue block** (6 out, 9 in). **New page:** [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]].
>
>
> **2026-11-12 (03:00 stocktake) — the freshness scan was clean, the live-vs-page diff was not.** The age scan flagged **1 stale / 129 fresh / 3 no-date of 133** (the sftpgo incident at 22d, re-verified live and cleared). But [[infrastructure/tailscale.md]] — only **2 days old** — had drifted: **`mailflow`, `qwenpaw` and `springroll` all read `Online: false` in the peer record while all three ping fine** (CT1262 and CT465 `running` in `pct list`; springroll answers via **DERP(sin)**, relay not direct), and `omarchymac` was stale in the *other* direction (page said `offline 2d / last seen 2026-11-08 00:45Z`; live reads `LastSeen 2026-11-07T06:41:48Z` = **offline 4d**, ping timeout). Node table rebuilt by script, all IPs matching, no rows missing, **58 rows = 57 live IPv4 peers + `hermes`**, 80 peers total (57 IPv4 / 23 IPv6-only) unchanged. Also re-probed: CT299 `sftpgo` running/online/healthz-401; both DEAD Serve targets (8769, 9123) still listener-less; CT201's Serve table unchanged; container counts **17 LXCs + 4 VMs on `pve`** and **20 LXCs + 2 VMs on `m93p`**; and **new guest CT1007 `homebridge` on m93p — running but `tailscaled` `inactive`**, so it does not join the tailnet.
>
> **2026-11-12 (03:00 stocktake) — a "router" failure mode turned out to be fleet-level.** [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] recorded *"all targets were skipped by pre-dispatch filters"* as a Job Radar one-off and left three open questions. Live re-read of `executions.db` shows the **same signature on a second job and a second pin**: `cbfe434a3b5e` (Wiki Audit, `custom:omni-101` / **`free-stack`**) failed on **2026-11-10** with `HTTP 429 … (credits 100%)`, incident row `cbfe43_ed01845310d8` (`rate_limit`) closed `resolved` **2026-11-12T00:25**. The original subject has **not** repeated — `dca8482e4f76` ran clean 11-06/09/10/11. So open question 1 (a job-specific mis-configured filter) now looks **less likely** than a router-side guard that blanks every target, and open question 3 (**no failover pin**) applies to the whole free-tier cohort, not just the radar. ⚠ Caveat recorded on the page: `executions.db`'s retention floor is **2026-11-06T05:09:30+08:00**, so the 2026-10-21 row is not re-readable today — the recurrence is an in-window read.
>
> **2026-11-09 (nightly audit — window 2026-11-08) — the day's fault was a hook, not a schedule, and it is fixed.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`), and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** — `errors.log` holds **12** lines on 11-07 and **46** on 11-08, plus **89** *"skipped after previous timeout or while still running"* suppression lines. Every tool call degraded to slow-then-suppressed, which is what stretched the day. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 → 120** in `~/.hermes/config.yaml` — verified live, and verified *effective* (the last timeout line is still 08:21:03, re-checked at audit time). ⚠ **The obvious suspect was measured and acquitted:** the plugin's caching is fine (`session_titles.py:21 CACHE_SECONDS = 30.0`; timed in-process at **3.1 s cold, 0.00 s warm**), so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong** — the trigger is **load** on the 2-core CT460 (`load average 5.89` at 13:27:50, `state.db` checkpointing a **40 MB WAL** on the same spindle) → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. **Cost of the fault:** *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and **12:00:51 → 18:00:42 (5 h 59 m 51 s)** — its two slowest slots on record, both delivered; *FreshRSS Midday Briefing* took **2 h 09 m 53 s**; and **the stocktake failed a third consecutive day** (`1a8e90ea2271`, 03:01:11 → 04:36:08, `failure_streak: 3`) on a **new terminal signature** — `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.` where the previous two red days read *"lost its durable fire claim ownership"*. *Working hypothesis, not proven:* an open tool result is exactly what a suppressed `pre_tool_call` would produce, but the string does not name the hook. **11-08 settled: 169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:33); deliveries **8 / 8 `delivered`**; fleet **29 jobs (26 / 3)** — a **sixth** consecutive audit at the same count with nothing added, retired, disabled or repinned, though `updated_at` moved to `2026-11-09T00:06:50+08:00` as a run-state stamp. **Unclosed incident rows 2 → 3**, all three the stocktake's and all silent (`detected`, `alerted_at: null`) — but **the alerting path is not the fault**: the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` **08:06:54** in the same window. ✅ **Two verified resolutions:** the **Mnemosyne Health Monitor's no-agent conversion works** — its first script slot ran **2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, silent, writing a **159 B** output file against **105,982 B** for the last agent-mode run (coverage live at **working 1030/1030 = 100.0 %, episodic 1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet; the script uses `awk` because **`bc` is not installed on this host**); and the **1.89 GB `state.db`** was re-measured (**1,890,656,256 B**, **2,901 sessions / 197,682 messages**, live **40 MB** WAL) with the 2026-08-11 remnant **`state.db.corrupted.20260811_151052` (624.1 MB) deleted**. **New page:** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
>
> **2026-11-08 (nightly audit — window 2026-11-07) — two runs orphaned mid-restart, and two incident rows that never alerted.** The gateway restart that arrived at **05:01:00** (after a requested restart sat in *after-turn* mode for the full 1800 s) drained two in-flight cron executions; both closed `failed` on `lost its durable fire claim ownership` — *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42 → 05:02:22) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31 → 05:02:22, **second consecutive failed day**, `failure_streak: 2`). The restart was requested at ~04:29:52 — **inside the `conduit_push` self-update window** (`0.122.2 → 0.134.4`, 04:27:55→04:32:32, its own ledger recording `restarting_gateway` = *failed*) — working hypothesis, not proven; the 0.22.5 → 0.23.0 updater is excluded (its receipt starts 05:05:13). Day counts: **163 rows — 159 `completed` / 3 `failed` / 1 `running`**; the third failure is the agent-log watchdog correctly reporting the event 19 s later. ⚠ **Both stocktake incident rows are still `detected` with `alerted_at: null`** — closed 0 → 2, and neither alerted. Elsewhere in the window: **Hermes 0.22.5 → 0.23.0** on CT460 (snapshot stage **skipped again** — still no rollback point), the `camofox` browser backend arrived as an **enabled plugin (1.1.0)**, `conduit_push` self-updated to **0.134.4**, `hermes-hq-push` reinstalled **0.9.27 → 0.9.42** (routes reloaded by the 11:11 restart), and the **OpenCode Go credential is expired** (HTTP 401 `api_key_expired` — configuration alive, credential dead, nothing consuming it yet). New page: [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]].
>
> **2026-11-05 (stocktake) -** **the `systems/skills-index.md` flip-flop is fixed at the root**: two different scripts were writing the page, and the nightly audit was pointed at the stale one; it is now a delegating wrapper -> [[incidents/2026-11-05-skills-index-generator-divergence]]. Docker inventory corrected against live probes on all five hosts - ~~the `worldmonitor-seeder-scheduled` container recorded yesterday **does not exist on any host**~~ **⚠ CORRECTED 2026-11-06: false - the seeders are transient systemd-timer-driven containers, absent only between runs, not removed.** Tailscale node table re-verified programmatically - 52 rows, 52 live peers, ~~every state matching~~ **⚠ CORRECTED 2026-11-06: `anthonys-mac-mini` has since come back online.** Freshness **0 stale / 122 fresh / 3 no-date of 125**.)
>
> **2026-11-07 (nightly audit - window 2026-11-06) - a red row with no alert:** 2026-11-06 settled **172 execution rows across 16 jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit). The single failure is a **new, non-restart signature**: the Job Radar's **11:30 slot** (`5fc292ff0f3b`, 11:30:32 -> 13:26:44) was **discarded as stale** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job (08:27:09 -> 13:26:22, 4 h 59 m 13 s)** was still holding the claim and delivered first. **Every earlier occurrence of this string came from a restart/reload orphaning a run; this one had no restart** -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. It raised **no `cron_incidents` row** (newest ledger activity remains the 2026-11-05T12:36:17 closures) and **nothing alerted**; the failed run left no output file. The day still landed in full via the 14:30 slot (**12 m 27 s**, `SAME_DAY_STRONG_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`, cache **494**, *Run status: COMPLETE*). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; fleet **29 jobs (26 / 3)** - **fifth consecutive audit at the same count**, nothing added/retired/disabled/repinned, non-clean `last_status` still **1** (`066bb5191280`). **The stocktake's overrun fully unwound** (`1a8e90ea2271`, 03:00:32 -> 04:11:35 = **1 h 11 m 03 s** against 7 h 40 m / 5 h 41 m / 3 h 02 m on 11-03 -> 11-05) and **Stage 1 ran clean for a second day** (`dca8482e4f76`, 1 h 28 m 37 s, `suppressed`), clearing the 11-04 Open entry in substance. The last audit's open question closed itself: the **Mnemosyne Health Monitor**'s 11-05 12:00:45 slot (12 h+ in flight, undelivered at the last audit) **completed at 2026-11-06T00:14:39 after 12 h 13 m 54 s and delivered**; its 11-06 slot took **7 h 47 m 56 s** and delivered too - ballooned on consecutive days, never lost, **watching, not open**. The one open page remains [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (**untouched in-window**).
>
> **2026-11-06 (nightly audit - window 2026-11-05) - a clean ledger day and an empty incident ledger:** 2026-11-05 settled **175 rows - 173 `completed` / 2 `running` / 0 `failed`** across 18 jobs, the **first zero-`failed` day since 2026-11-02**. The only two `running` rows are the **Mnemosyne Health Monitor** (12:00:45 slot, still in flight **12 h+** at audit time) and this audit itself. **Unclosed incident rows 3 -> 0** - the *Newsletter Digest* row closed at **07:07:42** and both *Job Radar - Discovery* rows at **12:36:17**, when `dca8482e4f76` finally completed after **5 h 35 m 35 s**; that closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, `delivery_failed`, streak 0); deliveries **10 rows / 10 `delivered`**; the 03:00 stocktake overran **3 h 01 m 58 s** (down from 7 h 40 m and 5 h 41 m) and **committed `2f3fc99` at 05:55**. **Fleet unchanged at 29 jobs (26 / 3)** - a fourth audit at the same count. ⚠ **New and open:** live `.env` values (`TELEGRAM_BOT_TOKEN`, `MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) sit verbatim in **17 rows of `mnemosyne.db`** (`memories`, `episodic_memory`, `working_memory`, `gists`, `memoria_facts.context_snippet` and the FTS mirrors) - reproduced in full by each of the **7 uncompressed rotation snapshots** - and all four values sit verbatim in the **1.7 GB session store `state.db`** too; `CAMOFOX_API_KEY` sits verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and therefore in `cron/deliveries.db` -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
>
> **2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze:** the **11-04 fleet settled 170 rows - 167 `completed` / 2 `failed` / 1 `running`**, **non-clean `last_status` 2 -> 3**, **unclosed incident rows 1 -> 3**, **deliveries 8 rows / 8 `delivered`**. Both failures are the *same* upstream fault: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**) died **36 seconds apart at 09:41-09:42**, each after **2 h 41 m**, on `RuntimeError: Upstream stream failed before completion.` - both pinned `custom:omni-101` / `smart-route`. **The day was not lost:** the 11:30 Stage 2 slot **reconstructed the missing handoff from Stage 1's own candidate files**, re-verified every candidate live, and still wrote **5 roles (board 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at 16:08:59 -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. The cover-letter route survived only on an override: `regenerate_cover_letter.py` defaults to `deepseek-paid-smart`, which threw **HTTP 402** mid-run, so the letter was pinned to `gemini/gemini-3.5-flash` via `CL_MODEL` - and that override **was never written to `.env`**, so it did not persist. And **CT299 was frozen twice, not once**: a second, **manual** `vzdump 299 ... --storage ssd0` wedged from **11-04 08:30:04** until **11-05 ~00:30** (~16 h); the live probe at 00:54:21 AWST shows `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. ⚠ **Corrected the same night:** the first draft of this entry read the fleet as **158 jobs (144 / 14)** and the
> 11-04 ledger as closing `1 item_suppressed`; live reads give **29 jobs (26 / 3)** and a final row of **1 `running`**
> (this audit). Keyed probe **2026-11-05 01:33**: `deepseek-paid-smart` is **undispatchable on `omniroute:20128`**
> (HTTP **503**, all targets skipped) but **serves normally on `omniroute-1`** (HTTP **200** -> `deepseek/deepseek-v4-flash`).
> **2026-11-04 (nightly audit) — the ledger raised its own red row:** the **11-03 fleet settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) — **unclosed incident rows 0 → 1**, non-clean `last_status` **1 → 2**, deliveries **7 rows / 7 `delivered`**. The single failure is the **Job Radar Stage 1** (`dca8482e4f76`), which ran **07:00:33 → 13:28:52 (6 h 28 m 18 s)** and died on **`RuntimeError: stream active timeout`** — a **new terminal signature**, not the iteration cap — after going **silent for 65 m 21 s**; its incident `dca848_dfd4bea84f2a` (`timeout`) is **`alerted` and open** and the notice **was delivered** (13:28:51.741724) → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. That answers the 11-02 page's open question: **output is suppressed when the run ends `completed`, delivered when it ends `failed`** — which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so **2026-11-02's and 2026-11-03's candidate sets are both unrecovered**. ⚠ **The 03:00 stocktake overran for a second day** (`1a8e90ea2271`, 03:00:30 → 10:41:13 = **7 h 40 m 43 s**, `suppressed`, nothing delivered, artefacts uncommitted again). **New page:** [[systems/hermes-hq-push]] (v0.9.27, enabled; manifest says it **replaces `dispatch-push`**, and no trace of the old plugin survives).
>
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.10), was discovered during the 2026-10-14 stocktake.
> Guest IDs collide between the two hosts.
## Start here
- [[current-state]] — Current systems status
- [[systems/hermes-agent]] — Hermes Agent overview and version
- [[systems/current-profile]] — Active Hermes profile details
## Systems
### Core
- [[systems/hermes-agent]] — Agent version, install, config paths
- [[systems/current-profile]] — Active profile details
- [[systems/nextcloud]] — Nextcloud file sharing (CT270)
- [[systems/guanaco]] — Guanaco LLM proxy on CT205 (Ollama stack)
- [[systems/model-providers]] — LiteLLM + OmniRoute providers, routing setup
- [[systems/memory-backend]] — Mnemosyne vector memory, settings
- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence. **As of 2026-11-20: capture broke its 105 h 52 m freeze at 2026-11-18 19:25 AWST with no intervention, and every interactive session in the window was captured — the "stale pipeline" alert measures conversational silence, not a dead hook** (the gate reads the age of the newest non-`cron`/`subagent` message); monitor still failing on that metric (`failure_streak: 2`), consolidator **199 of the last 200 cycles zero-item**, unclosed `cron_incidents` **4**; the watchdog's own fault is **59 unresolved conflicts vs its threshold of 50**.
- [[systems/phonon-tts]] — **Phonon TTS stack on CT200** (`:8881` OpenAI-compatible API, `:8882` WASM demo, `:8883` WebSocket; added 2026-11-14)
- [[systems/omniroute]] — OmniRoute catalog, combos, and host path
- [[systems/openrouter]] — OpenRouter current state and history
- [[systems/opencode-go]] — OpenCode Go subscription, model limits, pricing
- [[systems/nanobot]] — Nanobot current state and history
- [[systems/browser-backend]] — Camofox (primary, `systemd --user` on CT460, REST `:9377`) — **the only backend**; **Steel was fully removed from the stack on 2026-10-29** (plugin deleted, `cloud_provider` back to `camofox`, CT201 and CT333 deployments torn down, Serve routes deleted; `STEEL_BASE_URL` survives only in already-running process environments); memory-containment drop-in and the 2026-10-12 ceiling event documented
- [[systems/hermes-ui]] — Browser Hermes Desktop (przbadu/hermes-ui)
- [[systems/paseo]] — Paseo coding-agent daemon + web terminal workspace (CT460, :6767)
- [[systems/ignis]] — Ignis browser-native Obsidian (wiki interface)
- [[systems/terminal-backend]] — Shell persistence, working directory
- [[systems/tool-search]] — Always-on lazy discovery for external tools
### Integrations
- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.122.2` (HEAD `9eac857`, 2026-10-31 12:04 — *"Every profile that turns Conduit on runs the main install (#482)"*; **+148 commits** on since the `0.69.2` / `942ddb8` baseline, **working tree clean**)**, upstream `kaishi00/hermes-conduit-notifier`. **Gateway on `:9119` is now pid 176771, started 2026-10-31 19:35:02** (superseded: restarted 2026-10-25 18:42:33, which is when the upstream commits went live). One registered device (iPhone app **2.2.0**). ✅ **The first two real calls in the product's history are now on record — 2026-10-29 08:38:04 and 08:49:36** — after three faults in series were repaired (truncated speech-server URL → `…/v1/realtime`; 12 h dashboard session TTL → 30 d; dead `groq` STT → `local`, dead `openai` TTS → `ollama_local`) → [[incidents/2026-10-29-conduit-voice-silence]]. **So the memory fix is no longer evidence-free.** ✅ **That patch is gone** — the tree is clean and **`MEMORY_TIMEOUT_S` ships upstream** in `dashboard/api/memory.py`. ⚠ **New fault: "local" voice mode never applies** — the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **501**, `/v1/usage` reads **`llm_proxy.requests: 0`**, realtime socket rejects `session.update`), while the app's "test connection" checks **socket reachability only** → [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]]. The call store was re-shaped: **`history` is empty** and `last_request_at` is `null`, so the two 10-29 calls are no longer in it. The 10-24 refusal fault has not recurred → [[incidents/2026-10-24-conduit-call-request-refused]].
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26** (peer offline **36 d**, `tx 312 / rx 0`; host answers neither `:8065` nor ICMP). **268** connect timeouts on 10-25 vs 264 on 10-24; the reconnect **attempt counter reset to 54** at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost.
- [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip.
- [[systems/hermes-hq-push]] — **Hermes HQ iPhone app plugin** (standalone, **v0.9.42**, `kind: standalone`, author *Hermes HQ*; tree mtime **2026-11-08 02:19**). `hermes plugins list` → **enabled** (state `user`); `config.yaml` → `plugins.enabled` (line **13226** as of 2026-11-13; the number drifts with each catalogue write). ⚠ **2026-11-12: the upgrade past 0.9.42 is blocked by the pre-install security scan** (dangerous verdict, one critical in the plugin's own tile skill) — see [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]]. Host side of the Hermes HQ app: Apple push (finished turns, lock-screen approvals, "waiting for a password/login/code" alerts, **live Lock Screen cards** sealed to the phone), phone **device access** (contacts, location, reminders, alarms, photos, Health, motion, music), the desktop-app **Hermes HQ theme**, the app's **Terminal page**, fast **Sessions list** on a busy host, cross-posting answers back to Telegram/Discord, a **restart guard**, and **Grok Live** voice-token minting (`XAI_API_KEY` or subscription sign-in; bearer never reaches the phone). Tools `device_contacts_search` / `device_request_result`; 7 hooks. Its manifest states it **replaces `dispatch-push`** — and the replacement is clean: **no `dispatch-push` reference survives in `config.yaml`** and no such directory remains. **Different stack** from the Cadu/Vory bridge on [[systems/push-notifications]]. ⚠ Its `pre_tool_call` guard also **blocks any tool call whose arguments contain the literal text of a launchd one-off-restart command** — describe it, never quote it.
- [[systems/scheduled-tasks]] — **As of 2026-11-20 (three-window audit 11-17 / 11-18 / 11-19, because both prior audits died): fleet 29 (26 / 3), a twelfth consecutive window at the same count; non-clean `last_status` 1 → 4 (Wiki Audit `error` 2, Watchdog `error` 115, Health Monitor `error` 2, Gmail Action Digest `delivery_failed`); ledger 11-17 173 (171/2), 11-18 172 (153/19), 11-19 176 (78/98); deliveries since 11-18 24 (23 `delivered` / 1 `failed` — a content Telegram timeout 11-19 08:34:13); `cron_incidents` unclosed 5 → 4.** Earlier — **As of 2026-11-14 (window 11-14): 167 rows / 166 `completed` / 0 `failed` / 1 `running` (this audit); deliveries 4 / 4 `delivered`; `cron_incidents` 0 → 1 unclosed (138 total) — the Mnemosyne Health Monitor wrote the estate's first red row since 11-13 (`last_status: error`, streak 1); fleet 29 (26 / 3).** Earlier — **As of 2026-11-12 (window 11-12): 176 rows / 175 `completed` / **0 `failed`** / 1 `running` (this audit); deliveries 9 / 9 `delivered`; unclosed incident rows **0 of 137** — the 429 row closed `resolved` at 2026-11-12T00:25:36; fleet 29 (26 / 3).** Earlier — **As of 2026-11-09 (window 11-09): 174 rows / 173 `completed` / **0 `failed`** — the first `failed`-free day since 11-05; deliveries 8/8 `delivered`; unclosed incident rows 3 → 0; the stocktake closed `completed` after 9 h 10 m with **zero edits landed**; Job Radar Stage 1 overran 9 h 09 m and landed nothing (new incident).** Earlier: **As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), and `ea79a2107e36` is ENABLED again (`last_status: ok`, streak 0) after its 00:01:08 slot closed `completed` at 10:02:48; deliveries 8/8 `delivered`.** **Prior window — as of 2026-10-30 03:3x:** 29 jobs (25 enabled, 4 disabled); non-clean `last_status` 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all `delivered`. ⚠ `ea79a2107e36` (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed `unknown` on a gateway-connection failure — and its failure notice was NEVER DELIVERED (`deliveries.db` has zero rows for 2026-10-30), so the incident row is the only trace → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. Every other previously-red job has cleared to `ok`/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
- [[systems/skills-index]] — Installed skills inventory (**300 skills** (**67 categories**) as of the **2026-11-13** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
## Incidents
- [[incidents/2026-11-18-wiki-audit-credit-outage]] - **OPEN — the nightly audit failed on two consecutive nights, so 2026-11-17 and 2026-11-18 were never audited, and no failure notice was ever sent.** `cbfe434a3b5e` (Wiki Audit) ran **2026-11-18T00:00:08 → 00:01:43** and **2026-11-19T00:00:16 → 00:03:02**, both closing `failed` on a chained-provider **credit/auth** fault (`failure_streak: 2`, last clean run 11-17). Both slots carry **`delivery_outcome: suppressed`**, so the estate's own auditor was the one thing nobody heard about; incident row `cbfe43_fa12cca1ad4b` (`failure_type: auth`, opened 2026-11-18 00:01:42) is still `alerted` with **`alerted_at: None`** — the same silent-skip shape as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]. The 2026-11-20 00:00 slot (this run) covers all three windows.
- [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] - **OPEN — 115 consecutive watchdog failures from one cause: the conflict queue.** `96eb95ce12f2` (Mnemosyne Watchdog) has failed **every** 15-minute slot since **2026-11-18 20:01:06**, reaching **`failure_streak: 115`** at **2026-11-20 00:31:16** (`last_status: error`), because **59 unresolved conflicts sit above its threshold of 50** — a single durable trip, not 115 faults. Each slot delivers a *failure notice* (9 of the 24 deliveries since 11-18 are `for_failure: 1`), so the noisy channel is also the only channel; every slot re-opens incident `96eb95_98ade422d195` and nothing escalates or closes.
- [[incidents/2026-11-14-mnemosyne-capture-stall]] - ⚠ **OPEN (corrected 2026-11-20) — the freeze ended by itself at 2026-11-18 19:25 AWST after 105 h 52 m, and the "captured nothing" reading was a quiet estate, not a broken hook.** Earlier headline: **Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST (~15 h at this read), and the health monitor took 14 h 31 m to say so.** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); the `pre_llm_call` capture hook logs `Mnemosyne capture ok` on every desktop turn while writing **0** rows, so the hook's own success is not evidence of capture. The monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault; this job's twelve earlier rows read `agent` / `timeout` / `unknown` — state **`alerted`**, unclosed), and its alert text prints the raw **UTC** instant, so it reads as a 23 h gap when the true AWST gap was 14 h 31 m. ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 AWST run saw nothing** (`capture delay 29.4 min`, exit 0, `suppressed`) — the `-589.5 min` ℹ️ line belongs to the **00:03:35** run itself (monitor log lines 957 / 1030 / 1046). Detection is **activity-aware** (`mnemosyne_freshness_gate.py` fires on `delay < −120 min`, i.e. on **magnitude**) and the first message to postdate the stall arrived at **19:22 AWST**, after that slot — so the ~12 h of quiet was the `0 */12` **cadence**, not a threshold bug. **What went unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, visible as `(0 items)` in every run but tested only for age (36 h). Consequences: ~15 h of desktop / Telegram / cron context uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established** → [[systems/mnemosyne]], [[systems/phonon-tts]].
- [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]] - **OPEN — the `hermes-hq-push` upgrade past v0.9.42 was refused by the pre-install plugin security scan.** `hermes plugins install mrcharlesiv/hermes-hq-plugins/hermes-hq-push --enable --force` returned **"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."** The sole critical is the plugin's **own** `skills/dashboard-tile/SKILL.md:41` shell one-liner (`echo '<json>' | python3 …` = *obfuscation*); the other 38 findings are low/medium and alone would have installed as `caution`. **`--force` lifts `caution` only**; a `dangerous` verdict is an unconditional block (`tools/plugin_guard.py:406-418`), and **no per-verdict override exists** — the only key is `plugins.scan_on_install` (default `true`, currently unset), which disables scanning for *every* future install; the `trust_level`/allowlist machinery lives in `tools/skills_guard.py` (the skills path) and `plugin_guard.py` hardcodes `trust_level="community"`. **Installed state unchanged:** v0.9.42, `enabled`, still delivering. **Fix = PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc` → `main`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file, created 2026-11-12T10:00:01Z) — pushed from a fork because this host's `gh` account has **read-only** access upstream; local clone `/home/hermes/workspace/hq-plugins-fix` (`90c9d28`).
- [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **RESOLVED 2026-11-12T00:25:36** (self-cleared on the next slot's clean completion; *cron_incidents* now 0 unclosed of 137) **— the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and is **still open with `alerted_at: NULL` — the estate's only open incident row**. Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]).
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - ⚠ **REOPENED 2026-11-14 — the hook was the fault, and the 30 → 120 s fix did not hold at current load.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** ⚠ **Recurred 2026-11-14 00:05:40** — **1 timeout + 10 skips**, overrunning the **120 s** budget at `load average 12.08` on 2 vCPU (**~2×** the 5.89 at diagnosis), with `state.db` grown to **2.04 GB**. Status **`resolved` → `active`**; the lever is now urgent. → [[systems/conduit]], [[systems/hermes-agent]]
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]] (resolved 2026-11-11) - **RESOLVED BY ITSELF.** The nightly stocktake closed `completed` after **9 h 10 m** having landed **zero edits and no commit** — `failure_streak` 3 → 0 while the work stayed undone, and the top-priority item on its brief had been the monotonically worsening failure streak. Third consecutive day its findings never landed.
- [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]] - **NO INCIDENT RAISED.** Job Radar Stage 1 overran to **9 h 09 m**, skipped its own verification, wrote **no handoff file**, and both Stage-2 slots ran before it finished and declined silently. No incident row was created — the completion-time-only alerting gap leaves the failure invisible.
- [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] - **OPEN.** A restart requested at **~04:29:52** sat in *after-turn* mode for the full **1800 s**, then timed out and drained at **05:00:43/47** — the same `#77184` behaviour as the 10-31 reload, but this time it killed **two** in-flight cron executions: *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42.918 -> 05:02:22.870) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31.515 -> 05:02:22.174), both closed **`failed`** on `lost its durable fire claim ownership`; systemd restarted the unit at **05:01:00** (`restart counter is at 3`). A **third** failure that night was the agent-log ERROR watchdog correctly catching the event 19 s later. The request landed **inside the `conduit_push` self-update window** (04:27:55 -> 04:32:32, its own ledger recording `restarting_gateway` = *failed*) — the **working hypothesis for the requester, not proven**; the 0.22.5 -> 0.23.0 updater is excluded (receipt starts 05:05:13). ⚠ **Closed 0 -> 2 incident rows and neither alerted**: both `1a8e90_*` rows sit `detected` with `alerted_at: null`. The stocktake slot has now failed **two days running** (`failure_streak: 2`). Job Radar ran clean. → [[systems/scheduled-tasks]], [[systems/hermes-agent]], [[systems/conduit]], [[systems/mnemosyne]]
- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - **OPEN.** A `direct`-source fire of `5fc292ff0f3b` (08:27:09 -> 13:26:22, 4 h 59 m 13 s, delivered) settled the claim while the scheduler's **own 11:30:32 slot** was still running, so that slot was interrupted at **13:26:38** and closed **`failed`** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` - a **new string variant** and the first claim-loss failure **with no gateway restart involved**. It raised **no `cron_incidents` row**, so a `failed` execution with no alert was the day's only red row; the discarded run left no output file. The 14:30 slot recovered the day in a normal **12 m 27 s** cycle. Candidate fixes (slot suppression when a claim is live, consuming the next slot for out-of-band fires, raising an incident row on discard) are listed on the page and **none was applied**.
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **OPEN.** Live `.env` values were found stored verbatim in **17 rows of `mnemosyne.db`** (`memories` 4, `episodic_memory` 3, `working_memory` 1, `gists` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the **7 uncompressed rotation snapshots**, and all four values also sit verbatim in the **1.89 GB session store `state.db`** (**1,890,656,256 B**, **2,901 sessions / 197,682 messages**, live **40 MB** WAL as at 2026-11-09), which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Nothing rotated, deleted or scrubbed; no value recorded.
- [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] - `dca8482e4f76` + `85edf6772bca` died 36 s apart on `RuntimeError: Upstream stream failed before completion.` after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter).
- [[incidents/2026-11-05-skills-index-generator-divergence]] — **RESOLVED — the skills index flip-flop had two writers.** The repo `wiki-maintain.sh` (302 skills / 68 categories) and the stale `~/.hermes/scripts/wiki-maintain.sh` copy (349 / 26, because it used `Path.rglob()` and counted the parked `.archive/`) were **different scripts**, and SKILL.md pointed the 23:59 nightly audit at the stale one — so each job overwrote the other's page and the wiki oscillated nightly with **no error raised by either**. The stale copy is now a **delegating wrapper**; the repo script gained `--no-build` so the wrapper honours the old interface without a second generator.
- [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]] — **OPEN — the app's "local" voice mode never applies, and the connection test lies.** The test checks **socket reachability only**, while every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **HTTP 501** *"Start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`**, and its realtime socket **accepts the connection but rejects `session.update` as an unknown event**. `voice.mode` is a **phone-side** selector — **no `mode` key exists** in `config.yaml → voice` or `conduit-settings.json` — so a "mode did not apply" cannot be diagnosed on host config. Found via session `20261031_193709_40b9c2` (19:37:09, 30 msgs / 17 tool calls); `voice.local_live.front_model` was repointed to `mistral/voxtral-small-latest` during it and **nothing was changed on the Mac**. → [[systems/conduit]]
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]] — **A gateway reload waited its full 1,800 s drain cap, then force-drained and orphaned a live cron run.** At **18:06:41** the gateway took the reload; the drain reported `timed_out=True` at the cap and force-drained at **18:37:38** (new gateway **18:38:43**). `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:01:06 slot, **6 h 38 m 09 s** in) closed **`failed`** at **18:39:15** on *"lost its durable fire claim ownership"*. Same class as [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **second instance, and the first on a plain reload**. A **second, clean** restart followed at **19:23:39 → 19:24:40**; 2026-10-31 closed **160 / 4**. → [[systems/scheduled-tasks]], [[systems/hermes-agent]]
- [[incidents/2026-10-31-asana-access-failures]] — **Both Asana paths on CT460 are non-functional.** The MCP server `asana` (`https://mcp.asana.com/v2/mcp`, `auth: oauth`, `enabled: true`) has **never authenticated** — `~/.hermes/mcp-tokens/` holds only `asana.client.json` (**319 B**, mtime 2026-10-26 01:07) and the logs carry *"MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run `hermes mcp login asana` interactively first"*; the hosted connector also rejects the call shape. The fix needs an **interactive** `hermes mcp login asana`, which a cron context cannot perform. → [[systems/hermes-agent]]
- [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]] — **RESOLVED (root cause not established).** CT460 went dark for **5 h 30 m on 2026-10-30**: the PVE host `pve` rebooted (boot **16:24:00**), CT460 returned **16:28:00**, the previous gateway life (`pid=3913841`) was logged as exited **UNCLEANLY** at **16:35:50** (*"no exit path ran — SIGKILL / OOM / VM death"*), and the scheduler resumed at **16:37:21** with **8 catch-up runs**. **Zero execution rows exist between 10:58:20 and 16:37:21**; the day still closed **138 / 0** against 10-29's 177 / 0 (≈39 slots lost, no failures). **No monitor alerted** — the two jobs that could have (`cbfe434a3b5e`, `1a8e90ea2271`) are disabled.
- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — **RESOLVED 2026-10-30T10:02:48** (its 00:01:08 slot was still *running*, not dead: it closed `completed` after 10 h 01 m 40 s and delivered, auto-closing `ea79a2_d93c3acdc4f2`; `ea79a2107e36` is enabled again with `last_status: ok` / streak 0). **The Mnemosyne Health Monitor cron job was auto-paused overnight and its failure notice never reached Telegram — the only trace is the incident row.** `ea79a2107e36`'s 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, `detected`) — and the scheduler **paused the job at that moment**. Fleet is now **29 jobs (25 enabled / 4 disabled)**; live `last_status: error`, `failure_streak: 1`. ⚠ **Two traps in one fault:** the failure notice itself failed to send (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed` — `deliveries.db` has **zero rows for 2026-10-30**), and the run closed `unknown` rather than `failed`. The job was **not re-enabled** and the gateway was **not restarted** by the stocktake — root cause is **unverified**. → [[systems/scheduled-tasks]], [[systems/mnemosyne]]
- [[incidents/2026-10-29-conduit-voice-silence]] — **Every Conduit voice call connected and then died silently; three faults sat in series, and the first two real calls in the product's history were placed on 2026-10-29 after all three were fixed.** (A) `voice.speech_server.url` was the truncated `wss://…:8443/v1/` — the plugin dials it **verbatim** and the bare path answers **403**, reported as *"the speech server refused the key."* (B) `dashboard.basic_auth.session_ttl_seconds` was the **12 h default**, so an expired phone session gets **WS 4401** with no Hermes turn and no error on the device (the phone re-authenticated at 08:00:59, minutes *after* a silent call). (C) Both host providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq` → **403/Cloudflare 1010** (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → `…/v1/realtime`, TTL → **2 592 000 s (30 d)**, `stt.provider: local` (faster-whisper `base`), `tts.provider: ollama_local`; dashboard restarted 08:03:26. Verified by driving the whole call path for real, then by calls at **08:38:04** and **08:49:36**. ⚠ Two TTL keys exist — `dashboard.basic_auth.session_ttl_seconds` (raised) vs `dashboard.session_ttl_seconds` (**still 43 200**). `status: resolved`.
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] — **The radar's two-stage split ran for real on 2026-10-28 and landed nothing: stage 1 stopped at the agent iteration cap before writing its handoff, so both stage-2 slots correctly no-oped (silent). Two consecutive stage-1 failures, two different ceilings (tool-call budget 10-27, iteration cap 10-28); no Notion writes for a second day.** **RESOLVED 2026-10-29** — the next stage-1 run (`dca8482e4f76`, 07:00:59 → 10:16:12, 3 h 15 m 13 s) closed `completed` *and* wrote `handoff/2026-10-29.json`, which stage 2 consumed at 10:40:59 and delivered as **20 pages + 2 cover letters** (`missing_cover_letter_count 0`). `status: resolved`, review 2026-11-13.
- [[incidents/2026-10-27-job-radar-budget-exhausted]] — **The Daily Marketing Job Radar exhausted its tool-call budget for the second day running — and this time the run is recorded as `failed`, not `completed`.** On 2026-10-27 the 07:00 slot ran **07:01:08 → 11:03:54** (4 h 2 m 46 s) and its delivered report opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery/verify/dedup/classify finished, **Notion writes, cover letters and the final audit never ran** (Notion DB held 451 entries, 0 added for the day). Incident row `dca848_e93a538ade6b` opened 11:03:53 (`alerted` 11:03:59, `failure_type: auth` — wrong for a budget fault) and was still open at the 2026-10-28 audit. **Response the same day:** the radar was split into stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) and new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030), created 17:34:15, two `completed` test fires that evening → [[runbooks/job-radar]]. **RESOLVED 2026-10-29:** both of this incident's ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12** on the first successful post-split run — `dca8482e4f76` restored delivery to `local`, wrote the handoff inside budget, and no cover letter went missing (`missing_cover_letter_count 0`). `status: resolved`, review 2026-11-13.
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **A *planned* gateway restart orphaned a live cron run — the second instance of the orphan class, and the first that alerted.** On 2026-10-26 at **14:33:35 AWST** the gateway took a planned `SIGTERM` (`under_systemd=yes`); the shutdown drain reported **`timed_out=True` after just 1.35 s** of its 30 s budget and interrupted `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) **2 h 34 m** in. It closed **`failed`** on a **new** string — `lost its durable fire claim ownership` — raised **two** `cron_incidents` rows (14:33:37 *unverified / reconcile admission* + 14:35:59 *fire claim*), **delivered** the failure at 14:35:59, and left the job on **`failure_streak: 2`** from one interrupted fire. Two Mnemosyne Watchdog reds followed at 14:46 and 15:01 and self-resolved at **15:16:12**. Net = **169 / 3** for 10-26. Unresolved: why a **planned** stop bypasses the drain deadline, and whether `failure_streak` should count the unverified row. → also updates [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] (the class now recurs without a crash).
- [[incidents/2026-10-25-gmail-digest-runaway-and-refire]] — **The Gmail Action Digest ran 1 h 55 m and fired six times on 2026-10-25.** The 08:30 slot started 08:31:14 and closed **10:26:37 `failed`** on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed)` — against **91 s** (10-23) and **104 s** (10-24). The same job then fired five more times (17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53); the 19:18:41 fire died on a **new signature** — `Cron admission missing: the owner never admitted this fire` (distinct from the 10-20 orphan) — and all six fires, failures included, carry `delivery_outcome: delivered`, so up to **six digests reached Anthony**. Neither failure raised a `cron_incidents` row; the job now reads `last_status: delivery_queued` with an **empty** `last_error`. Hypothesis (unconfirmed): the 08:31 fire **adopted an unrelated long-lived session** — its worker made **70 `terminal` calls on VM101 `/dev/sdb` backup work** with context growing 66 k → 148 k tokens, while **context compression was unavailable all day** (`invalid_grant`). → [[systems/scheduled-tasks]], [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] — **`Invalid refresh token` — Nous Portal runtime credentials are dead, so context summarisation failed.** Gateway-log counts: **1** line 2026-10-22, **3** on 10-24, **20** on 10-25 (last 14:53:04, `agent.context_compressor: Failed to generate context summary`), **0** since the 19:27:13 restart. ⚠ **Masked, not fixed:** at **2026-10-25 19:13** all **eight** `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off `nous` to **`no-think/freeapi/auto` on `custom:omni-101`** (fallback `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, `free_only: true`; compression timeout 90 s, vision 180 s). No `hermes model` re-authentication has been run and **`nous` is still the first entry in `fallback_providers`**, so failover tries a dead credential. → [[systems/model-providers]], [[systems/current-profile]]
- [[incidents/2026-10-24-conduit-call-request-refused]] — **The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56).** He asked to be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); all three replies asserted the agent has **no calling capability**, with **zero tool calls** in the session. `conduit_call_user` (toolset `conduit`) exists, is registered for every session, is reachable via `tool_search`, and the plugin ships the `calling-the-user` skill; his call settings (`enabled` / `when_asked` / `decides`) were permissive and the device was paired. **Root cause: capability asserted instead of discovered** — the first reply came from `kilo-auto/free`, but the switch to `smart-route` repeated it. Same audit also **corrects the 10-22 incident's closing evidence**: `conduit-calls.json → ends` records turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** sessions — `history` (real calls) is **empty** and no call has been placed since *(superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → [[incidents/2026-10-29-conduit-voice-silence]])* the memory fix. → [[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]]
- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since *(superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → [[incidents/2026-10-29-conduit-voice-silence]])*. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]]
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]]
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery.
- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] — **Third iteration-cap in seven days, and the first that raised no incident at all.** Job Radar Stage 1 (`dca8482e4f76`) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** on 2026-11-02, hit the **agent iteration cap before the handoff was written**, and closed **`status: completed` / `error: NULL` / `delivery_outcome: suppressed`**. Both Stage 2 slots declined correctly (`92665337…` 11:30 and the `05a3f00b…` 14:30 safety net both exited `[SILENT]`), so **nothing reached Notion, no cover letters were generated, and the day's 164 deduped candidates were never persisted** — a state a `status`-only health check reports as green. Contrast 10-27 (`dca848_e93a538ade6b`) and 10-28 (`dca848_0a2f03b5952b`), where the identical cap **did** open rows whose `error` text is the run's own INCOMPLETE report: on 11-02 the ledger **lost the red row it used to produce**. Discovery itself worked — SEEK **51**, LinkedIn **60 IDs**, Indeed live, WA Government **35** (197 raw / **164 new** after three-level dedup) — and a **URL-canonicalisation bug was found and fixed mid-run** (`au.seek.com` vs `au.seek.com.au` had been dropping SEEK *and* Indeed entries from the consolidated list). → [[systems/scheduled-tasks]]
- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] — **The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered:** its execution closed `unknown` at 18:05:13 with *"Cron execution unverified … Gateway connection failed; no local fallback"*, its incident row is **still `alerted`**, the job sits at `error` / `failure_streak: 1`, and `deliveries.db` shows **no row after 11:01:12** for the rest of the day. Its batch-mate `b63ed5817212` took the identical error and self-healed in 68 s. Same page records the **inverse silent failure**: `ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning **no execution row and no incident row at all**. ✅ **RESOLVED 2026-11-02T01:27:21 — the review was never lost.** The *same* execution row (`be6ec11a8e68428084af9ab18fb148db`) closed **`completed` / `delivered` at 01:27:22** after **7 h 26 m 43 s**, the delivery lands at **01:27:21.802246** and incident `79f9bb_9c443b602eec` auto-closed at **01:27:21.606691** — so the 18:05:13 `unknown` was a **reconciler timing artefact on a still-running job**, not a verdict (page now `resolved`). Batch-mate `b63ed5` closed in 68 s and `79f9bb` in 7 h 26 m, both `completed`; **an `unknown` row on a long job is a timing artefact, not a failure**. `ecc1fc13655a`'s ghost `error` cleared too (12:00:33 → `completed` / `delivered` 12:29:26), taking **non-clean `last_status` 2 → 1**.
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] — **Three restart-safe cron workers killed by `SIGTERM` (`-15`) after adopting their claim**, so the scheduler recorded *"whether side effects ran is unknown"*: `85edf6772bca` @07:30:46, `96eb95ce12f2` + `0e753e9720fd` @09:02:18. **A fourth variant of the orphan class** — and the first with **no gateway restart, reload or drain anywhere in the window** (PID `175269` up since 10-31 19:24:40; relay counter monotonic through both signals), so it cannot be attributed to reload behaviour. Likely trigger: a **30 s worker-ack window** smaller than this host's cold start (**83 plugins / 69 enabled** discovered in 70 s). All three rows now `resolved`; one execution left `unknown`.
- [[incidents/2026-11-03-job-radar-stream-active-timeout]] — **The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the ledger went red.** `dca8482e4f76` ran **07:00:33.819190 → 13:28:52.305995 = 6 h 28 m 18 s** on 2026-11-03 and closed **`failed` / `RuntimeError: stream active timeout`**, with the **failure notice delivered** (13:28:51.741724) and incident **`dca848_dfd4bea84f2a`** (`failure_type: timeout`, **`alerted`**, open) — **unclosed rows 0 → 1**. The run's activity stops at **12:23:28**; the harness does not declare the turn dead until **13:28:49** — a **65 m 21 s silence** with no tool call, no model turn and nothing in the ledger, and **no "maximum number of tool-calling iterations" message anywhere** (this is a provider-stream timeout, not the cap that ended 10-27 / 10-28 / 11-02). It compacted **five times** (08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21) in 363 messages. **Both Stage 2 slots declined correctly** (`5fc292ff0f3b` 11:30:36 → 11:32:16; 14:30:32 → 14:33:11) because no handoff was written, and `radar-ref/handoff/` still holds only 10-27 / 10-29 / 10-30 — so **11-02's 164 deduped candidates and 11-03's discovery are both unrecovered**. Pulse carried it to Anthony in the desktop session. **Answers the 11-02 page's open question 3:** suppression tracks the terminal state — `completed` → nothing delivered, `failed` → notice delivered. → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]], [[systems/scheduled-tasks]]
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]]
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? **Second surface (2026-10-20 18:37):** an interactive `desktop` session received the same canned line after `Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … response_len=0`, so the substitute is emitted by the harness's failure path **generally**, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — **was never answered**.
- [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **`omniroute-1` load-sheds model calls with `HTTP 503 … resource pressure` — two bursts: 2026-10-16 → 10-17, and again 2026-10-19.** Burst 1: **9 model calls across 7 jobs** over ~19 h, self-cleared (reported then as `10-17 = 162 / 5`; the DB reads **163 / 5** — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: **9 model calls across 8 jobs** on 2026-10-19 (06:01 + 07:01 → 12:01, whole day **159 / 13**). `/api/health` read `200 ok` throughout both. **Status: `active` → back to `monitoring` on 2026-10-21** — 2026-10-20 ran **168 / 0** with zero rows carrying the resource-pressure string, so the second burst is over; it stays a standing risk until the single-node topology changes. 4 rows closed on 10-20 (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`); **unclosed rows 13 → 9** (8 `alerted` / 1 `detected`); the ledger still tags every 503 row `failure_type: unknown`, so triage on error text. **Single-node topology still open — and every job that failed reads `model: smart-route`, `provider: custom:omni-101`.**
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved)
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST. **CLOSED — all 12 incident rows are now resolved**; the last two (`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, both on the weekly Friday job `1b23c042e17d`) closed **2026-10-23 08:04** when that slot ran clean, and the job reads `ok` again
- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question. **✅ RESOLVED 2026-11-01** — the ranking weights *were* persisted, in `~/.hermes/mnemosyne/config.yaml` rather than the file the 2026-10-10 probe read; the original regression query now returns only current-state facts, and conflict `152` was auto-resolved on 2026-10-29
- [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05)
- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
- [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); **closed 2026-10-05** when all three were repinned off the pin
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26)
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. **Recurred 2026-10-10 inside the *gateway* runtime env** — `mnemosyne` and `elevenlabs` missing from the committed uv env the gateway loads; both repaired and verified the same evening
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (Aug 2026, **superseded**): capture verified healthy 2026-10-12 — 786 working rows, +2 that day, 98.3 % embedded
- [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration
### Monitoring & Dashboards
- [[systems/homepage-dashboard]] — Homepage dashboard (CT201)
- [[systems/home-assistant]] — Home Assistant integration
### Reports
- [[reports/uber-eats/index]] — Uber Eats monthly spend reports
- [[reports/communities-wa-social-media-audit-2026]] — Department of Communities WA social media audit and first-day brief
- [[systems/obsidian]] — Obsidian vault structure and conventions
- [[queries/retained-research-answers]] — Retained research answers (fastest VNC clients, 2026-10-11)
## Infrastructure
### Services
- [[infrastructure/hosts]] — Hosts, containers, important paths
- [[infrastructure/docker-services]] — Docker compose files
- [[infrastructure/credentials-map]] — Where credentials live (no values)
- [[infrastructure/network-map]] — Zones, hosts, and ports across LAN, Tailscale, and the public tunnel
- [[infrastructure/domains-and-tunnels]] — martinwa.org hostnames and Cloudflare Tunnel routing
- [[infrastructure/backups]] — hermes backup CLI, retention, and gaps
### Networking
- [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 9.2.10 + `m93p` 9.2.10, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 78.29 % / ~152 GiB free, 2026-11-07)
- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **58-node** tailnet inventory (re-verified 2026-11-10: 57 peers + `hermes`, 42 online; `octop` back **online**, 8 offline ages refreshed, 6 ephemeral `codex-environment-*` peers documented; **CT107 `mem0`** added to the m93p guest list)
- [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly
## Concepts
- [[concepts/agent-memory]] — Memory model
- [[concepts/model-routing]] — Routing concepts
- [[concepts/persistent-shells]] — Terminal persistence
- [[concepts/browser-automation]] — Browser automation concepts
- [[concepts/tool-calling]] — Tool calling patterns
## Decisions
- [[decisions/index]] — Decision index
- [[decisions/keep-openrouter-as-fallback]] — Keep OpenRouter as fallback provider
- [[decisions/persistent-shell-enabled]] — Enable shell persistence
## Comparisons
- [[comparisons/camoufox-vs-chrome]] — Camoufox vs Chrome
- [[comparisons/hermes-vs-nanobot]] — Hermes vs Nanobot
- [[comparisons/memory-backends]] — Memory backend comparison
- [[comparisons/model-routers]] — Model router comparison
## Runbooks
- [[runbooks/multiplexer-setup]] — Multi-profile gateway multiplexer setup
- [[runbooks/diagnose-docker]] — Docker diagnosis
- [[runbooks/recover-docker-service]] — Docker service recovery
- [[runbooks/restart-hermes]] — Hermes restart procedure
- [[runbooks/restart-browser]] — Browser restart procedure
- [[runbooks/update-hermes-safely]] — Safe Hermes update
- [[runbooks/restore-openrouter]] — OpenRouter restoration
- [[runbooks/provider-health]] — Provider health check
- [[runbooks/troubleshoot-tool-loop]] — Tool loop troubleshooting
- [[runbooks/gateway-resource-resilience]] — Gateway resilience
- [[runbooks/job-radar]] — Job Radar search, Notion, dashboard and recovery procedure (**two-stage since 2026-10-27:** `dca8482e4f76` discovery → `5fc292ff0f3b` write and letters)
## Raw
- [[raw/configs/hermes-config-sanitized.txt]] — Sanitized config