--- title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 updated: 2026-11-11 verified_on: 2026-11-11 confidence: high tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout] sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake] --- # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, > model routing and automation. Last updated: 2026-11-09. > > **2026-11-09 (nightly audit — window 2026-11-08) — the day's fault was a hook, not a schedule, and it is fixed.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`), and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** — `errors.log` holds **12** lines on 11-07 and **46** on 11-08, plus **89** *"skipped after previous timeout or while still running"* suppression lines. Every tool call degraded to slow-then-suppressed, which is what stretched the day. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 → 120** in `~/.hermes/config.yaml` — verified live, and verified *effective* (the last timeout line is still 08:21:03, re-checked at audit time). ⚠ **The obvious suspect was measured and acquitted:** the plugin's caching is fine (`session_titles.py:21 CACHE_SECONDS = 30.0`; timed in-process at **3.1 s cold, 0.00 s warm**), so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong** — the trigger is **load** on the 2-core CT460 (`load average 5.89` at 13:27:50, `state.db` checkpointing a **40 MB WAL** on the same spindle) → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. **Cost of the fault:** *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and **12:00:51 → 18:00:42 (5 h 59 m 51 s)** — its two slowest slots on record, both delivered; *FreshRSS Midday Briefing* took **2 h 09 m 53 s**; and **the stocktake failed a third consecutive day** (`1a8e90ea2271`, 03:01:11 → 04:36:08, `failure_streak: 3`) on a **new terminal signature** — `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.` where the previous two red days read *"lost its durable fire claim ownership"*. *Working hypothesis, not proven:* an open tool result is exactly what a suppressed `pre_tool_call` would produce, but the string does not name the hook. **11-08 settled: 169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:33); deliveries **8 / 8 `delivered`**; fleet **29 jobs (26 / 3)** — a **sixth** consecutive audit at the same count with nothing added, retired, disabled or repinned, though `updated_at` moved to `2026-11-09T00:06:50+08:00` as a run-state stamp. **Unclosed incident rows 2 → 3**, all three the stocktake's and all silent (`detected`, `alerted_at: null`) — but **the alerting path is not the fault**: the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` **08:06:54** in the same window. ✅ **Two verified resolutions:** the **Mnemosyne Health Monitor's no-agent conversion works** — its first script slot ran **2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, silent, writing a **159 B** output file against **105,982 B** for the last agent-mode run (coverage live at **working 1030/1030 = 100.0 %, episodic 1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet; the script uses `awk` because **`bc` is not installed on this host**); and the **1.89 GB `state.db`** was re-measured (**1,890,656,256 B**, **2,901 sessions / 197,682 messages**, live **40 MB** WAL) with the 2026-08-11 remnant **`state.db.corrupted.20260811_151052` (624.1 MB) deleted**. **New page:** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. > > **2026-11-08 (nightly audit — window 2026-11-07) — two runs orphaned mid-restart, and two incident rows that never alerted.** The gateway restart that arrived at **05:01:00** (after a requested restart sat in *after-turn* mode for the full 1800 s) drained two in-flight cron executions; both closed `failed` on `lost its durable fire claim ownership` — *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42 → 05:02:22) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31 → 05:02:22, **second consecutive failed day**, `failure_streak: 2`). The restart was requested at ~04:29:52 — **inside the `conduit_push` self-update window** (`0.122.2 → 0.134.4`, 04:27:55→04:32:32, its own ledger recording `restarting_gateway` = *failed*) — working hypothesis, not proven; the 0.22.5 → 0.23.0 updater is excluded (its receipt starts 05:05:13). Day counts: **163 rows — 159 `completed` / 3 `failed` / 1 `running`**; the third failure is the agent-log watchdog correctly reporting the event 19 s later. ⚠ **Both stocktake incident rows are still `detected` with `alerted_at: null`** — closed 0 → 2, and neither alerted. Elsewhere in the window: **Hermes 0.22.5 → 0.23.0** on CT460 (snapshot stage **skipped again** — still no rollback point), the `camofox` browser backend arrived as an **enabled plugin (1.1.0)**, `conduit_push` self-updated to **0.134.4**, `hermes-hq-push` reinstalled **0.9.27 → 0.9.42** (routes reloaded by the 11:11 restart), and the **OpenCode Go credential is expired** (HTTP 401 `api_key_expired` — configuration alive, credential dead, nothing consuming it yet). New page: [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]. > > **2026-11-05 (stocktake) -** **the `systems/skills-index.md` flip-flop is fixed at the root**: two different scripts were writing the page, and the nightly audit was pointed at the stale one; it is now a delegating wrapper -> [[incidents/2026-11-05-skills-index-generator-divergence]]. Docker inventory corrected against live probes on all five hosts - ~~the `worldmonitor-seeder-scheduled` container recorded yesterday **does not exist on any host**~~ **⚠ CORRECTED 2026-11-06: false - the seeders are transient systemd-timer-driven containers, absent only between runs, not removed.** Tailscale node table re-verified programmatically - 52 rows, 52 live peers, ~~every state matching~~ **⚠ CORRECTED 2026-11-06: `anthonys-mac-mini` has since come back online.** Freshness **0 stale / 122 fresh / 3 no-date of 125**.) > > **2026-11-07 (nightly audit - window 2026-11-06) - a red row with no alert:** 2026-11-06 settled **172 execution rows across 16 jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit). The single failure is a **new, non-restart signature**: the Job Radar's **11:30 slot** (`5fc292ff0f3b`, 11:30:32 -> 13:26:44) was **discarded as stale** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job (08:27:09 -> 13:26:22, 4 h 59 m 13 s)** was still holding the claim and delivered first. **Every earlier occurrence of this string came from a restart/reload orphaning a run; this one had no restart** -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. It raised **no `cron_incidents` row** (newest ledger activity remains the 2026-11-05T12:36:17 closures) and **nothing alerted**; the failed run left no output file. The day still landed in full via the 14:30 slot (**12 m 27 s**, `SAME_DAY_STRONG_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`, cache **494**, *Run status: COMPLETE*). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; fleet **29 jobs (26 / 3)** - **fifth consecutive audit at the same count**, nothing added/retired/disabled/repinned, non-clean `last_status` still **1** (`066bb5191280`). **The stocktake's overrun fully unwound** (`1a8e90ea2271`, 03:00:32 -> 04:11:35 = **1 h 11 m 03 s** against 7 h 40 m / 5 h 41 m / 3 h 02 m on 11-03 -> 11-05) and **Stage 1 ran clean for a second day** (`dca8482e4f76`, 1 h 28 m 37 s, `suppressed`), clearing the 11-04 Open entry in substance. The last audit's open question closed itself: the **Mnemosyne Health Monitor**'s 11-05 12:00:45 slot (12 h+ in flight, undelivered at the last audit) **completed at 2026-11-06T00:14:39 after 12 h 13 m 54 s and delivered**; its 11-06 slot took **7 h 47 m 56 s** and delivered too - ballooned on consecutive days, never lost, **watching, not open**. The one open page remains [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (**untouched in-window**). > > **2026-11-06 (nightly audit - window 2026-11-05) - a clean ledger day and an empty incident ledger:** 2026-11-05 settled **175 rows - 173 `completed` / 2 `running` / 0 `failed`** across 18 jobs, the **first zero-`failed` day since 2026-11-02**. The only two `running` rows are the **Mnemosyne Health Monitor** (12:00:45 slot, still in flight **12 h+** at audit time) and this audit itself. **Unclosed incident rows 3 -> 0** - the *Newsletter Digest* row closed at **07:07:42** and both *Job Radar - Discovery* rows at **12:36:17**, when `dca8482e4f76` finally completed after **5 h 35 m 35 s**; that closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, `delivery_failed`, streak 0); deliveries **10 rows / 10 `delivered`**; the 03:00 stocktake overran **3 h 01 m 58 s** (down from 7 h 40 m and 5 h 41 m) and **committed `2f3fc99` at 05:55**. **Fleet unchanged at 29 jobs (26 / 3)** - a fourth audit at the same count. ⚠ **New and open:** live `.env` values (`TELEGRAM_BOT_TOKEN`, `MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) sit verbatim in **17 rows of `mnemosyne.db`** (`memories`, `episodic_memory`, `working_memory`, `gists`, `memoria_facts.context_snippet` and the FTS mirrors) - reproduced in full by each of the **7 uncompressed rotation snapshots** - and all four values sit verbatim in the **1.7 GB session store `state.db`** too; `CAMOFOX_API_KEY` sits verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and therefore in `cron/deliveries.db` -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. > > **2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze:** the **11-04 fleet settled 170 rows - 167 `completed` / 2 `failed` / 1 `running`**, **non-clean `last_status` 2 -> 3**, **unclosed incident rows 1 -> 3**, **deliveries 8 rows / 8 `delivered`**. Both failures are the *same* upstream fault: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**) died **36 seconds apart at 09:41-09:42**, each after **2 h 41 m**, on `RuntimeError: Upstream stream failed before completion.` - both pinned `custom:omni-101` / `smart-route`. **The day was not lost:** the 11:30 Stage 2 slot **reconstructed the missing handoff from Stage 1's own candidate files**, re-verified every candidate live, and still wrote **5 roles (board 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at 16:08:59 -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. The cover-letter route survived only on an override: `regenerate_cover_letter.py` defaults to `deepseek-paid-smart`, which threw **HTTP 402** mid-run, so the letter was pinned to `gemini/gemini-3.5-flash` via `CL_MODEL` - and that override **was never written to `.env`**, so it did not persist. And **CT299 was frozen twice, not once**: a second, **manual** `vzdump 299 ... --storage ssd0` wedged from **11-04 08:30:04** until **11-05 ~00:30** (~16 h); the live probe at 00:54:21 AWST shows `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. ⚠ **Corrected the same night:** the first draft of this entry read the fleet as **158 jobs (144 / 14)** and the > 11-04 ledger as closing `1 item_suppressed`; live reads give **29 jobs (26 / 3)** and a final row of **1 `running`** > (this audit). Keyed probe **2026-11-05 01:33**: `deepseek-paid-smart` is **undispatchable on `omniroute:20128`** > (HTTP **503**, all targets skipped) but **serves normally on `omniroute-1`** (HTTP **200** -> `deepseek/deepseek-v4-flash`). > **2026-11-04 (nightly audit) — the ledger raised its own red row:** the **11-03 fleet settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) — **unclosed incident rows 0 → 1**, non-clean `last_status` **1 → 2**, deliveries **7 rows / 7 `delivered`**. The single failure is the **Job Radar Stage 1** (`dca8482e4f76`), which ran **07:00:33 → 13:28:52 (6 h 28 m 18 s)** and died on **`RuntimeError: stream active timeout`** — a **new terminal signature**, not the iteration cap — after going **silent for 65 m 21 s**; its incident `dca848_dfd4bea84f2a` (`timeout`) is **`alerted` and open** and the notice **was delivered** (13:28:51.741724) → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. That answers the 11-02 page's open question: **output is suppressed when the run ends `completed`, delivered when it ends `failed`** — which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so **2026-11-02's and 2026-11-03's candidate sets are both unrecovered**. ⚠ **The 03:00 stocktake overran for a second day** (`1a8e90ea2271`, 03:00:30 → 10:41:13 = **7 h 40 m 43 s**, `suppressed`, nothing delivered, artefacts uncommitted again). **New page:** [[systems/hermes-hq-push]] (v0.9.27, enabled; manifest says it **replaces `dispatch-push`**, and no trace of the old plugin survives). > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.10), was discovered during the 2026-10-14 stocktake. > Guest IDs collide between the two hosts. ## Start here - [[current-state]] — Current systems status - [[systems/hermes-agent]] — Hermes Agent overview and version - [[systems/current-profile]] — Active Hermes profile details ## Systems ### Core - [[systems/hermes-agent]] — Agent version, install, config paths - [[systems/current-profile]] — Active profile details - [[systems/nextcloud]] — Nextcloud file sharing (CT270) - [[systems/guanaco]] — Guanaco LLM proxy on CT205 (Ollama stack) - [[systems/model-providers]] — LiteLLM + OmniRoute providers, routing setup - [[systems/memory-backend]] — Mnemosyne vector memory, settings - [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence - [[systems/omniroute]] — OmniRoute catalog, combos, and host path - [[systems/openrouter]] — OpenRouter current state and history - [[systems/opencode-go]] — OpenCode Go subscription, model limits, pricing - [[systems/nanobot]] — Nanobot current state and history - [[systems/browser-backend]] — Camofox (primary, `systemd --user` on CT460, REST `:9377`) — **the only backend**; **Steel was fully removed from the stack on 2026-10-29** (plugin deleted, `cloud_provider` back to `camofox`, CT201 and CT333 deployments torn down, Serve routes deleted; `STEEL_BASE_URL` survives only in already-running process environments); memory-containment drop-in and the 2026-10-12 ceiling event documented - [[systems/hermes-ui]] — Browser Hermes Desktop (przbadu/hermes-ui) - [[systems/paseo]] — Paseo coding-agent daemon + web terminal workspace (CT460, :6767) - [[systems/ignis]] — Ignis browser-native Obsidian (wiki interface) - [[systems/terminal-backend]] — Shell persistence, working directory - [[systems/tool-search]] — Always-on lazy discovery for external tools ### Integrations - [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.122.2` (HEAD `9eac857`, 2026-10-31 12:04 — *"Every profile that turns Conduit on runs the main install (#482)"*; **+148 commits** on since the `0.69.2` / `942ddb8` baseline, **working tree clean**)**, upstream `kaishi00/hermes-conduit-notifier`. **Gateway on `:9119` is now pid 176771, started 2026-10-31 19:35:02** (superseded: restarted 2026-10-25 18:42:33, which is when the upstream commits went live). One registered device (iPhone app **2.2.0**). ✅ **The first two real calls in the product's history are now on record — 2026-10-29 08:38:04 and 08:49:36** — after three faults in series were repaired (truncated speech-server URL → `…/v1/realtime`; 12 h dashboard session TTL → 30 d; dead `groq` STT → `local`, dead `openai` TTS → `ollama_local`) → [[incidents/2026-10-29-conduit-voice-silence]]. **So the memory fix is no longer evidence-free.** ✅ **That patch is gone** — the tree is clean and **`MEMORY_TIMEOUT_S` ships upstream** in `dashboard/api/memory.py`. ⚠ **New fault: "local" voice mode never applies** — the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **501**, `/v1/usage` reads **`llm_proxy.requests: 0`**, realtime socket rejects `session.update`), while the app's "test connection" checks **socket reachability only** → [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]]. The call store was re-shaped: **`history` is empty** and `last_request_at` is `null`, so the two 10-29 calls are no longer in it. The 10-24 refusal fault has not recurred → [[incidents/2026-10-24-conduit-call-request-refused]]. - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26** (peer offline **36 d**, `tx 312 / rx 0`; host answers neither `:8065` nor ICMP). **268** connect timeouts on 10-25 vs 264 on 10-24; the reconnect **attempt counter reset to 54** at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost. - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. - [[systems/hermes-hq-push]] — **Hermes HQ iPhone app plugin** (standalone, **v0.9.27**, `kind: standalone`, author *Hermes HQ*; tree mtime **2026-11-03 11:06**). `hermes plugins list` → **enabled** (state `user`); `config.yaml` `plugins.enabled` line **7556**. Host side of the Hermes HQ app: Apple push (finished turns, lock-screen approvals, "waiting for a password/login/code" alerts, **live Lock Screen cards** sealed to the phone), phone **device access** (contacts, location, reminders, alarms, photos, Health, motion, music), the desktop-app **Hermes HQ theme**, the app's **Terminal page**, fast **Sessions list** on a busy host, cross-posting answers back to Telegram/Discord, a **restart guard**, and **Grok Live** voice-token minting (`XAI_API_KEY` or subscription sign-in; bearer never reaches the phone). Tools `device_contacts_search` / `device_request_result`; 7 hooks. Its manifest states it **replaces `dispatch-push`** — and the replacement is clean: **no `dispatch-push` reference survives in `config.yaml`** and no such directory remains. **Different stack** from the Cadu/Vory bridge on [[systems/push-notifications]]. ⚠ Its `pre_tool_call` guard also **blocks any tool call whose arguments contain the literal text of a launchd one-off-restart command** — describe it, never quote it. - [[systems/scheduled-tasks]] — **As of 2026-11-09 (window 11-09): 174 rows / 173 `completed` / **0 `failed`** — the first `failed`-free day since 11-05; deliveries 8/8 `delivered`; unclosed incident rows 3 → 0; the stocktake closed `completed` after 9 h 10 m with **zero edits landed**; Job Radar Stage 1 overran 9 h 09 m and landed nothing (new incident).** Earlier: **As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), and `ea79a2107e36` is ENABLED again (`last_status: ok`, streak 0) after its 00:01:08 slot closed `completed` at 10:02:48; deliveries 8/8 `delivered`.** **Prior window — as of 2026-10-30 03:3x:** 29 jobs (25 enabled, 4 disabled); non-clean `last_status` 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all `delivered`. ⚠ `ea79a2107e36` (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed `unknown` on a gateway-connection failure — and its failure notice was NEVER DELIVERED (`deliveries.db` has zero rows for 2026-10-30), so the incident row is the only trace → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. Every other previously-red job has cleared to `ok`/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** - [[systems/skills-index]] — Installed skills inventory (**299 skills** (**67 categories**) as of the **2026-11-08** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents - [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **OPEN — the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and is **still open with `alerted_at: NULL` — the estate's only open incident row**. Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]). - [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - **RESOLVED (same day) — the hook was the fault, not the schedule.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** → [[systems/conduit]], [[systems/hermes-agent]] - [[incidents/2026-11-09-stocktake-completes-with-zero-edits]] (resolved 2026-11-11) - **RESOLVED BY ITSELF.** The nightly stocktake closed `completed` after **9 h 10 m** having landed **zero edits and no commit** — `failure_streak` 3 → 0 while the work stayed undone, and the top-priority item on its brief had been the monotonically worsening failure streak. Third consecutive day its findings never landed. - [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]] - **NO INCIDENT RAISED.** Job Radar Stage 1 overran to **9 h 09 m**, skipped its own verification, wrote **no handoff file**, and both Stage-2 slots ran before it finished and declined silently. No incident row was created — the completion-time-only alerting gap leaves the failure invisible. - [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] - **OPEN.** A restart requested at **~04:29:52** sat in *after-turn* mode for the full **1800 s**, then timed out and drained at **05:00:43/47** — the same `#77184` behaviour as the 10-31 reload, but this time it killed **two** in-flight cron executions: *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42.918 -> 05:02:22.870) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31.515 -> 05:02:22.174), both closed **`failed`** on `lost its durable fire claim ownership`; systemd restarted the unit at **05:01:00** (`restart counter is at 3`). A **third** failure that night was the agent-log ERROR watchdog correctly catching the event 19 s later. The request landed **inside the `conduit_push` self-update window** (04:27:55 -> 04:32:32, its own ledger recording `restarting_gateway` = *failed*) — the **working hypothesis for the requester, not proven**; the 0.22.5 -> 0.23.0 updater is excluded (receipt starts 05:05:13). ⚠ **Closed 0 -> 2 incident rows and neither alerted**: both `1a8e90_*` rows sit `detected` with `alerted_at: null`. The stocktake slot has now failed **two days running** (`failure_streak: 2`). Job Radar ran clean. → [[systems/scheduled-tasks]], [[systems/hermes-agent]], [[systems/conduit]], [[systems/mnemosyne]] - [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - **OPEN.** A `direct`-source fire of `5fc292ff0f3b` (08:27:09 -> 13:26:22, 4 h 59 m 13 s, delivered) settled the claim while the scheduler's **own 11:30:32 slot** was still running, so that slot was interrupted at **13:26:38** and closed **`failed`** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` - a **new string variant** and the first claim-loss failure **with no gateway restart involved**. It raised **no `cron_incidents` row**, so a `failed` execution with no alert was the day's only red row; the discarded run left no output file. The 14:30 slot recovered the day in a normal **12 m 27 s** cycle. Candidate fixes (slot suppression when a claim is live, consuming the next slot for out-of-band fires, raising an incident row on discard) are listed on the page and **none was applied**. - [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **OPEN.** Live `.env` values were found stored verbatim in **17 rows of `mnemosyne.db`** (`memories` 4, `episodic_memory` 3, `working_memory` 1, `gists` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the **7 uncompressed rotation snapshots**, and all four values also sit verbatim in the **1.89 GB session store `state.db`** (**1,890,656,256 B**, **2,901 sessions / 197,682 messages**, live **40 MB** WAL as at 2026-11-09), which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Nothing rotated, deleted or scrubbed; no value recorded. - [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] - `dca8482e4f76` + `85edf6772bca` died 36 s apart on `RuntimeError: Upstream stream failed before completion.` after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter). - [[incidents/2026-11-05-skills-index-generator-divergence]] — **RESOLVED — the skills index flip-flop had two writers.** The repo `wiki-maintain.sh` (302 skills / 68 categories) and the stale `~/.hermes/scripts/wiki-maintain.sh` copy (349 / 26, because it used `Path.rglob()` and counted the parked `.archive/`) were **different scripts**, and SKILL.md pointed the 23:59 nightly audit at the stale one — so each job overwrote the other's page and the wiki oscillated nightly with **no error raised by either**. The stale copy is now a **delegating wrapper**; the repo script gained `--no-build` so the wrapper honours the old interface without a second generator. - [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]] — **OPEN — the app's "local" voice mode never applies, and the connection test lies.** The test checks **socket reachability only**, while every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **HTTP 501** *"Start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`**, and its realtime socket **accepts the connection but rejects `session.update` as an unknown event**. `voice.mode` is a **phone-side** selector — **no `mode` key exists** in `config.yaml → voice` or `conduit-settings.json` — so a "mode did not apply" cannot be diagnosed on host config. Found via session `20261031_193709_40b9c2` (19:37:09, 30 msgs / 17 tool calls); `voice.local_live.front_model` was repointed to `mistral/voxtral-small-latest` during it and **nothing was changed on the Mac**. → [[systems/conduit]] - [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]] — **A gateway reload waited its full 1,800 s drain cap, then force-drained and orphaned a live cron run.** At **18:06:41** the gateway took the reload; the drain reported `timed_out=True` at the cap and force-drained at **18:37:38** (new gateway **18:38:43**). `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:01:06 slot, **6 h 38 m 09 s** in) closed **`failed`** at **18:39:15** on *"lost its durable fire claim ownership"*. Same class as [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **second instance, and the first on a plain reload**. A **second, clean** restart followed at **19:23:39 → 19:24:40**; 2026-10-31 closed **160 / 4**. → [[systems/scheduled-tasks]], [[systems/hermes-agent]] - [[incidents/2026-10-31-asana-access-failures]] — **Both Asana paths on CT460 are non-functional.** The MCP server `asana` (`https://mcp.asana.com/v2/mcp`, `auth: oauth`, `enabled: true`) has **never authenticated** — `~/.hermes/mcp-tokens/` holds only `asana.client.json` (**319 B**, mtime 2026-10-26 01:07) and the logs carry *"MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run `hermes mcp login asana` interactively first"*; the hosted connector also rejects the call shape. The fix needs an **interactive** `hermes mcp login asana`, which a cron context cannot perform. → [[systems/hermes-agent]] - [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]] — **RESOLVED (root cause not established).** CT460 went dark for **5 h 30 m on 2026-10-30**: the PVE host `pve` rebooted (boot **16:24:00**), CT460 returned **16:28:00**, the previous gateway life (`pid=3913841`) was logged as exited **UNCLEANLY** at **16:35:50** (*"no exit path ran — SIGKILL / OOM / VM death"*), and the scheduler resumed at **16:37:21** with **8 catch-up runs**. **Zero execution rows exist between 10:58:20 and 16:37:21**; the day still closed **138 / 0** against 10-29's 177 / 0 (≈39 slots lost, no failures). **No monitor alerted** — the two jobs that could have (`cbfe434a3b5e`, `1a8e90ea2271`) are disabled. - [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — **RESOLVED 2026-10-30T10:02:48** (its 00:01:08 slot was still *running*, not dead: it closed `completed` after 10 h 01 m 40 s and delivered, auto-closing `ea79a2_d93c3acdc4f2`; `ea79a2107e36` is enabled again with `last_status: ok` / streak 0). **The Mnemosyne Health Monitor cron job was auto-paused overnight and its failure notice never reached Telegram — the only trace is the incident row.** `ea79a2107e36`'s 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, `detected`) — and the scheduler **paused the job at that moment**. Fleet is now **29 jobs (25 enabled / 4 disabled)**; live `last_status: error`, `failure_streak: 1`. ⚠ **Two traps in one fault:** the failure notice itself failed to send (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed` — `deliveries.db` has **zero rows for 2026-10-30**), and the run closed `unknown` rather than `failed`. The job was **not re-enabled** and the gateway was **not restarted** by the stocktake — root cause is **unverified**. → [[systems/scheduled-tasks]], [[systems/mnemosyne]] - [[incidents/2026-10-29-conduit-voice-silence]] — **Every Conduit voice call connected and then died silently; three faults sat in series, and the first two real calls in the product's history were placed on 2026-10-29 after all three were fixed.** (A) `voice.speech_server.url` was the truncated `wss://…:8443/v1/` — the plugin dials it **verbatim** and the bare path answers **403**, reported as *"the speech server refused the key."* (B) `dashboard.basic_auth.session_ttl_seconds` was the **12 h default**, so an expired phone session gets **WS 4401** with no Hermes turn and no error on the device (the phone re-authenticated at 08:00:59, minutes *after* a silent call). (C) Both host providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq` → **403/Cloudflare 1010** (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → `…/v1/realtime`, TTL → **2 592 000 s (30 d)**, `stt.provider: local` (faster-whisper `base`), `tts.provider: ollama_local`; dashboard restarted 08:03:26. Verified by driving the whole call path for real, then by calls at **08:38:04** and **08:49:36**. ⚠ Two TTL keys exist — `dashboard.basic_auth.session_ttl_seconds` (raised) vs `dashboard.session_ttl_seconds` (**still 43 200**). `status: resolved`. - [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] — **The radar's two-stage split ran for real on 2026-10-28 and landed nothing: stage 1 stopped at the agent iteration cap before writing its handoff, so both stage-2 slots correctly no-oped (silent). Two consecutive stage-1 failures, two different ceilings (tool-call budget 10-27, iteration cap 10-28); no Notion writes for a second day.** **RESOLVED 2026-10-29** — the next stage-1 run (`dca8482e4f76`, 07:00:59 → 10:16:12, 3 h 15 m 13 s) closed `completed` *and* wrote `handoff/2026-10-29.json`, which stage 2 consumed at 10:40:59 and delivered as **20 pages + 2 cover letters** (`missing_cover_letter_count 0`). `status: resolved`, review 2026-11-13. - [[incidents/2026-10-27-job-radar-budget-exhausted]] — **The Daily Marketing Job Radar exhausted its tool-call budget for the second day running — and this time the run is recorded as `failed`, not `completed`.** On 2026-10-27 the 07:00 slot ran **07:01:08 → 11:03:54** (4 h 2 m 46 s) and its delivered report opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery/verify/dedup/classify finished, **Notion writes, cover letters and the final audit never ran** (Notion DB held 451 entries, 0 added for the day). Incident row `dca848_e93a538ade6b` opened 11:03:53 (`alerted` 11:03:59, `failure_type: auth` — wrong for a budget fault) and was still open at the 2026-10-28 audit. **Response the same day:** the radar was split into stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) and new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030), created 17:34:15, two `completed` test fires that evening → [[runbooks/job-radar]]. **RESOLVED 2026-10-29:** both of this incident's ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12** on the first successful post-split run — `dca8482e4f76` restored delivery to `local`, wrote the handoff inside budget, and no cover letter went missing (`missing_cover_letter_count 0`). `status: resolved`, review 2026-11-13. - [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **A *planned* gateway restart orphaned a live cron run — the second instance of the orphan class, and the first that alerted.** On 2026-10-26 at **14:33:35 AWST** the gateway took a planned `SIGTERM` (`under_systemd=yes`); the shutdown drain reported **`timed_out=True` after just 1.35 s** of its 30 s budget and interrupted `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) **2 h 34 m** in. It closed **`failed`** on a **new** string — `lost its durable fire claim ownership` — raised **two** `cron_incidents` rows (14:33:37 *unverified / reconcile admission* + 14:35:59 *fire claim*), **delivered** the failure at 14:35:59, and left the job on **`failure_streak: 2`** from one interrupted fire. Two Mnemosyne Watchdog reds followed at 14:46 and 15:01 and self-resolved at **15:16:12**. Net = **169 / 3** for 10-26. Unresolved: why a **planned** stop bypasses the drain deadline, and whether `failure_streak` should count the unverified row. → also updates [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] (the class now recurs without a crash). - [[incidents/2026-10-25-gmail-digest-runaway-and-refire]] — **The Gmail Action Digest ran 1 h 55 m and fired six times on 2026-10-25.** The 08:30 slot started 08:31:14 and closed **10:26:37 `failed`** on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed)` — against **91 s** (10-23) and **104 s** (10-24). The same job then fired five more times (17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53); the 19:18:41 fire died on a **new signature** — `Cron admission missing: the owner never admitted this fire` (distinct from the 10-20 orphan) — and all six fires, failures included, carry `delivery_outcome: delivered`, so up to **six digests reached Anthony**. Neither failure raised a `cron_incidents` row; the job now reads `last_status: delivery_queued` with an **empty** `last_error`. Hypothesis (unconfirmed): the 08:31 fire **adopted an unrelated long-lived session** — its worker made **70 `terminal` calls on VM101 `/dev/sdb` backup work** with context growing 66 k → 148 k tokens, while **context compression was unavailable all day** (`invalid_grant`). → [[systems/scheduled-tasks]], [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] - [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] — **`Invalid refresh token` — Nous Portal runtime credentials are dead, so context summarisation failed.** Gateway-log counts: **1** line 2026-10-22, **3** on 10-24, **20** on 10-25 (last 14:53:04, `agent.context_compressor: Failed to generate context summary`), **0** since the 19:27:13 restart. ⚠ **Masked, not fixed:** at **2026-10-25 19:13** all **eight** `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off `nous` to **`no-think/freeapi/auto` on `custom:omni-101`** (fallback `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, `free_only: true`; compression timeout 90 s, vision 180 s). No `hermes model` re-authentication has been run and **`nous` is still the first entry in `fallback_providers`**, so failover tries a dead credential. → [[systems/model-providers]], [[systems/current-profile]] - [[incidents/2026-10-24-conduit-call-request-refused]] — **The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56).** He asked to be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); all three replies asserted the agent has **no calling capability**, with **zero tool calls** in the session. `conduit_call_user` (toolset `conduit`) exists, is registered for every session, is reachable via `tool_search`, and the plugin ships the `calling-the-user` skill; his call settings (`enabled` / `when_asked` / `decides`) were permissive and the device was paired. **Root cause: capability asserted instead of discovered** — the first reply came from `kilo-auto/free`, but the switch to `smart-route` repeated it. Same audit also **corrects the 10-22 incident's closing evidence**: `conduit-calls.json → ends` records turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** sessions — `history` (real calls) is **empty** and no call has been placed since *(superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → [[incidents/2026-10-29-conduit-voice-silence]])* the memory fix. → [[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]] - [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since *(superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → [[incidents/2026-10-29-conduit-voice-silence]])*. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]] - [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]] - [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery. - [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] — **Third iteration-cap in seven days, and the first that raised no incident at all.** Job Radar Stage 1 (`dca8482e4f76`) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** on 2026-11-02, hit the **agent iteration cap before the handoff was written**, and closed **`status: completed` / `error: NULL` / `delivery_outcome: suppressed`**. Both Stage 2 slots declined correctly (`92665337…` 11:30 and the `05a3f00b…` 14:30 safety net both exited `[SILENT]`), so **nothing reached Notion, no cover letters were generated, and the day's 164 deduped candidates were never persisted** — a state a `status`-only health check reports as green. Contrast 10-27 (`dca848_e93a538ade6b`) and 10-28 (`dca848_0a2f03b5952b`), where the identical cap **did** open rows whose `error` text is the run's own INCOMPLETE report: on 11-02 the ledger **lost the red row it used to produce**. Discovery itself worked — SEEK **51**, LinkedIn **60 IDs**, Indeed live, WA Government **35** (197 raw / **164 new** after three-level dedup) — and a **URL-canonicalisation bug was found and fixed mid-run** (`au.seek.com` vs `au.seek.com.au` had been dropping SEEK *and* Indeed entries from the consolidated list). → [[systems/scheduled-tasks]] - [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] — **The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered:** its execution closed `unknown` at 18:05:13 with *"Cron execution unverified … Gateway connection failed; no local fallback"*, its incident row is **still `alerted`**, the job sits at `error` / `failure_streak: 1`, and `deliveries.db` shows **no row after 11:01:12** for the rest of the day. Its batch-mate `b63ed5817212` took the identical error and self-healed in 68 s. Same page records the **inverse silent failure**: `ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning **no execution row and no incident row at all**. ✅ **RESOLVED 2026-11-02T01:27:21 — the review was never lost.** The *same* execution row (`be6ec11a8e68428084af9ab18fb148db`) closed **`completed` / `delivered` at 01:27:22** after **7 h 26 m 43 s**, the delivery lands at **01:27:21.802246** and incident `79f9bb_9c443b602eec` auto-closed at **01:27:21.606691** — so the 18:05:13 `unknown` was a **reconciler timing artefact on a still-running job**, not a verdict (page now `resolved`). Batch-mate `b63ed5` closed in 68 s and `79f9bb` in 7 h 26 m, both `completed`; **an `unknown` row on a long job is a timing artefact, not a failure**. `ecc1fc13655a`'s ghost `error` cleared too (12:00:33 → `completed` / `delivered` 12:29:26), taking **non-clean `last_status` 2 → 1**. - [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] — **Three restart-safe cron workers killed by `SIGTERM` (`-15`) after adopting their claim**, so the scheduler recorded *"whether side effects ran is unknown"*: `85edf6772bca` @07:30:46, `96eb95ce12f2` + `0e753e9720fd` @09:02:18. **A fourth variant of the orphan class** — and the first with **no gateway restart, reload or drain anywhere in the window** (PID `175269` up since 10-31 19:24:40; relay counter monotonic through both signals), so it cannot be attributed to reload behaviour. Likely trigger: a **30 s worker-ack window** smaller than this host's cold start (**83 plugins / 69 enabled** discovered in 70 s). All three rows now `resolved`; one execution left `unknown`. - [[incidents/2026-11-03-job-radar-stream-active-timeout]] — **The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the ledger went red.** `dca8482e4f76` ran **07:00:33.819190 → 13:28:52.305995 = 6 h 28 m 18 s** on 2026-11-03 and closed **`failed` / `RuntimeError: stream active timeout`**, with the **failure notice delivered** (13:28:51.741724) and incident **`dca848_dfd4bea84f2a`** (`failure_type: timeout`, **`alerted`**, open) — **unclosed rows 0 → 1**. The run's activity stops at **12:23:28**; the harness does not declare the turn dead until **13:28:49** — a **65 m 21 s silence** with no tool call, no model turn and nothing in the ledger, and **no "maximum number of tool-calling iterations" message anywhere** (this is a provider-stream timeout, not the cap that ended 10-27 / 10-28 / 11-02). It compacted **five times** (08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21) in 363 messages. **Both Stage 2 slots declined correctly** (`5fc292ff0f3b` 11:30:36 → 11:32:16; 14:30:32 → 14:33:11) because no handoff was written, and `radar-ref/handoff/` still holds only 10-27 / 10-29 / 10-30 — so **11-02's 164 deduped candidates and 11-03's discovery are both unrecovered**. Pulse carried it to Anthony in the desktop session. **Answers the 11-02 page's open question 3:** suppression tracks the terminal state — `completed` → nothing delivered, `failed` → notice delivered. → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]], [[systems/scheduled-tasks]] - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]] - [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? **Second surface (2026-10-20 18:37):** an interactive `desktop` session received the same canned line after `Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … response_len=0`, so the substitute is emitted by the harness's failure path **generally**, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — **was never answered**. - [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **`omniroute-1` load-sheds model calls with `HTTP 503 … resource pressure` — two bursts: 2026-10-16 → 10-17, and again 2026-10-19.** Burst 1: **9 model calls across 7 jobs** over ~19 h, self-cleared (reported then as `10-17 = 162 / 5`; the DB reads **163 / 5** — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: **9 model calls across 8 jobs** on 2026-10-19 (06:01 + 07:01 → 12:01, whole day **159 / 13**). `/api/health` read `200 ok` throughout both. **Status: `active` → back to `monitoring` on 2026-10-21** — 2026-10-20 ran **168 / 0** with zero rows carrying the resource-pressure string, so the second burst is over; it stays a standing risk until the single-node topology changes. 4 rows closed on 10-20 (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`); **unclosed rows 13 → 9** (8 `alerted` / 1 `detected`); the ledger still tags every 503 row `failure_type: unknown`, so triage on error text. **Single-node topology still open — and every job that failed reads `model: smart-route`, `provider: custom:omni-101`.** - [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved) - [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST. **CLOSED — all 12 incident rows are now resolved**; the last two (`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, both on the weekly Friday job `1b23c042e17d`) closed **2026-10-23 08:04** when that slot ran clean, and the job reads `ok` again - [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question. **✅ RESOLVED 2026-11-01** — the ranking weights *were* persisted, in `~/.hermes/mnemosyne/config.yaml` rather than the file the 2026-10-10 probe read; the original regression query now returns only current-state facts, and conflict `152` was auto-resolved on 2026-10-29 - [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05) - [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic - [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented - [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); **closed 2026-10-05** when all three were repinned off the pin - [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26) - [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. **Recurred 2026-10-10 inside the *gateway* runtime env** — `mnemosyne` and `elevenlabs` missing from the committed uv env the gateway loads; both repaired and verified the same evening - [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (Aug 2026, **superseded**): capture verified healthy 2026-10-12 — 786 working rows, +2 that day, 98.3 % embedded - [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration ### Monitoring & Dashboards - [[systems/homepage-dashboard]] — Homepage dashboard (CT201) - [[systems/home-assistant]] — Home Assistant integration ### Reports - [[reports/uber-eats/index]] — Uber Eats monthly spend reports - [[reports/communities-wa-social-media-audit-2026]] — Department of Communities WA social media audit and first-day brief - [[systems/obsidian]] — Obsidian vault structure and conventions - [[queries/retained-research-answers]] — Retained research answers (fastest VNC clients, 2026-10-11) ## Infrastructure ### Services - [[infrastructure/hosts]] — Hosts, containers, important paths - [[infrastructure/docker-services]] — Docker compose files - [[infrastructure/credentials-map]] — Where credentials live (no values) - [[infrastructure/network-map]] — Zones, hosts, and ports across LAN, Tailscale, and the public tunnel - [[infrastructure/domains-and-tunnels]] — martinwa.org hostnames and Cloudflare Tunnel routing - [[infrastructure/backups]] — hermes backup CLI, retention, and gaps ### Networking - [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 9.2.10 + `m93p` 9.2.10, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 78.29 % / ~152 GiB free, 2026-11-07) - [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **58-node** tailnet inventory (re-verified 2026-11-10: 57 peers + `hermes`, 42 online; `octop` back **online**, 8 offline ages refreshed, 6 ephemeral `codex-environment-*` peers documented; **CT107 `mem0`** added to the m93p guest list) - [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly ## Concepts - [[concepts/agent-memory]] — Memory model - [[concepts/model-routing]] — Routing concepts - [[concepts/persistent-shells]] — Terminal persistence - [[concepts/browser-automation]] — Browser automation concepts - [[concepts/tool-calling]] — Tool calling patterns ## Decisions - [[decisions/index]] — Decision index - [[decisions/keep-openrouter-as-fallback]] — Keep OpenRouter as fallback provider - [[decisions/persistent-shell-enabled]] — Enable shell persistence ## Comparisons - [[comparisons/camoufox-vs-chrome]] — Camoufox vs Chrome - [[comparisons/hermes-vs-nanobot]] — Hermes vs Nanobot - [[comparisons/memory-backends]] — Memory backend comparison - [[comparisons/model-routers]] — Model router comparison ## Runbooks - [[runbooks/multiplexer-setup]] — Multi-profile gateway multiplexer setup - [[runbooks/diagnose-docker]] — Docker diagnosis - [[runbooks/recover-docker-service]] — Docker service recovery - [[runbooks/restart-hermes]] — Hermes restart procedure - [[runbooks/restart-browser]] — Browser restart procedure - [[runbooks/update-hermes-safely]] — Safe Hermes update - [[runbooks/restore-openrouter]] — OpenRouter restoration - [[runbooks/provider-health]] — Provider health check - [[runbooks/troubleshoot-tool-loop]] — Tool loop troubleshooting - [[runbooks/gateway-resource-resilience]] — Gateway resilience - [[runbooks/job-radar]] — Job Radar search, Notion, dashboard and recovery procedure (**two-stage since 2026-10-27:** `dca8482e4f76` discovery → `5fc292ff0f3b` write and letters) ## Raw - [[raw/configs/hermes-config-sanitized.txt]] — Sanitized config