Files
agent-estate-wiki/index.md
T
Hermes Wiki Audit 729bb597cf audit(2026-11-11): the audit's own 429 killed the 11-10 window; stocktake + radar recover with output
Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed).

NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
 - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed
 - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6)
   reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded
 - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row
 - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2

RESOLVED with output (dated Recovery sections added):
 - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0
 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B)
 - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved

UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md
(opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x),
incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
2026-11-12 00:16:39 +08:00

174 lines
66 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Anthony's Systems Wiki
type: collection
status: active
created: 2026-07-22
updated: 2026-11-11
verified_on: 2026-11-11
confidence: high
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
---
# Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-11-09.
>
> **2026-11-09 (nightly audit — window 2026-11-08) — the day's fault was a hook, not a schedule, and it is fixed.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`), and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** — `errors.log` holds **12** lines on 11-07 and **46** on 11-08, plus **89** *"skipped after previous timeout or while still running"* suppression lines. Every tool call degraded to slow-then-suppressed, which is what stretched the day. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 → 120** in `~/.hermes/config.yaml` — verified live, and verified *effective* (the last timeout line is still 08:21:03, re-checked at audit time). ⚠ **The obvious suspect was measured and acquitted:** the plugin's caching is fine (`session_titles.py:21 CACHE_SECONDS = 30.0`; timed in-process at **3.1 s cold, 0.00 s warm**), so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong** — the trigger is **load** on the 2-core CT460 (`load average 5.89` at 13:27:50, `state.db` checkpointing a **40 MB WAL** on the same spindle) → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. **Cost of the fault:** *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and **12:00:51 → 18:00:42 (5 h 59 m 51 s)** — its two slowest slots on record, both delivered; *FreshRSS Midday Briefing* took **2 h 09 m 53 s**; and **the stocktake failed a third consecutive day** (`1a8e90ea2271`, 03:01:11 → 04:36:08, `failure_streak: 3`) on a **new terminal signature** — `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.` where the previous two red days read *"lost its durable fire claim ownership"*. *Working hypothesis, not proven:* an open tool result is exactly what a suppressed `pre_tool_call` would produce, but the string does not name the hook. **11-08 settled: 169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:33); deliveries **8 / 8 `delivered`**; fleet **29 jobs (26 / 3)** — a **sixth** consecutive audit at the same count with nothing added, retired, disabled or repinned, though `updated_at` moved to `2026-11-09T00:06:50+08:00` as a run-state stamp. **Unclosed incident rows 2 → 3**, all three the stocktake's and all silent (`detected`, `alerted_at: null`) — but **the alerting path is not the fault**: the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` **08:06:54** in the same window. ✅ **Two verified resolutions:** the **Mnemosyne Health Monitor's no-agent conversion works** — its first script slot ran **2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, silent, writing a **159 B** output file against **105,982 B** for the last agent-mode run (coverage live at **working 1030/1030 = 100.0 %, episodic 1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet; the script uses `awk` because **`bc` is not installed on this host**); and the **1.89 GB `state.db`** was re-measured (**1,890,656,256 B**, **2,901 sessions / 197,682 messages**, live **40 MB** WAL) with the 2026-08-11 remnant **`state.db.corrupted.20260811_151052` (624.1 MB) deleted**. **New page:** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
>
> **2026-11-08 (nightly audit — window 2026-11-07) — two runs orphaned mid-restart, and two incident rows that never alerted.** The gateway restart that arrived at **05:01:00** (after a requested restart sat in *after-turn* mode for the full 1800 s) drained two in-flight cron executions; both closed `failed` on `lost its durable fire claim ownership` — *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42 → 05:02:22) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31 → 05:02:22, **second consecutive failed day**, `failure_streak: 2`). The restart was requested at ~04:29:52 — **inside the `conduit_push` self-update window** (`0.122.2 → 0.134.4`, 04:27:55→04:32:32, its own ledger recording `restarting_gateway` = *failed*) — working hypothesis, not proven; the 0.22.5 → 0.23.0 updater is excluded (its receipt starts 05:05:13). Day counts: **163 rows — 159 `completed` / 3 `failed` / 1 `running`**; the third failure is the agent-log watchdog correctly reporting the event 19 s later. ⚠ **Both stocktake incident rows are still `detected` with `alerted_at: null`** — closed 0 → 2, and neither alerted. Elsewhere in the window: **Hermes 0.22.5 → 0.23.0** on CT460 (snapshot stage **skipped again** — still no rollback point), the `camofox` browser backend arrived as an **enabled plugin (1.1.0)**, `conduit_push` self-updated to **0.134.4**, `hermes-hq-push` reinstalled **0.9.27 → 0.9.42** (routes reloaded by the 11:11 restart), and the **OpenCode Go credential is expired** (HTTP 401 `api_key_expired` — configuration alive, credential dead, nothing consuming it yet). New page: [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]].
>
> **2026-11-05 (stocktake) -** **the `systems/skills-index.md` flip-flop is fixed at the root**: two different scripts were writing the page, and the nightly audit was pointed at the stale one; it is now a delegating wrapper -> [[incidents/2026-11-05-skills-index-generator-divergence]]. Docker inventory corrected against live probes on all five hosts - ~~the `worldmonitor-seeder-scheduled` container recorded yesterday **does not exist on any host**~~ **⚠ CORRECTED 2026-11-06: false - the seeders are transient systemd-timer-driven containers, absent only between runs, not removed.** Tailscale node table re-verified programmatically - 52 rows, 52 live peers, ~~every state matching~~ **⚠ CORRECTED 2026-11-06: `anthonys-mac-mini` has since come back online.** Freshness **0 stale / 122 fresh / 3 no-date of 125**.)
>
> **2026-11-07 (nightly audit - window 2026-11-06) - a red row with no alert:** 2026-11-06 settled **172 execution rows across 16 jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit). The single failure is a **new, non-restart signature**: the Job Radar's **11:30 slot** (`5fc292ff0f3b`, 11:30:32 -> 13:26:44) was **discarded as stale** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job (08:27:09 -> 13:26:22, 4 h 59 m 13 s)** was still holding the claim and delivered first. **Every earlier occurrence of this string came from a restart/reload orphaning a run; this one had no restart** -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. It raised **no `cron_incidents` row** (newest ledger activity remains the 2026-11-05T12:36:17 closures) and **nothing alerted**; the failed run left no output file. The day still landed in full via the 14:30 slot (**12 m 27 s**, `SAME_DAY_STRONG_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`, cache **494**, *Run status: COMPLETE*). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; fleet **29 jobs (26 / 3)** - **fifth consecutive audit at the same count**, nothing added/retired/disabled/repinned, non-clean `last_status` still **1** (`066bb5191280`). **The stocktake's overrun fully unwound** (`1a8e90ea2271`, 03:00:32 -> 04:11:35 = **1 h 11 m 03 s** against 7 h 40 m / 5 h 41 m / 3 h 02 m on 11-03 -> 11-05) and **Stage 1 ran clean for a second day** (`dca8482e4f76`, 1 h 28 m 37 s, `suppressed`), clearing the 11-04 Open entry in substance. The last audit's open question closed itself: the **Mnemosyne Health Monitor**'s 11-05 12:00:45 slot (12 h+ in flight, undelivered at the last audit) **completed at 2026-11-06T00:14:39 after 12 h 13 m 54 s and delivered**; its 11-06 slot took **7 h 47 m 56 s** and delivered too - ballooned on consecutive days, never lost, **watching, not open**. The one open page remains [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (**untouched in-window**).
>
> **2026-11-06 (nightly audit - window 2026-11-05) - a clean ledger day and an empty incident ledger:** 2026-11-05 settled **175 rows - 173 `completed` / 2 `running` / 0 `failed`** across 18 jobs, the **first zero-`failed` day since 2026-11-02**. The only two `running` rows are the **Mnemosyne Health Monitor** (12:00:45 slot, still in flight **12 h+** at audit time) and this audit itself. **Unclosed incident rows 3 -> 0** - the *Newsletter Digest* row closed at **07:07:42** and both *Job Radar - Discovery* rows at **12:36:17**, when `dca8482e4f76` finally completed after **5 h 35 m 35 s**; that closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, `delivery_failed`, streak 0); deliveries **10 rows / 10 `delivered`**; the 03:00 stocktake overran **3 h 01 m 58 s** (down from 7 h 40 m and 5 h 41 m) and **committed `2f3fc99` at 05:55**. **Fleet unchanged at 29 jobs (26 / 3)** - a fourth audit at the same count. ⚠ **New and open:** live `.env` values (`TELEGRAM_BOT_TOKEN`, `MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) sit verbatim in **17 rows of `mnemosyne.db`** (`memories`, `episodic_memory`, `working_memory`, `gists`, `memoria_facts.context_snippet` and the FTS mirrors) - reproduced in full by each of the **7 uncompressed rotation snapshots** - and all four values sit verbatim in the **1.7 GB session store `state.db`** too; `CAMOFOX_API_KEY` sits verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and therefore in `cron/deliveries.db` -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
>
> **2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze:** the **11-04 fleet settled 170 rows - 167 `completed` / 2 `failed` / 1 `running`**, **non-clean `last_status` 2 -> 3**, **unclosed incident rows 1 -> 3**, **deliveries 8 rows / 8 `delivered`**. Both failures are the *same* upstream fault: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**) died **36 seconds apart at 09:41-09:42**, each after **2 h 41 m**, on `RuntimeError: Upstream stream failed before completion.` - both pinned `custom:omni-101` / `smart-route`. **The day was not lost:** the 11:30 Stage 2 slot **reconstructed the missing handoff from Stage 1's own candidate files**, re-verified every candidate live, and still wrote **5 roles (board 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at 16:08:59 -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. The cover-letter route survived only on an override: `regenerate_cover_letter.py` defaults to `deepseek-paid-smart`, which threw **HTTP 402** mid-run, so the letter was pinned to `gemini/gemini-3.5-flash` via `CL_MODEL` - and that override **was never written to `.env`**, so it did not persist. And **CT299 was frozen twice, not once**: a second, **manual** `vzdump 299 ... --storage ssd0` wedged from **11-04 08:30:04** until **11-05 ~00:30** (~16 h); the live probe at 00:54:21 AWST shows `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. ⚠ **Corrected the same night:** the first draft of this entry read the fleet as **158 jobs (144 / 14)** and the
> 11-04 ledger as closing `1 item_suppressed`; live reads give **29 jobs (26 / 3)** and a final row of **1 `running`**
> (this audit). Keyed probe **2026-11-05 01:33**: `deepseek-paid-smart` is **undispatchable on `omniroute:20128`**
> (HTTP **503**, all targets skipped) but **serves normally on `omniroute-1`** (HTTP **200** -> `deepseek/deepseek-v4-flash`).
> **2026-11-04 (nightly audit) — the ledger raised its own red row:** the **11-03 fleet settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) — **unclosed incident rows 0 → 1**, non-clean `last_status` **1 → 2**, deliveries **7 rows / 7 `delivered`**. The single failure is the **Job Radar Stage 1** (`dca8482e4f76`), which ran **07:00:33 → 13:28:52 (6 h 28 m 18 s)** and died on **`RuntimeError: stream active timeout`** — a **new terminal signature**, not the iteration cap — after going **silent for 65 m 21 s**; its incident `dca848_dfd4bea84f2a` (`timeout`) is **`alerted` and open** and the notice **was delivered** (13:28:51.741724) → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. That answers the 11-02 page's open question: **output is suppressed when the run ends `completed`, delivered when it ends `failed`** — which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so **2026-11-02's and 2026-11-03's candidate sets are both unrecovered**. ⚠ **The 03:00 stocktake overran for a second day** (`1a8e90ea2271`, 03:00:30 → 10:41:13 = **7 h 40 m 43 s**, `suppressed`, nothing delivered, artefacts uncommitted again). **New page:** [[systems/hermes-hq-push]] (v0.9.27, enabled; manifest says it **replaces `dispatch-push`**, and no trace of the old plugin survives).
>
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.10), was discovered during the 2026-10-14 stocktake.
> Guest IDs collide between the two hosts.
## Start here
- [[current-state]] — Current systems status
- [[systems/hermes-agent]] — Hermes Agent overview and version
- [[systems/current-profile]] — Active Hermes profile details
## Systems
### Core
- [[systems/hermes-agent]] — Agent version, install, config paths
- [[systems/current-profile]] — Active profile details
- [[systems/nextcloud]] — Nextcloud file sharing (CT270)
- [[systems/guanaco]] — Guanaco LLM proxy on CT205 (Ollama stack)
- [[systems/model-providers]] — LiteLLM + OmniRoute providers, routing setup
- [[systems/memory-backend]] — Mnemosyne vector memory, settings
- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence
- [[systems/omniroute]] — OmniRoute catalog, combos, and host path
- [[systems/openrouter]] — OpenRouter current state and history
- [[systems/opencode-go]] — OpenCode Go subscription, model limits, pricing
- [[systems/nanobot]] — Nanobot current state and history
- [[systems/browser-backend]] — Camofox (primary, `systemd --user` on CT460, REST `:9377`) — **the only backend**; **Steel was fully removed from the stack on 2026-10-29** (plugin deleted, `cloud_provider` back to `camofox`, CT201 and CT333 deployments torn down, Serve routes deleted; `STEEL_BASE_URL` survives only in already-running process environments); memory-containment drop-in and the 2026-10-12 ceiling event documented
- [[systems/hermes-ui]] — Browser Hermes Desktop (przbadu/hermes-ui)
- [[systems/paseo]] — Paseo coding-agent daemon + web terminal workspace (CT460, :6767)
- [[systems/ignis]] — Ignis browser-native Obsidian (wiki interface)
- [[systems/terminal-backend]] — Shell persistence, working directory
- [[systems/tool-search]] — Always-on lazy discovery for external tools
### Integrations
- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.122.2` (HEAD `9eac857`, 2026-10-31 12:04 — *"Every profile that turns Conduit on runs the main install (#482)"*; **+148 commits** on since the `0.69.2` / `942ddb8` baseline, **working tree clean**)**, upstream `kaishi00/hermes-conduit-notifier`. **Gateway on `:9119` is now pid 176771, started 2026-10-31 19:35:02** (superseded: restarted 2026-10-25 18:42:33, which is when the upstream commits went live). One registered device (iPhone app **2.2.0**). ✅ **The first two real calls in the product's history are now on record — 2026-10-29 08:38:04 and 08:49:36** — after three faults in series were repaired (truncated speech-server URL → `…/v1/realtime`; 12 h dashboard session TTL → 30 d; dead `groq` STT → `local`, dead `openai` TTS → `ollama_local`) → [[incidents/2026-10-29-conduit-voice-silence]]. **So the memory fix is no longer evidence-free.** ✅ **That patch is gone** — the tree is clean and **`MEMORY_TIMEOUT_S` ships upstream** in `dashboard/api/memory.py`. ⚠ **New fault: "local" voice mode never applies** — the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **501**, `/v1/usage` reads **`llm_proxy.requests: 0`**, realtime socket rejects `session.update`), while the app's "test connection" checks **socket reachability only** → [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]]. The call store was re-shaped: **`history` is empty** and `last_request_at` is `null`, so the two 10-29 calls are no longer in it. The 10-24 refusal fault has not recurred → [[incidents/2026-10-24-conduit-call-request-refused]].
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26** (peer offline **36 d**, `tx 312 / rx 0`; host answers neither `:8065` nor ICMP). **268** connect timeouts on 10-25 vs 264 on 10-24; the reconnect **attempt counter reset to 54** at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost.
- [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip.
- [[systems/hermes-hq-push]] — **Hermes HQ iPhone app plugin** (standalone, **v0.9.27**, `kind: standalone`, author *Hermes HQ*; tree mtime **2026-11-03 11:06**). `hermes plugins list` → **enabled** (state `user`); `config.yaml` `plugins.enabled` line **7556**. Host side of the Hermes HQ app: Apple push (finished turns, lock-screen approvals, "waiting for a password/login/code" alerts, **live Lock Screen cards** sealed to the phone), phone **device access** (contacts, location, reminders, alarms, photos, Health, motion, music), the desktop-app **Hermes HQ theme**, the app's **Terminal page**, fast **Sessions list** on a busy host, cross-posting answers back to Telegram/Discord, a **restart guard**, and **Grok Live** voice-token minting (`XAI_API_KEY` or subscription sign-in; bearer never reaches the phone). Tools `device_contacts_search` / `device_request_result`; 7 hooks. Its manifest states it **replaces `dispatch-push`** — and the replacement is clean: **no `dispatch-push` reference survives in `config.yaml`** and no such directory remains. **Different stack** from the Cadu/Vory bridge on [[systems/push-notifications]]. ⚠ Its `pre_tool_call` guard also **blocks any tool call whose arguments contain the literal text of a launchd one-off-restart command** — describe it, never quote it.
- [[systems/scheduled-tasks]] — **As of 2026-11-09 (window 11-09): 174 rows / 173 `completed` / **0 `failed`** — the first `failed`-free day since 11-05; deliveries 8/8 `delivered`; unclosed incident rows 3 → 0; the stocktake closed `completed` after 9 h 10 m with **zero edits landed**; Job Radar Stage 1 overran 9 h 09 m and landed nothing (new incident).** Earlier: **As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), and `ea79a2107e36` is ENABLED again (`last_status: ok`, streak 0) after its 00:01:08 slot closed `completed` at 10:02:48; deliveries 8/8 `delivered`.** **Prior window — as of 2026-10-30 03:3x:** 29 jobs (25 enabled, 4 disabled); non-clean `last_status` 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all `delivered`. ⚠ `ea79a2107e36` (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed `unknown` on a gateway-connection failure — and its failure notice was NEVER DELIVERED (`deliveries.db` has zero rows for 2026-10-30), so the incident row is the only trace → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. Every other previously-red job has cleared to `ok`/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
- [[systems/skills-index]] — Installed skills inventory (**299 skills** (**67 categories**) as of the **2026-11-08** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
## Incidents
- [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **OPEN — the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and is **still open with `alerted_at: NULL` — the estate's only open incident row**. Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]).
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - **RESOLVED (same day) — the hook was the fault, not the schedule.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** → [[systems/conduit]], [[systems/hermes-agent]]
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]] (resolved 2026-11-11) - **RESOLVED BY ITSELF.** The nightly stocktake closed `completed` after **9 h 10 m** having landed **zero edits and no commit** — `failure_streak` 3 → 0 while the work stayed undone, and the top-priority item on its brief had been the monotonically worsening failure streak. Third consecutive day its findings never landed.
- [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]] - **NO INCIDENT RAISED.** Job Radar Stage 1 overran to **9 h 09 m**, skipped its own verification, wrote **no handoff file**, and both Stage-2 slots ran before it finished and declined silently. No incident row was created — the completion-time-only alerting gap leaves the failure invisible.
- [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] - **OPEN.** A restart requested at **~04:29:52** sat in *after-turn* mode for the full **1800 s**, then timed out and drained at **05:00:43/47** — the same `#77184` behaviour as the 10-31 reload, but this time it killed **two** in-flight cron executions: *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42.918 -> 05:02:22.870) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31.515 -> 05:02:22.174), both closed **`failed`** on `lost its durable fire claim ownership`; systemd restarted the unit at **05:01:00** (`restart counter is at 3`). A **third** failure that night was the agent-log ERROR watchdog correctly catching the event 19 s later. The request landed **inside the `conduit_push` self-update window** (04:27:55 -> 04:32:32, its own ledger recording `restarting_gateway` = *failed*) — the **working hypothesis for the requester, not proven**; the 0.22.5 -> 0.23.0 updater is excluded (receipt starts 05:05:13). ⚠ **Closed 0 -> 2 incident rows and neither alerted**: both `1a8e90_*` rows sit `detected` with `alerted_at: null`. The stocktake slot has now failed **two days running** (`failure_streak: 2`). Job Radar ran clean. → [[systems/scheduled-tasks]], [[systems/hermes-agent]], [[systems/conduit]], [[systems/mnemosyne]]
- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - **OPEN.** A `direct`-source fire of `5fc292ff0f3b` (08:27:09 -> 13:26:22, 4 h 59 m 13 s, delivered) settled the claim while the scheduler's **own 11:30:32 slot** was still running, so that slot was interrupted at **13:26:38** and closed **`failed`** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` - a **new string variant** and the first claim-loss failure **with no gateway restart involved**. It raised **no `cron_incidents` row**, so a `failed` execution with no alert was the day's only red row; the discarded run left no output file. The 14:30 slot recovered the day in a normal **12 m 27 s** cycle. Candidate fixes (slot suppression when a claim is live, consuming the next slot for out-of-band fires, raising an incident row on discard) are listed on the page and **none was applied**.
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **OPEN.** Live `.env` values were found stored verbatim in **17 rows of `mnemosyne.db`** (`memories` 4, `episodic_memory` 3, `working_memory` 1, `gists` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the **7 uncompressed rotation snapshots**, and all four values also sit verbatim in the **1.89 GB session store `state.db`** (**1,890,656,256 B**, **2,901 sessions / 197,682 messages**, live **40 MB** WAL as at 2026-11-09), which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Nothing rotated, deleted or scrubbed; no value recorded.
- [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] - `dca8482e4f76` + `85edf6772bca` died 36 s apart on `RuntimeError: Upstream stream failed before completion.` after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter).
- [[incidents/2026-11-05-skills-index-generator-divergence]] — **RESOLVED — the skills index flip-flop had two writers.** The repo `wiki-maintain.sh` (302 skills / 68 categories) and the stale `~/.hermes/scripts/wiki-maintain.sh` copy (349 / 26, because it used `Path.rglob()` and counted the parked `.archive/`) were **different scripts**, and SKILL.md pointed the 23:59 nightly audit at the stale one — so each job overwrote the other's page and the wiki oscillated nightly with **no error raised by either**. The stale copy is now a **delegating wrapper**; the repo script gained `--no-build` so the wrapper honours the old interface without a second generator.
- [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]] — **OPEN — the app's "local" voice mode never applies, and the connection test lies.** The test checks **socket reachability only**, while every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **HTTP 501** *"Start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`**, and its realtime socket **accepts the connection but rejects `session.update` as an unknown event**. `voice.mode` is a **phone-side** selector — **no `mode` key exists** in `config.yaml → voice` or `conduit-settings.json` — so a "mode did not apply" cannot be diagnosed on host config. Found via session `20261031_193709_40b9c2` (19:37:09, 30 msgs / 17 tool calls); `voice.local_live.front_model` was repointed to `mistral/voxtral-small-latest` during it and **nothing was changed on the Mac**. → [[systems/conduit]]
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]] — **A gateway reload waited its full 1,800 s drain cap, then force-drained and orphaned a live cron run.** At **18:06:41** the gateway took the reload; the drain reported `timed_out=True` at the cap and force-drained at **18:37:38** (new gateway **18:38:43**). `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:01:06 slot, **6 h 38 m 09 s** in) closed **`failed`** at **18:39:15** on *"lost its durable fire claim ownership"*. Same class as [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **second instance, and the first on a plain reload**. A **second, clean** restart followed at **19:23:39 → 19:24:40**; 2026-10-31 closed **160 / 4**. → [[systems/scheduled-tasks]], [[systems/hermes-agent]]
- [[incidents/2026-10-31-asana-access-failures]] — **Both Asana paths on CT460 are non-functional.** The MCP server `asana` (`https://mcp.asana.com/v2/mcp`, `auth: oauth`, `enabled: true`) has **never authenticated** — `~/.hermes/mcp-tokens/` holds only `asana.client.json` (**319 B**, mtime 2026-10-26 01:07) and the logs carry *"MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run `hermes mcp login asana` interactively first"*; the hosted connector also rejects the call shape. The fix needs an **interactive** `hermes mcp login asana`, which a cron context cannot perform. → [[systems/hermes-agent]]
- [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]] — **RESOLVED (root cause not established).** CT460 went dark for **5 h 30 m on 2026-10-30**: the PVE host `pve` rebooted (boot **16:24:00**), CT460 returned **16:28:00**, the previous gateway life (`pid=3913841`) was logged as exited **UNCLEANLY** at **16:35:50** (*"no exit path ran — SIGKILL / OOM / VM death"*), and the scheduler resumed at **16:37:21** with **8 catch-up runs**. **Zero execution rows exist between 10:58:20 and 16:37:21**; the day still closed **138 / 0** against 10-29's 177 / 0 (≈39 slots lost, no failures). **No monitor alerted** — the two jobs that could have (`cbfe434a3b5e`, `1a8e90ea2271`) are disabled.
- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — **RESOLVED 2026-10-30T10:02:48** (its 00:01:08 slot was still *running*, not dead: it closed `completed` after 10 h 01 m 40 s and delivered, auto-closing `ea79a2_d93c3acdc4f2`; `ea79a2107e36` is enabled again with `last_status: ok` / streak 0). **The Mnemosyne Health Monitor cron job was auto-paused overnight and its failure notice never reached Telegram — the only trace is the incident row.** `ea79a2107e36`'s 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, `detected`) — and the scheduler **paused the job at that moment**. Fleet is now **29 jobs (25 enabled / 4 disabled)**; live `last_status: error`, `failure_streak: 1`. ⚠ **Two traps in one fault:** the failure notice itself failed to send (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed` — `deliveries.db` has **zero rows for 2026-10-30**), and the run closed `unknown` rather than `failed`. The job was **not re-enabled** and the gateway was **not restarted** by the stocktake — root cause is **unverified**. → [[systems/scheduled-tasks]], [[systems/mnemosyne]]
- [[incidents/2026-10-29-conduit-voice-silence]] — **Every Conduit voice call connected and then died silently; three faults sat in series, and the first two real calls in the product's history were placed on 2026-10-29 after all three were fixed.** (A) `voice.speech_server.url` was the truncated `wss://…:8443/v1/` — the plugin dials it **verbatim** and the bare path answers **403**, reported as *"the speech server refused the key."* (B) `dashboard.basic_auth.session_ttl_seconds` was the **12 h default**, so an expired phone session gets **WS 4401** with no Hermes turn and no error on the device (the phone re-authenticated at 08:00:59, minutes *after* a silent call). (C) Both host providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq` → **403/Cloudflare 1010** (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → `…/v1/realtime`, TTL → **2 592 000 s (30 d)**, `stt.provider: local` (faster-whisper `base`), `tts.provider: ollama_local`; dashboard restarted 08:03:26. Verified by driving the whole call path for real, then by calls at **08:38:04** and **08:49:36**. ⚠ Two TTL keys exist — `dashboard.basic_auth.session_ttl_seconds` (raised) vs `dashboard.session_ttl_seconds` (**still 43 200**). `status: resolved`.
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] — **The radar's two-stage split ran for real on 2026-10-28 and landed nothing: stage 1 stopped at the agent iteration cap before writing its handoff, so both stage-2 slots correctly no-oped (silent). Two consecutive stage-1 failures, two different ceilings (tool-call budget 10-27, iteration cap 10-28); no Notion writes for a second day.** **RESOLVED 2026-10-29** — the next stage-1 run (`dca8482e4f76`, 07:00:59 → 10:16:12, 3 h 15 m 13 s) closed `completed` *and* wrote `handoff/2026-10-29.json`, which stage 2 consumed at 10:40:59 and delivered as **20 pages + 2 cover letters** (`missing_cover_letter_count 0`). `status: resolved`, review 2026-11-13.
- [[incidents/2026-10-27-job-radar-budget-exhausted]] — **The Daily Marketing Job Radar exhausted its tool-call budget for the second day running — and this time the run is recorded as `failed`, not `completed`.** On 2026-10-27 the 07:00 slot ran **07:01:08 → 11:03:54** (4 h 2 m 46 s) and its delivered report opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery/verify/dedup/classify finished, **Notion writes, cover letters and the final audit never ran** (Notion DB held 451 entries, 0 added for the day). Incident row `dca848_e93a538ade6b` opened 11:03:53 (`alerted` 11:03:59, `failure_type: auth` — wrong for a budget fault) and was still open at the 2026-10-28 audit. **Response the same day:** the radar was split into stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) and new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030), created 17:34:15, two `completed` test fires that evening → [[runbooks/job-radar]]. **RESOLVED 2026-10-29:** both of this incident's ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12** on the first successful post-split run — `dca8482e4f76` restored delivery to `local`, wrote the handoff inside budget, and no cover letter went missing (`missing_cover_letter_count 0`). `status: resolved`, review 2026-11-13.
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **A *planned* gateway restart orphaned a live cron run — the second instance of the orphan class, and the first that alerted.** On 2026-10-26 at **14:33:35 AWST** the gateway took a planned `SIGTERM` (`under_systemd=yes`); the shutdown drain reported **`timed_out=True` after just 1.35 s** of its 30 s budget and interrupted `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) **2 h 34 m** in. It closed **`failed`** on a **new** string — `lost its durable fire claim ownership` — raised **two** `cron_incidents` rows (14:33:37 *unverified / reconcile admission* + 14:35:59 *fire claim*), **delivered** the failure at 14:35:59, and left the job on **`failure_streak: 2`** from one interrupted fire. Two Mnemosyne Watchdog reds followed at 14:46 and 15:01 and self-resolved at **15:16:12**. Net = **169 / 3** for 10-26. Unresolved: why a **planned** stop bypasses the drain deadline, and whether `failure_streak` should count the unverified row. → also updates [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] (the class now recurs without a crash).
- [[incidents/2026-10-25-gmail-digest-runaway-and-refire]] — **The Gmail Action Digest ran 1 h 55 m and fired six times on 2026-10-25.** The 08:30 slot started 08:31:14 and closed **10:26:37 `failed`** on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed)` — against **91 s** (10-23) and **104 s** (10-24). The same job then fired five more times (17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53); the 19:18:41 fire died on a **new signature** — `Cron admission missing: the owner never admitted this fire` (distinct from the 10-20 orphan) — and all six fires, failures included, carry `delivery_outcome: delivered`, so up to **six digests reached Anthony**. Neither failure raised a `cron_incidents` row; the job now reads `last_status: delivery_queued` with an **empty** `last_error`. Hypothesis (unconfirmed): the 08:31 fire **adopted an unrelated long-lived session** — its worker made **70 `terminal` calls on VM101 `/dev/sdb` backup work** with context growing 66 k → 148 k tokens, while **context compression was unavailable all day** (`invalid_grant`). → [[systems/scheduled-tasks]], [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] — **`Invalid refresh token` — Nous Portal runtime credentials are dead, so context summarisation failed.** Gateway-log counts: **1** line 2026-10-22, **3** on 10-24, **20** on 10-25 (last 14:53:04, `agent.context_compressor: Failed to generate context summary`), **0** since the 19:27:13 restart. ⚠ **Masked, not fixed:** at **2026-10-25 19:13** all **eight** `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off `nous` to **`no-think/freeapi/auto` on `custom:omni-101`** (fallback `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, `free_only: true`; compression timeout 90 s, vision 180 s). No `hermes model` re-authentication has been run and **`nous` is still the first entry in `fallback_providers`**, so failover tries a dead credential. → [[systems/model-providers]], [[systems/current-profile]]
- [[incidents/2026-10-24-conduit-call-request-refused]] — **The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56).** He asked to be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); all three replies asserted the agent has **no calling capability**, with **zero tool calls** in the session. `conduit_call_user` (toolset `conduit`) exists, is registered for every session, is reachable via `tool_search`, and the plugin ships the `calling-the-user` skill; his call settings (`enabled` / `when_asked` / `decides`) were permissive and the device was paired. **Root cause: capability asserted instead of discovered** — the first reply came from `kilo-auto/free`, but the switch to `smart-route` repeated it. Same audit also **corrects the 10-22 incident's closing evidence**: `conduit-calls.json → ends` records turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** sessions — `history` (real calls) is **empty** and no call has been placed since *(superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → [[incidents/2026-10-29-conduit-voice-silence]])* the memory fix. → [[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]]
- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since *(superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → [[incidents/2026-10-29-conduit-voice-silence]])*. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]]
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]]
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery.
- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] — **Third iteration-cap in seven days, and the first that raised no incident at all.** Job Radar Stage 1 (`dca8482e4f76`) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** on 2026-11-02, hit the **agent iteration cap before the handoff was written**, and closed **`status: completed` / `error: NULL` / `delivery_outcome: suppressed`**. Both Stage 2 slots declined correctly (`92665337…` 11:30 and the `05a3f00b…` 14:30 safety net both exited `[SILENT]`), so **nothing reached Notion, no cover letters were generated, and the day's 164 deduped candidates were never persisted** — a state a `status`-only health check reports as green. Contrast 10-27 (`dca848_e93a538ade6b`) and 10-28 (`dca848_0a2f03b5952b`), where the identical cap **did** open rows whose `error` text is the run's own INCOMPLETE report: on 11-02 the ledger **lost the red row it used to produce**. Discovery itself worked — SEEK **51**, LinkedIn **60 IDs**, Indeed live, WA Government **35** (197 raw / **164 new** after three-level dedup) — and a **URL-canonicalisation bug was found and fixed mid-run** (`au.seek.com` vs `au.seek.com.au` had been dropping SEEK *and* Indeed entries from the consolidated list). → [[systems/scheduled-tasks]]
- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] — **The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered:** its execution closed `unknown` at 18:05:13 with *"Cron execution unverified … Gateway connection failed; no local fallback"*, its incident row is **still `alerted`**, the job sits at `error` / `failure_streak: 1`, and `deliveries.db` shows **no row after 11:01:12** for the rest of the day. Its batch-mate `b63ed5817212` took the identical error and self-healed in 68 s. Same page records the **inverse silent failure**: `ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning **no execution row and no incident row at all**. ✅ **RESOLVED 2026-11-02T01:27:21 — the review was never lost.** The *same* execution row (`be6ec11a8e68428084af9ab18fb148db`) closed **`completed` / `delivered` at 01:27:22** after **7 h 26 m 43 s**, the delivery lands at **01:27:21.802246** and incident `79f9bb_9c443b602eec` auto-closed at **01:27:21.606691** — so the 18:05:13 `unknown` was a **reconciler timing artefact on a still-running job**, not a verdict (page now `resolved`). Batch-mate `b63ed5` closed in 68 s and `79f9bb` in 7 h 26 m, both `completed`; **an `unknown` row on a long job is a timing artefact, not a failure**. `ecc1fc13655a`'s ghost `error` cleared too (12:00:33 → `completed` / `delivered` 12:29:26), taking **non-clean `last_status` 2 → 1**.
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] — **Three restart-safe cron workers killed by `SIGTERM` (`-15`) after adopting their claim**, so the scheduler recorded *"whether side effects ran is unknown"*: `85edf6772bca` @07:30:46, `96eb95ce12f2` + `0e753e9720fd` @09:02:18. **A fourth variant of the orphan class** — and the first with **no gateway restart, reload or drain anywhere in the window** (PID `175269` up since 10-31 19:24:40; relay counter monotonic through both signals), so it cannot be attributed to reload behaviour. Likely trigger: a **30 s worker-ack window** smaller than this host's cold start (**83 plugins / 69 enabled** discovered in 70 s). All three rows now `resolved`; one execution left `unknown`.
- [[incidents/2026-11-03-job-radar-stream-active-timeout]] — **The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the ledger went red.** `dca8482e4f76` ran **07:00:33.819190 → 13:28:52.305995 = 6 h 28 m 18 s** on 2026-11-03 and closed **`failed` / `RuntimeError: stream active timeout`**, with the **failure notice delivered** (13:28:51.741724) and incident **`dca848_dfd4bea84f2a`** (`failure_type: timeout`, **`alerted`**, open) — **unclosed rows 0 → 1**. The run's activity stops at **12:23:28**; the harness does not declare the turn dead until **13:28:49** — a **65 m 21 s silence** with no tool call, no model turn and nothing in the ledger, and **no "maximum number of tool-calling iterations" message anywhere** (this is a provider-stream timeout, not the cap that ended 10-27 / 10-28 / 11-02). It compacted **five times** (08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21) in 363 messages. **Both Stage 2 slots declined correctly** (`5fc292ff0f3b` 11:30:36 → 11:32:16; 14:30:32 → 14:33:11) because no handoff was written, and `radar-ref/handoff/` still holds only 10-27 / 10-29 / 10-30 — so **11-02's 164 deduped candidates and 11-03's discovery are both unrecovered**. Pulse carried it to Anthony in the desktop session. **Answers the 11-02 page's open question 3:** suppression tracks the terminal state — `completed` → nothing delivered, `failed` → notice delivered. → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]], [[systems/scheduled-tasks]]
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]]
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? **Second surface (2026-10-20 18:37):** an interactive `desktop` session received the same canned line after `Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … response_len=0`, so the substitute is emitted by the harness's failure path **generally**, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — **was never answered**.
- [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **`omniroute-1` load-sheds model calls with `HTTP 503 … resource pressure` — two bursts: 2026-10-16 → 10-17, and again 2026-10-19.** Burst 1: **9 model calls across 7 jobs** over ~19 h, self-cleared (reported then as `10-17 = 162 / 5`; the DB reads **163 / 5** — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: **9 model calls across 8 jobs** on 2026-10-19 (06:01 + 07:01 → 12:01, whole day **159 / 13**). `/api/health` read `200 ok` throughout both. **Status: `active` → back to `monitoring` on 2026-10-21** — 2026-10-20 ran **168 / 0** with zero rows carrying the resource-pressure string, so the second burst is over; it stays a standing risk until the single-node topology changes. 4 rows closed on 10-20 (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`); **unclosed rows 13 → 9** (8 `alerted` / 1 `detected`); the ledger still tags every 503 row `failure_type: unknown`, so triage on error text. **Single-node topology still open — and every job that failed reads `model: smart-route`, `provider: custom:omni-101`.**
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved)
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST. **CLOSED — all 12 incident rows are now resolved**; the last two (`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, both on the weekly Friday job `1b23c042e17d`) closed **2026-10-23 08:04** when that slot ran clean, and the job reads `ok` again
- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question. **✅ RESOLVED 2026-11-01** — the ranking weights *were* persisted, in `~/.hermes/mnemosyne/config.yaml` rather than the file the 2026-10-10 probe read; the original regression query now returns only current-state facts, and conflict `152` was auto-resolved on 2026-10-29
- [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05)
- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
- [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); **closed 2026-10-05** when all three were repinned off the pin
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26)
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. **Recurred 2026-10-10 inside the *gateway* runtime env** — `mnemosyne` and `elevenlabs` missing from the committed uv env the gateway loads; both repaired and verified the same evening
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (Aug 2026, **superseded**): capture verified healthy 2026-10-12 — 786 working rows, +2 that day, 98.3 % embedded
- [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration
### Monitoring & Dashboards
- [[systems/homepage-dashboard]] — Homepage dashboard (CT201)
- [[systems/home-assistant]] — Home Assistant integration
### Reports
- [[reports/uber-eats/index]] — Uber Eats monthly spend reports
- [[reports/communities-wa-social-media-audit-2026]] — Department of Communities WA social media audit and first-day brief
- [[systems/obsidian]] — Obsidian vault structure and conventions
- [[queries/retained-research-answers]] — Retained research answers (fastest VNC clients, 2026-10-11)
## Infrastructure
### Services
- [[infrastructure/hosts]] — Hosts, containers, important paths
- [[infrastructure/docker-services]] — Docker compose files
- [[infrastructure/credentials-map]] — Where credentials live (no values)
- [[infrastructure/network-map]] — Zones, hosts, and ports across LAN, Tailscale, and the public tunnel
- [[infrastructure/domains-and-tunnels]] — martinwa.org hostnames and Cloudflare Tunnel routing
- [[infrastructure/backups]] — hermes backup CLI, retention, and gaps
### Networking
- [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 9.2.10 + `m93p` 9.2.10, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 78.29 % / ~152 GiB free, 2026-11-07)
- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **58-node** tailnet inventory (re-verified 2026-11-10: 57 peers + `hermes`, 42 online; `octop` back **online**, 8 offline ages refreshed, 6 ephemeral `codex-environment-*` peers documented; **CT107 `mem0`** added to the m93p guest list)
- [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly
## Concepts
- [[concepts/agent-memory]] — Memory model
- [[concepts/model-routing]] — Routing concepts
- [[concepts/persistent-shells]] — Terminal persistence
- [[concepts/browser-automation]] — Browser automation concepts
- [[concepts/tool-calling]] — Tool calling patterns
## Decisions
- [[decisions/index]] — Decision index
- [[decisions/keep-openrouter-as-fallback]] — Keep OpenRouter as fallback provider
- [[decisions/persistent-shell-enabled]] — Enable shell persistence
## Comparisons
- [[comparisons/camoufox-vs-chrome]] — Camoufox vs Chrome
- [[comparisons/hermes-vs-nanobot]] — Hermes vs Nanobot
- [[comparisons/memory-backends]] — Memory backend comparison
- [[comparisons/model-routers]] — Model router comparison
## Runbooks
- [[runbooks/multiplexer-setup]] — Multi-profile gateway multiplexer setup
- [[runbooks/diagnose-docker]] — Docker diagnosis
- [[runbooks/recover-docker-service]] — Docker service recovery
- [[runbooks/restart-hermes]] — Hermes restart procedure
- [[runbooks/restart-browser]] — Browser restart procedure
- [[runbooks/update-hermes-safely]] — Safe Hermes update
- [[runbooks/restore-openrouter]] — OpenRouter restoration
- [[runbooks/provider-health]] — Provider health check
- [[runbooks/troubleshoot-tool-loop]] — Tool loop troubleshooting
- [[runbooks/gateway-resource-resilience]] — Gateway resilience
- [[runbooks/job-radar]] — Job Radar search, Notion, dashboard and recovery procedure (**two-stage since 2026-10-27:** `dca8482e4f76` discovery → `5fc292ff0f3b` write and letters)
## Raw
- [[raw/configs/hermes-config-sanitized.txt]] — Sanitized config