nightly audit 2026-11-09: pre_tool_call hook-timeout cascade (new incident page), hook budget 30->120s, state.db 1.89GB + corruption remnant, no-agent health monitor verified 73s, third silent stocktake incident, skills 299/67, freshness 0 stale/127 fresh of 130
This commit is contained in:
@@ -3,16 +3,18 @@ title: Anthony's Systems Wiki
|
||||
type: collection
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
updated: 2026-11-09
|
||||
verified_on: 2026-11-09
|
||||
confidence: high
|
||||
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar]
|
||||
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
|
||||
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
|
||||
---
|
||||
|
||||
# Anthony's Systems Wiki
|
||||
> Operational knowledge for Anthony's agents, infrastructure,
|
||||
> model routing and automation. Last updated: 2026-11-08.
|
||||
> model routing and automation. Last updated: 2026-11-09.
|
||||
>
|
||||
> **2026-11-09 (nightly audit — window 2026-11-08) — the day's fault was a hook, not a schedule, and it is fixed.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`), and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** — `errors.log` holds **12** lines on 11-07 and **46** on 11-08, plus **89** *"skipped after previous timeout or while still running"* suppression lines. Every tool call degraded to slow-then-suppressed, which is what stretched the day. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 → 120** in `~/.hermes/config.yaml` — verified live, and verified *effective* (the last timeout line is still 08:21:03, re-checked at audit time). ⚠ **The obvious suspect was measured and acquitted:** the plugin's caching is fine (`session_titles.py:21 CACHE_SECONDS = 30.0`; timed in-process at **3.1 s cold, 0.00 s warm**), so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong** — the trigger is **load** on the 2-core CT460 (`load average 5.89` at 13:27:50, `state.db` checkpointing a **40 MB WAL** on the same spindle) → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. **Cost of the fault:** *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and **12:00:51 → 18:00:42 (5 h 59 m 51 s)** — its two slowest slots on record, both delivered; *FreshRSS Midday Briefing* took **2 h 09 m 53 s**; and **the stocktake failed a third consecutive day** (`1a8e90ea2271`, 03:01:11 → 04:36:08, `failure_streak: 3`) on a **new terminal signature** — `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.` where the previous two red days read *"lost its durable fire claim ownership"*. *Working hypothesis, not proven:* an open tool result is exactly what a suppressed `pre_tool_call` would produce, but the string does not name the hook. **11-08 settled: 169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:33); deliveries **8 / 8 `delivered`**; fleet **29 jobs (26 / 3)** — a **sixth** consecutive audit at the same count with nothing added, retired, disabled or repinned, though `updated_at` moved to `2026-11-09T00:06:50+08:00` as a run-state stamp. **Unclosed incident rows 2 → 3**, all three the stocktake's and all silent (`detected`, `alerted_at: null`) — but **the alerting path is not the fault**: the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` **08:06:54** in the same window. ✅ **Two verified resolutions:** the **Mnemosyne Health Monitor's no-agent conversion works** — its first script slot ran **2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, silent, writing a **159 B** output file against **105,982 B** for the last agent-mode run (coverage live at **working 1030/1030 = 100.0 %, episodic 1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet; the script uses `awk` because **`bc` is not installed on this host**); and the **1.89 GB `state.db`** was re-measured (**1,890,656,256 B**, **2,901 sessions / 197,682 messages**, live **40 MB** WAL) with the 2026-08-11 remnant **`state.db.corrupted.20260811_151052` (624.1 MB) deleted**. **New page:** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
|
||||
>
|
||||
> **2026-11-08 (nightly audit — window 2026-11-07) — two runs orphaned mid-restart, and two incident rows that never alerted.** The gateway restart that arrived at **05:01:00** (after a requested restart sat in *after-turn* mode for the full 1800 s) drained two in-flight cron executions; both closed `failed` on `lost its durable fire claim ownership` — *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42 → 05:02:22) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31 → 05:02:22, **second consecutive failed day**, `failure_streak: 2`). The restart was requested at ~04:29:52 — **inside the `conduit_push` self-update window** (`0.122.2 → 0.134.4`, 04:27:55→04:32:32, its own ledger recording `restarting_gateway` = *failed*) — working hypothesis, not proven; the 0.22.5 → 0.23.0 updater is excluded (its receipt starts 05:05:13). Day counts: **163 rows — 159 `completed` / 3 `failed` / 1 `running`**; the third failure is the agent-log watchdog correctly reporting the event 19 s later. ⚠ **Both stocktake incident rows are still `detected` with `alerted_at: null`** — closed 0 → 2, and neither alerted. Elsewhere in the window: **Hermes 0.22.5 → 0.23.0** on CT460 (snapshot stage **skipped again** — still no rollback point), the `camofox` browser backend arrived as an **enabled plugin (1.1.0)**, `conduit_push` self-updated to **0.134.4**, `hermes-hq-push` reinstalled **0.9.27 → 0.9.42** (routes reloaded by the 11:11 restart), and the **OpenCode Go credential is expired** (HTTP 401 `api_key_expired` — configuration alive, credential dead, nothing consuming it yet). New page: [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]].
|
||||
>
|
||||
@@ -67,9 +69,10 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
|
||||
- [[systems/skills-index]] — Installed skills inventory (**299 skills** (**67 categories**) as of the **2026-11-08** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
|
||||
|
||||
## Incidents
|
||||
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - **RESOLVED (same day) — the hook was the fault, not the schedule.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** → [[systems/conduit]], [[systems/hermes-agent]]
|
||||
- [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] - **OPEN.** A restart requested at **~04:29:52** sat in *after-turn* mode for the full **1800 s**, then timed out and drained at **05:00:43/47** — the same `#77184` behaviour as the 10-31 reload, but this time it killed **two** in-flight cron executions: *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42.918 -> 05:02:22.870) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31.515 -> 05:02:22.174), both closed **`failed`** on `lost its durable fire claim ownership`; systemd restarted the unit at **05:01:00** (`restart counter is at 3`). A **third** failure that night was the agent-log ERROR watchdog correctly catching the event 19 s later. The request landed **inside the `conduit_push` self-update window** (04:27:55 -> 04:32:32, its own ledger recording `restarting_gateway` = *failed*) — the **working hypothesis for the requester, not proven**; the 0.22.5 -> 0.23.0 updater is excluded (receipt starts 05:05:13). ⚠ **Closed 0 -> 2 incident rows and neither alerted**: both `1a8e90_*` rows sit `detected` with `alerted_at: null`. The stocktake slot has now failed **two days running** (`failure_streak: 2`). Job Radar ran clean. → [[systems/scheduled-tasks]], [[systems/hermes-agent]], [[systems/conduit]], [[systems/mnemosyne]]
|
||||
- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - **OPEN.** A `direct`-source fire of `5fc292ff0f3b` (08:27:09 -> 13:26:22, 4 h 59 m 13 s, delivered) settled the claim while the scheduler's **own 11:30:32 slot** was still running, so that slot was interrupted at **13:26:38** and closed **`failed`** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` - a **new string variant** and the first claim-loss failure **with no gateway restart involved**. It raised **no `cron_incidents` row**, so a `failed` execution with no alert was the day's only red row; the discarded run left no output file. The 14:30 slot recovered the day in a normal **12 m 27 s** cycle. Candidate fixes (slot suppression when a claim is live, consuming the next slot for out-of-band fires, raising an incident row on discard) are listed on the page and **none was applied**.
|
||||
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **OPEN.** Live `.env` values were found stored verbatim in **17 rows of `mnemosyne.db`** (`memories` 4, `episodic_memory` 3, `working_memory` 1, `gists` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the **7 uncompressed rotation snapshots**, and all four values also sit verbatim in the **1.7 GB session store `state.db`**, which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Nothing rotated, deleted or scrubbed; no value recorded.
|
||||
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **OPEN.** Live `.env` values were found stored verbatim in **17 rows of `mnemosyne.db`** (`memories` 4, `episodic_memory` 3, `working_memory` 1, `gists` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the **7 uncompressed rotation snapshots**, and all four values also sit verbatim in the **1.89 GB session store `state.db`** (**1,890,656,256 B**, **2,901 sessions / 197,682 messages**, live **40 MB** WAL as at 2026-11-09), which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Nothing rotated, deleted or scrubbed; no value recorded.
|
||||
- [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] - `dca8482e4f76` + `85edf6772bca` died 36 s apart on `RuntimeError: Upstream stream failed before completion.` after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter).
|
||||
- [[incidents/2026-11-05-skills-index-generator-divergence]] — **RESOLVED — the skills index flip-flop had two writers.** The repo `wiki-maintain.sh` (302 skills / 68 categories) and the stale `~/.hermes/scripts/wiki-maintain.sh` copy (349 / 26, because it used `Path.rglob()` and counted the parked `.archive/`) were **different scripts**, and SKILL.md pointed the 23:59 nightly audit at the stale one — so each job overwrote the other's page and the wiki oscillated nightly with **no error raised by either**. The stale copy is now a **delegating wrapper**; the repo script gained `--no-build` so the wrapper honours the old interface without a second generator.
|
||||
- [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]] — **OPEN — the app's "local" voice mode never applies, and the connection test lies.** The test checks **socket reachability only**, while every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **HTTP 501** *"Start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`**, and its realtime socket **accepts the connection but rejects `session.update` as an unknown event**. `voice.mode` is a **phone-side** selector — **no `mode` key exists** in `config.yaml → voice` or `conduit-settings.json` — so a "mode did not apply" cannot be diagnosed on host config. Found via session `20261031_193709_40b9c2` (19:37:09, 30 msgs / 17 tool calls); `voice.local_live.front_model` was repointed to `mistral/voxtral-small-latest` during it and **nothing was changed on the Mac**. → [[systems/conduit]]
|
||||
|
||||
Reference in New Issue
Block a user