nightly audit 2026-11-09: pre_tool_call hook-timeout cascade (new incident page), hook budget 30->120s, state.db 1.89GB + corruption remnant, no-agent health monitor verified 73s, third silent stocktake incident, skills 299/67, freshness 0 stale/127 fresh of 130
This commit is contained in:
@@ -2379,8 +2379,66 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
|
||||
**after** this audit's writes — skills-index regenerated (**299 / 67**), freshness report written, Quartz
|
||||
rebuilt (**129 markdown → 779 files** deployed to `public/`) and `hermes-wiki-serve.service` restarted at
|
||||
**2026-11-08 00:38:29 AWST**.
|
||||
- **Wiki maintenance:** `wiki-maintain.sh` (canonical repo script) run **after** this audit's writes —
|
||||
`systems/skills-index.md` regenerated (**299 skills / 67 categories**; disk shows 300 `SKILL.md`, one nested
|
||||
duplicate de-duped by the generator), freshness report `raw/stocktake/freshness-20261109.txt`
|
||||
(**0 stale / 127 fresh / 3 no-date of 130 pages**), Quartz rebuilt (**130 markdown → 793 files** deployed to
|
||||
`public/`) and `hermes-wiki-serve.service` (a **user** unit — `systemctl --user`) restarted
|
||||
**2026-11-09 00:18:30 AWST**. Published pages verified **HTTP 200** at `http://127.0.0.1:9120/<path>` **and**
|
||||
on the external route `https://hermes.kangaroo-eel.ts.net/wiki/<path>`.
|
||||
- **No wiki page removed or archived.** No job added, retired, disabled or repinned; no system obsoleted; no
|
||||
decision reversed. (The four skill archives above happened on the filesystem, not in this wiki.)
|
||||
- **No secrets, keys, tokens or passwords written** — the opencode-go finding records the HTTP status and
|
||||
the upstream error code only.
|
||||
|
||||
|
||||
## [2026-11-09] nightly audit | Window 2026-11-08 — a 30 s hook budget, not a schedule, ate the day
|
||||
- **New page:** `incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md` — **RESOLVED same day.**
|
||||
- **The finding:** `conduit_push` is the **only** `pre_tool_call` registrant on the host
|
||||
(`plugins/conduit_push/__init__.py:119`), and its callback overran Hermes's built-in **30 s** hook budget
|
||||
**58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** — `~/.hermes/logs/errors.log`
|
||||
holds **12** lines on 11-07 and **46** on 11-08, plus **89** *"skipped after previous timeout or while
|
||||
still running"* suppression lines. Every tool call degraded from slow to suppressed.
|
||||
- **The fix (2026-11-08 13:35 AWST):** `plugins.hook_callback_timeout` **30 → 120** in
|
||||
`~/.hermes/config.yaml`. **Verified live**, and **verified effective** — the last timeout line is still
|
||||
08:21:03, re-checked at audit time.
|
||||
- **A plausible diagnosis measured and rejected:** the plugin's caching was *not* the fault. Timed
|
||||
in-process: **3.1 s cold, 0.00 s warm** (`session_titles.py:21 CACHE_SECONDS = 30.0`). The
|
||||
*"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the
|
||||
2-core CT460 (`load average 5.89` at 13:27:50; `state.db` checkpointing a **40 MB WAL** on the same
|
||||
spindle). **Next lever is `state.db` checkpoint/retention discipline, not more caching.**
|
||||
- **Cost of the fault:** *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and
|
||||
**12:00:51 → 18:00:42 (5 h 59 m 51 s)** — its two slowest slots on record, both delivered; *FreshRSS Midday
|
||||
Briefing* `ecc1fc13655a` took **2 h 09 m 53 s**; and the stocktake failed a third day.
|
||||
- **Cron ledger:** 11-08 settled **169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit,
|
||||
23:59:33). The single failure is **`1a8e90ea2271` (wiki-stocktake-maintenance), 03:01:11 → 04:36:08**, on
|
||||
a **new terminal signature**: `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still
|
||||
pending and the model produced no follow-up text.` (the previous two red days read *"lost its durable fire
|
||||
claim ownership"*). **`failure_streak: 3`.** *Working hypothesis, not proven:* a suppressed `pre_tool_call`
|
||||
holding a tool result open is exactly what the error describes, but the string does not name the hook.
|
||||
Deliveries **8 rows / 8 `delivered` / 0 `for_failure`**. Fleet **29 jobs (26 enabled / 3 disabled)** — a
|
||||
**sixth** consecutive audit at the same count, nothing added, retired, disabled or repinned (fleet
|
||||
`updated_at` moved to `2026-11-09T00:06:50+08:00`, read as a run-state stamp only).
|
||||
- **Incidents:** unclosed rows **2 → 3** — a third silent `detected` row for the stocktake
|
||||
(`1a8e90_a9cad99cb150`, `alerted_at: null`), all three still unreconciled. **The alerting path is not at
|
||||
fault:** the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and closed `resolved` at
|
||||
**08:06:54** (its admission-read `database is locked`, cleared when the run completed).
|
||||
- ✅ **Verified resolution 1 — the Mnemosyne Health Monitor's no-agent conversion works.** The 18:00 slot
|
||||
(agent mode) ran **12:00:51 → 18:00:42 = 5 h 59 m 51 s**; the converted job's first slot ran
|
||||
**2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, silent, writing a **159 B** output file against
|
||||
**105,982 B** for the last agent-mode run. Coverage live at **working 1030/1030 = 100.0 %**, **episodic
|
||||
1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet. The script compares with `awk` because
|
||||
**`bc` is not installed on this host**.
|
||||
- ✅ **Verified resolution 2 — `state.db` re-measured and one corruption remnant removed.** Now
|
||||
**1,890,656,256 B (1.89 GB)**, **2,901 sessions / 197,682 messages**, live **40,157,672 B** WAL.
|
||||
`state.db.corrupted.20260811_151052` (**624.1 MB** + 32 KB `-shm` + 0 B `-wal`) **deleted** in-window;
|
||||
`state.db.recovered.recovery.json` (14,566 B) remains and
|
||||
`mnemosyne/data/mnemosyne.db.corrupted.20260809_120404` (**23.6 MB**) was **not** cleaned.
|
||||
- **Pages updated:** `systems/conduit.md` (the pre-tool-call section, **total**), `systems/hermes-agent.md`
|
||||
(hook budget + `state.db` size, **total**), `systems/scheduled-tasks.md` (new Count bullet, **total**),
|
||||
`systems/mnemosyne.md` (health-monitor conversion verified, **total**), `current-state.md` (**total**),
|
||||
`incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md` (`state.db` 1.7 → 1.89 GB in three
|
||||
places, re `confident`), `incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md` (open items),
|
||||
`index.md` (new entry + incident list + `state.db` figure, confidence dropped), and this log.
|
||||
- **No wiki page removed or archived.** No system obsoleted; no decision reversed.
|
||||
- **No secrets, keys, tokens or passwords written** — no credential value appears anywhere in this audit.
|
||||
|
||||
Reference in New Issue
Block a user