nightly audit 2026-11-09: pre_tool_call hook-timeout cascade (new incident page), hook budget 30->120s, state.db 1.89GB + corruption remnant, no-agent health monitor verified 73s, third silent stocktake incident, skills 299/67, freshness 0 stale/127 fresh of 130

This commit is contained in:
Tony0410
2026-11-09 00:21:01 +08:00
parent 4fd138bb18
commit e2f1ae7a0b
12 changed files with 422 additions and 29 deletions
+37 -3
View File
@@ -3,8 +3,8 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-11-07
verified_on: 2026-11-07
updated: 2026-11-09
verified_on: 2026-11-09
review_after: 2026-11-09
confidence: high
tags: [system]
@@ -12,7 +12,41 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
---
# Current State
Updated: 2026-11-08
Updated: 2026-11-09
## Recent Changes (2026-11-09, nightly audit; live probes 2026-11-08 23:5x – 2026-11-09 00:2x AWST)
- ⚠ **The `conduit_push` `pre_tool_call` hook was the day's fault, and it is fixed — `plugins.hook_callback_timeout` 30 → 120 s.**
The plugin is the **only** `pre_tool_call` registrant on the host, and its callback overran Hermes's 30 s
hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: 12
lines on 11-07, 46 on 11-08; plus **89** *"skipped after previous timeout or while still running"*
suppression lines). Every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST**;
the last timeout line is still 08:21:03, re-verified at audit time. **The plugin's caching was measured,
not assumed, and was innocent** — 3.1 s cold / 0.00 s warm (`session_titles.py:21 CACHE_SECONDS = 30.0`);
the trigger is **load** on the 2-core CT460 (load average **5.89**, `state.db` with a **40 MB WAL**).
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]], [[systems/conduit]]
- ⚠ **`state.db` is now 1.89 GB** (1,890,656,256 B; **2,901 sessions / 197,682 messages**, live
**40,157,672 B** WAL) — up from 1.7 GB, and it is the load source behind the spike above. The 2026-08-11
corruption remnant **`state.db.corrupted.20260811_151052` (624.1 MB)** was **deleted** in this window;
`state.db.recovered.recovery.json` (14,566 B) remains, and `mnemosyne.db.corrupted.20260809_120404`
(23.6 MB) was **not** cleaned. → [[systems/hermes-agent]], [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]
- ✅ **The Mnemosyne Health Monitor's no-agent conversion is verified end-to-end.** The 18:00 slot (agent
mode) ran **12:00:51 → 18:00:42 = 5 h 59 m 51 s**; the converted job's first slot ran
**2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, **silent**, writing a **159 B** output file against
**105,982 B** for the last agent-mode one. Live coverage **working 1030/1030 = 100.0 %, episodic
1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet. → [[systems/mnemosyne]], [[systems/scheduled-tasks]]
- ⚠ **The stocktake failed a third consecutive day, on a new signature.** `1a8e90ea2271` ran
**2026-11-08 03:01:11 → 04:36:08** and died on `RuntimeError: ⚠️ No reply: the turn stopped while a tool
result was still pending and the model produced no follow-up text.` — the previous two red days read
*"lost its durable fire claim ownership"*. **`failure_streak` 3.** *Working hypothesis, not proven:* the
hook above holds a tool result open in exactly the way this error describes. **Unclosed incident rows
2 → 3**, and all three are silent (`detected`, `alerted_at: null`) — but **the alert path works**: the
*Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` at **08:06:54**.
→ [[systems/scheduled-tasks]], [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
- **2026-11-08 settled: 169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit). Deliveries
**8 / 8 `delivered`**. Fleet **29 jobs (26 / 3)** — a **sixth** consecutive audit at the same count, though
`updated_at` moved to `2026-11-09T00:06:50+08:00` with no definition change. Nothing added, retired,
disabled or repinned. → [[systems/scheduled-tasks]]
## Recent Changes (2026-11-08, nightly audit; live probes 2026-11-07 23:5x – 2026-11-08 00:5x AWST)