nightly audit 2026-11-09: pre_tool_call hook-timeout cascade (new incident page), hook budget 30->120s, state.db 1.89GB + corruption remnant, no-agent health monitor verified 73s, third silent stocktake incident, skills 299/67, freshness 0 stale/127 fresh of 130
This commit is contained in:
+37
-3
@@ -3,8 +3,8 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
updated: 2026-11-09
|
||||
verified_on: 2026-11-09
|
||||
review_after: 2026-11-09
|
||||
confidence: high
|
||||
tags: [system]
|
||||
@@ -12,7 +12,41 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-11-08
|
||||
Updated: 2026-11-09
|
||||
|
||||
## Recent Changes (2026-11-09, nightly audit; live probes 2026-11-08 23:5x – 2026-11-09 00:2x AWST)
|
||||
|
||||
- ⚠ **The `conduit_push` `pre_tool_call` hook was the day's fault, and it is fixed — `plugins.hook_callback_timeout` 30 → 120 s.**
|
||||
The plugin is the **only** `pre_tool_call` registrant on the host, and its callback overran Hermes's 30 s
|
||||
hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: 12
|
||||
lines on 11-07, 46 on 11-08; plus **89** *"skipped after previous timeout or while still running"*
|
||||
suppression lines). Every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST**;
|
||||
the last timeout line is still 08:21:03, re-verified at audit time. **The plugin's caching was measured,
|
||||
not assumed, and was innocent** — 3.1 s cold / 0.00 s warm (`session_titles.py:21 CACHE_SECONDS = 30.0`);
|
||||
the trigger is **load** on the 2-core CT460 (load average **5.89**, `state.db` with a **40 MB WAL**).
|
||||
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]], [[systems/conduit]]
|
||||
- ⚠ **`state.db` is now 1.89 GB** (1,890,656,256 B; **2,901 sessions / 197,682 messages**, live
|
||||
**40,157,672 B** WAL) — up from 1.7 GB, and it is the load source behind the spike above. The 2026-08-11
|
||||
corruption remnant **`state.db.corrupted.20260811_151052` (624.1 MB)** was **deleted** in this window;
|
||||
`state.db.recovered.recovery.json` (14,566 B) remains, and `mnemosyne.db.corrupted.20260809_120404`
|
||||
(23.6 MB) was **not** cleaned. → [[systems/hermes-agent]], [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]
|
||||
- ✅ **The Mnemosyne Health Monitor's no-agent conversion is verified end-to-end.** The 18:00 slot (agent
|
||||
mode) ran **12:00:51 → 18:00:42 = 5 h 59 m 51 s**; the converted job's first slot ran
|
||||
**2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, **silent**, writing a **159 B** output file against
|
||||
**105,982 B** for the last agent-mode one. Live coverage **working 1030/1030 = 100.0 %, episodic
|
||||
1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet. → [[systems/mnemosyne]], [[systems/scheduled-tasks]]
|
||||
- ⚠ **The stocktake failed a third consecutive day, on a new signature.** `1a8e90ea2271` ran
|
||||
**2026-11-08 03:01:11 → 04:36:08** and died on `RuntimeError: ⚠️ No reply: the turn stopped while a tool
|
||||
result was still pending and the model produced no follow-up text.` — the previous two red days read
|
||||
*"lost its durable fire claim ownership"*. **`failure_streak` 3.** *Working hypothesis, not proven:* the
|
||||
hook above holds a tool result open in exactly the way this error describes. **Unclosed incident rows
|
||||
2 → 3**, and all three are silent (`detected`, `alerted_at: null`) — but **the alert path works**: the
|
||||
*Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` at **08:06:54**.
|
||||
→ [[systems/scheduled-tasks]], [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
|
||||
- **2026-11-08 settled: 169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit). Deliveries
|
||||
**8 / 8 `delivered`**. Fleet **29 jobs (26 / 3)** — a **sixth** consecutive audit at the same count, though
|
||||
`updated_at` moved to `2026-11-09T00:06:50+08:00` with no definition change. Nothing added, retired,
|
||||
disabled or repinned. → [[systems/scheduled-tasks]]
|
||||
|
||||
## Recent Changes (2026-11-08, nightly audit; live probes 2026-11-07 23:5x – 2026-11-08 00:5x AWST)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user