nightly audit 2026-11-09: pre_tool_call hook-timeout cascade (new incident page), hook budget 30->120s, state.db 1.89GB + corruption remnant, no-agent health monitor verified 73s, third silent stocktake incident, skills 299/67, freshness 0 stale/127 fresh of 130

This commit is contained in:
Tony0410
2026-11-09 00:21:01 +08:00
parent 4fd138bb18
commit e2f1ae7a0b
12 changed files with 422 additions and 29 deletions
+31 -3
View File
@@ -3,11 +3,11 @@ title: Conduit
type: system
status: active
created: 2026-10-24
updated: 2026-11-08
verified_on: 2026-11-08
updated: 2026-11-09
verified_on: 2026-11-09
review_after: 2026-12-08
confidence: high
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl]
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl, hook, timeout, pre-tool-call]
sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329, session-20260908_031541_49c051]
---
@@ -230,7 +230,35 @@ The plugin updated itself in-window and its own ledger records the final step as
- The plugin remained enabled and functional throughout; no user-facing conduit fault was recorded on
2026-11-07.
## 2026-11-08 13:2x–13:3x AWST — its `pre_tool_call` hook was jamming **every** tool call
The plugin is the **only** `pre_tool_call` registrant on the host (`conduit_push/__init__.py:119`), and for
**~18 hours** that callback overran Hermes's **30 s** hook budget and was skipped:
~/.hermes/logs/errors.log
2026-11-07 14:14:03,075 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
... (58 lines total)
2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
- **58 timeout lines** in `errors.log` (**12** on 2026-11-07, **46** on 2026-11-08) plus **89**
*"skipped after previous timeout or while still running"* suppression lines across the logs. **The last
occurrence is 2026-11-08 08:21:03** — nothing since the fix below.
- **The plugin's caching was NOT the fault — this was measured, not assumed.** `session_titles.py:21
CACHE_SECONDS = 30.0`; timed in-process, a hook call costs **3.1 s cold and 0.00 s warm**. The
*"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**. The timeouts track **load
spikes** on the 2-core CT460 (`load average 5.89` at 13:27:50; `state.db` with a **40 MB WAL**
checkpointing on the same spindle).
- **Fix:** `plugins.hook_callback_timeout` **30 → 120** in `~/.hermes/config.yaml` (2026-11-08 13:35 AWST) —
verified live in config, and verified *effective* (no timeout logged after 08:21:03).
- **Cost of the fault:** *Mnemosyne Health Monitor* ran **8 h 27 m** and **5 h 59 m 51 s** on its two
2026-11-08 slots; *wiki-stocktake-maintenance* failed at 04:36:08 on *"No reply: the turn stopped while a
tool result was still pending"*; *FreshRSS Midday* took **2 h 09 m 53 s**.
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
## Open items
- **Hook budget is 120 s, not 30 s — watch it.** A second overrun would eat two minutes per tool call.
The binding constraint is `state.db` (1.89 GB, 2,901 sessions, live 40 MB WAL), not this plugin.
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
- ✅ **Resolved 2026-11-01** — the `plugin_api.py` timeout patch is no longer needed: the tree is clean
at `0.122.2` and `MEMORY_TIMEOUT_S` ships upstream in `dashboard/api/memory.py`.
- **Verify a call end-to-end from the phone.** The 2026-10-29 08:49:36 call has no recorded outcome,