nightly audit 2026-11-09: pre_tool_call hook-timeout cascade (new incident page), hook budget 30->120s, state.db 1.89GB + corruption remnant, no-agent health monitor verified 73s, third silent stocktake incident, skills 299/67, freshness 0 stale/127 fresh of 130
This commit is contained in:
+31
-3
@@ -3,11 +3,11 @@ title: Conduit
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-10-24
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
updated: 2026-11-09
|
||||
verified_on: 2026-11-09
|
||||
review_after: 2026-12-08
|
||||
confidence: high
|
||||
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl]
|
||||
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl, hook, timeout, pre-tool-call]
|
||||
sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329, session-20260908_031541_49c051]
|
||||
---
|
||||
|
||||
@@ -230,7 +230,35 @@ The plugin updated itself in-window and its own ledger records the final step as
|
||||
- The plugin remained enabled and functional throughout; no user-facing conduit fault was recorded on
|
||||
2026-11-07.
|
||||
|
||||
## 2026-11-08 13:2x–13:3x AWST — its `pre_tool_call` hook was jamming **every** tool call
|
||||
|
||||
The plugin is the **only** `pre_tool_call` registrant on the host (`conduit_push/__init__.py:119`), and for
|
||||
**~18 hours** that callback overran Hermes's **30 s** hook budget and was skipped:
|
||||
|
||||
~/.hermes/logs/errors.log
|
||||
2026-11-07 14:14:03,075 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
|
||||
... (58 lines total)
|
||||
2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
|
||||
|
||||
- **58 timeout lines** in `errors.log` (**12** on 2026-11-07, **46** on 2026-11-08) plus **89**
|
||||
*"skipped after previous timeout or while still running"* suppression lines across the logs. **The last
|
||||
occurrence is 2026-11-08 08:21:03** — nothing since the fix below.
|
||||
- **The plugin's caching was NOT the fault — this was measured, not assumed.** `session_titles.py:21
|
||||
CACHE_SECONDS = 30.0`; timed in-process, a hook call costs **3.1 s cold and 0.00 s warm**. The
|
||||
*"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**. The timeouts track **load
|
||||
spikes** on the 2-core CT460 (`load average 5.89` at 13:27:50; `state.db` with a **40 MB WAL**
|
||||
checkpointing on the same spindle).
|
||||
- **Fix:** `plugins.hook_callback_timeout` **30 → 120** in `~/.hermes/config.yaml` (2026-11-08 13:35 AWST) —
|
||||
verified live in config, and verified *effective* (no timeout logged after 08:21:03).
|
||||
- **Cost of the fault:** *Mnemosyne Health Monitor* ran **8 h 27 m** and **5 h 59 m 51 s** on its two
|
||||
2026-11-08 slots; *wiki-stocktake-maintenance* failed at 04:36:08 on *"No reply: the turn stopped while a
|
||||
tool result was still pending"*; *FreshRSS Midday* took **2 h 09 m 53 s**.
|
||||
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
|
||||
## Open items
|
||||
- **Hook budget is 120 s, not 30 s — watch it.** A second overrun would eat two minutes per tool call.
|
||||
The binding constraint is `state.db` (1.89 GB, 2,901 sessions, live 40 MB WAL), not this plugin.
|
||||
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
- ✅ **Resolved 2026-11-01** — the `plugin_api.py` timeout patch is no longer needed: the tree is clean
|
||||
at `0.122.2` and `MEMORY_TIMEOUT_S` ships upstream in `dashboard/api/memory.py`.
|
||||
- **Verify a call end-to-end from the phone.** The 2026-10-29 08:49:36 call has no recorded outcome,
|
||||
|
||||
Reference in New Issue
Block a user