nightly audit 2026-11-09: pre_tool_call hook-timeout cascade (new incident page), hook budget 30->120s, state.db 1.89GB + corruption remnant, no-agent health monitor verified 73s, third silent stocktake incident, skills 299/67, freshness 0 stale/127 fresh of 130

This commit is contained in:
Tony0410
2026-11-09 00:21:01 +08:00
parent 4fd138bb18
commit e2f1ae7a0b
12 changed files with 422 additions and 29 deletions
@@ -0,0 +1,101 @@
---
title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)
type: incident
status: resolved
created: 2026-11-09
updated: 2026-11-09
verified_on: 2026-11-09
review_after: 2026-11-16
confidence: high
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
---
# The `conduit_push` `pre_tool_call` hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)
**Summary.** From **2026-11-07 14:14:03** to **2026-11-08 08:21:03** the `conduit_push` plugin's
**sole `pre_tool_call` callback** overran Hermes's **30 s hook budget** and was skipped. Each skip
suppresses the callback for a window, so the *pre-tool* side of tool dispatch degraded from *slow* to
*blocked* — this is the fault that ate the 2026-11-08 morning and stretched that day's cron runs to
**6 h** (*Mnemosyne Health Monitor* 12:00:51 → 18:00:42) and **1 h 13 m** (*FreshRSS Midday* 12:00:50 →
14:10:43). Fixed **2026-11-08 13:35 AWST** by raising `plugins.hook_callback_timeout` **30 → 120**.
## Evidence (live reads, 2026-11-08 23:5x – 2026-11-09 00:1x AWST)
grep -c "callback _pre_tool_call timed out" ~/.hermes/logs/errors.log -> 58
# 2026-11-07: 12 lines 2026-11-08: 46 lines
# first 2026-11-07 14:14:03,075 last 2026-11-08 08:21:03,863
grep -c "skipped after previous timeout" ~/.hermes/logs/*.log -> 89
The exact pair of lines (`~/.hermes/logs/errors.log`):
2026-11-08 08:19:40,942 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call skipped after previous timeout or while still running
2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
**The register (correction to a plausible-but-wrong diagnosis).** `conduit_push/__init__.py:119` is the
**only** `pre_tool_call` registration on the host, and the plugin's caches are already in place —
`session_titles.py:21 CACHE_SECONDS = 30.0`. Timed in-process by the 2026-11-08 session:
**3.1 s cold, 0.00 s warm**. The *"the hook re-opens a 1.87 GB state.db per call, ~9 s each"* theory was
therefore **wrong** — warm calls were free. The timeouts track **load spikes** on the 2-core CT460
(load average **5.89** at 13:27:50; `state.db` carrying a **40 MB WAL** checkpointing on the same spindle),
not per-call database opens.
## Fix applied (2026-11-08 13:35 AWST)
~/.hermes/config.yaml
plugins:
hook_callback_timeout: 120 # was the built-in 30 s default
- **Verified live:** `plugins.hook_callback_timeout = 120`.
- **Verified effective:** no `_pre_tool_call` timeout has been logged since **08:21:03** — re-checked at
the end of the fix window and again at audit time (2026-11-09 00:0x). The **58th** line is still the last.
- **Deliberately NOT done:** rewriting the plugin's caching layer. The caches work; the budget was the
binding constraint. **Next lever is `state.db` checkpoint/retention discipline**, not more caching.
## Consequence — the same signature landed in the cron ledger
The 2026-11-08 03:00 *wiki-stocktake-maintenance* slot (`1a8e90ea2271`, 03:01:11 → 04:36:08) closed
**`failed`** on a **new terminal signature for that job**:
RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model
produced no follow-up text. Send `continue` to let it summarize.
*Working hypothesis, not proven:* a `pre_tool_call` block holding a tool result open is exactly what this
error describes, and it sits inside the same 14 h timeout window. It is **consistent**, not established —
the error string does not name the hook.
## Impact
| item | effect |
|---|---|
| *Mnemosyne Health Monitor* `ea79a2107e36` | 00:01:20 → **08:29:05** (8 h 27 m) and 12:00:51 → **18:00:42** (5 h 59 m 51 s) — its two slowest slots on record, both delivered |
| *wiki-stocktake-maintenance* `1a8e90ea2271` | 03:01:11 → 04:36:08 `failed`; **`failure_streak` 3**; a **third** unalerted `detected` incident row |
| *FreshRSS Midday Briefing* `ecc1fc13655a` | 12:00:50 → **14:10:43** (2 h 09 m 53 s) |
| *Tech & AI Newsletter Digest* `85edf6772bca` | an admission-read `database is locked` incident (`85edf6_0877ca1e0eac`) at **07:23:02**, **resolved** at 08:06:54 when the run completed |
| every interactive tool call | slow, then suppressed — the user-visible symptom that opened the investigation |
## Verification evidence
~/.hermes/logs/errors.log (58 timeout lines; last 2026-11-08 08:21:03)
~/.hermes/config.yaml (plugins.hook_callback_timeout: 120)
~/.hermes/plugins/conduit_push/__init__.py:119,257 (the only pre_tool_call registration)
~/.hermes/plugins/conduit_push/session_titles.py:21 (CACHE_SECONDS = 30.0)
~/.hermes/cron/executions.db, cron_incidents, jobs.json (run windows, streak, incident rows)
## Open items
- **`state.db` is 1.89 GB with 2,901 sessions / 197,682 messages and a live 40 MB WAL** — it is the load
source behind the spikes. Checkpoint and retention policy are still undecided
([[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
- Confirm whether the stocktake's `No reply: … tool result was still pending` failure is really the same
fault; if it recurs while the hook budget is 120 s, this hypothesis is dead.
- Decide whether a skipped `pre_tool_call` should **fail loudly** rather than silently degrade tool dispatch.
## See also
- [[systems/conduit]]
- [[systems/hermes-agent]]
- [[systems/scheduled-tasks]]
- [[systems/mnemosyne]]
- [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
- [[current-state]]