--- title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08) type: incident status: resolved created: 2026-11-09 updated: 2026-11-09 verified_on: 2026-11-09 review_after: 2026-11-16 confidence: high tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor] sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json] --- # The `conduit_push` `pre_tool_call` hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08) **Summary.** From **2026-11-07 14:14:03** to **2026-11-08 08:21:03** the `conduit_push` plugin's **sole `pre_tool_call` callback** overran Hermes's **30 s hook budget** and was skipped. Each skip suppresses the callback for a window, so the *pre-tool* side of tool dispatch degraded from *slow* to *blocked* — this is the fault that ate the 2026-11-08 morning and stretched that day's cron runs to **6 h** (*Mnemosyne Health Monitor* 12:00:51 → 18:00:42) and **1 h 13 m** (*FreshRSS Midday* 12:00:50 → 14:10:43). Fixed **2026-11-08 13:35 AWST** by raising `plugins.hook_callback_timeout` **30 → 120**. ## Evidence (live reads, 2026-11-08 23:5x – 2026-11-09 00:1x AWST) grep -c "callback _pre_tool_call timed out" ~/.hermes/logs/errors.log -> 58 # 2026-11-07: 12 lines 2026-11-08: 46 lines # first 2026-11-07 14:14:03,075 last 2026-11-08 08:21:03,863 grep -c "skipped after previous timeout" ~/.hermes/logs/*.log -> 89 The exact pair of lines (`~/.hermes/logs/errors.log`): 2026-11-08 08:19:40,942 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call skipped after previous timeout or while still running 2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping **The register (correction to a plausible-but-wrong diagnosis).** `conduit_push/__init__.py:119` is the **only** `pre_tool_call` registration on the host, and the plugin's caches are already in place — `session_titles.py:21 CACHE_SECONDS = 30.0`. Timed in-process by the 2026-11-08 session: **3.1 s cold, 0.00 s warm**. The *"the hook re-opens a 1.87 GB state.db per call, ~9 s each"* theory was therefore **wrong** — warm calls were free. The timeouts track **load spikes** on the 2-core CT460 (load average **5.89** at 13:27:50; `state.db` carrying a **40 MB WAL** checkpointing on the same spindle), not per-call database opens. ## Fix applied (2026-11-08 13:35 AWST) ~/.hermes/config.yaml plugins: hook_callback_timeout: 120 # was the built-in 30 s default - **Verified live:** `plugins.hook_callback_timeout = 120`. - **Verified effective:** no `_pre_tool_call` timeout has been logged since **08:21:03** — re-checked at the end of the fix window and again at audit time (2026-11-09 00:0x). The **58th** line is still the last. - **Deliberately NOT done:** rewriting the plugin's caching layer. The caches work; the budget was the binding constraint. **Next lever is `state.db` checkpoint/retention discipline**, not more caching. ## Consequence — the same signature landed in the cron ledger The 2026-11-08 03:00 *wiki-stocktake-maintenance* slot (`1a8e90ea2271`, 03:01:11 → 04:36:08) closed **`failed`** on a **new terminal signature for that job**: RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text. Send `continue` to let it summarize. *Working hypothesis, not proven:* a `pre_tool_call` block holding a tool result open is exactly what this error describes, and it sits inside the same 14 h timeout window. It is **consistent**, not established — the error string does not name the hook. ## Impact | item | effect | |---|---| | *Mnemosyne Health Monitor* `ea79a2107e36` | 00:01:20 → **08:29:05** (8 h 27 m) and 12:00:51 → **18:00:42** (5 h 59 m 51 s) — its two slowest slots on record, both delivered | | *wiki-stocktake-maintenance* `1a8e90ea2271` | 03:01:11 → 04:36:08 `failed`; **`failure_streak` 3**; a **third** unalerted `detected` incident row | | *FreshRSS Midday Briefing* `ecc1fc13655a` | 12:00:50 → **14:10:43** (2 h 09 m 53 s) | | *Tech & AI Newsletter Digest* `85edf6772bca` | an admission-read `database is locked` incident (`85edf6_0877ca1e0eac`) at **07:23:02**, **resolved** at 08:06:54 when the run completed | | every interactive tool call | slow, then suppressed — the user-visible symptom that opened the investigation | ## Verification evidence ~/.hermes/logs/errors.log (58 timeout lines; last 2026-11-08 08:21:03) ~/.hermes/config.yaml (plugins.hook_callback_timeout: 120) ~/.hermes/plugins/conduit_push/__init__.py:119,257 (the only pre_tool_call registration) ~/.hermes/plugins/conduit_push/session_titles.py:21 (CACHE_SECONDS = 30.0) ~/.hermes/cron/executions.db, cron_incidents, jobs.json (run windows, streak, incident rows) ## Open items - **`state.db` is 1.89 GB with 2,901 sessions / 197,682 messages and a live 40 MB WAL** — it is the load source behind the spikes. Checkpoint and retention policy are still undecided ([[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]). - Confirm whether the stocktake's `No reply: … tool result was still pending` failure is really the same fault; if it recurs while the hook budget is 120 s, this hypothesis is dead. - Decide whether a skipped `pre_tool_call` should **fail loudly** rather than silently degrade tool dispatch. ## See also - [[systems/conduit]] - [[systems/hermes-agent]] - [[systems/scheduled-tasks]] - [[systems/mnemosyne]] - [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] - [[current-state]]