--- title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08) type: incident status: active created: 2026-11-09 updated: 2026-11-14 verified_on: 2026-11-14 review_after: 2026-11-17 confidence: high tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor] sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json] --- # The `conduit_push` `pre_tool_call` hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08) **Summary.** From **2026-11-07 14:14:03** to **2026-11-08 08:21:03** the `conduit_push` plugin's **sole `pre_tool_call` callback** overran Hermes's **30 s hook budget** and was skipped. Each skip suppresses the callback for a window, so the *pre-tool* side of tool dispatch degraded from *slow* to *blocked* — this is the fault that ate the 2026-11-08 morning and stretched that day's cron runs to **6 h** (*Mnemosyne Health Monitor* 12:00:51 → 18:00:42) and **1 h 13 m** (*FreshRSS Midday* 12:00:50 → 14:10:43). Fixed **2026-11-08 13:35 AWST** by raising `plugins.hook_callback_timeout` **30 → 120**. ## Evidence (live reads, 2026-11-08 23:5x – 2026-11-09 00:1x AWST) grep -c "callback _pre_tool_call timed out" ~/.hermes/logs/errors.log -> 58 # 2026-11-07: 12 lines 2026-11-08: 46 lines # first 2026-11-07 14:14:03,075 last 2026-11-08 08:21:03,863 grep -c "skipped after previous timeout" ~/.hermes/logs/*.log -> 89 The exact pair of lines (`~/.hermes/logs/errors.log`): 2026-11-08 08:19:40,942 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call skipped after previous timeout or while still running 2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping **The register (correction to a plausible-but-wrong diagnosis).** `conduit_push/__init__.py:119` is the **only** `pre_tool_call` registration on the host, and the plugin's caches are already in place — `session_titles.py:21 CACHE_SECONDS = 30.0`. Timed in-process by the 2026-11-08 session: **3.1 s cold, 0.00 s warm**. The *"the hook re-opens a 1.87 GB state.db per call, ~9 s each"* theory was therefore **wrong** — warm calls were free. The timeouts track **load spikes** on the 2-core CT460 (load average **5.89** at 13:27:50; `state.db` carrying a **40 MB WAL** checkpointing on the same spindle), not per-call database opens. ## Fix applied (2026-11-08 13:35 AWST) ~/.hermes/config.yaml plugins: hook_callback_timeout: 120 # was the built-in 30 s default - **Verified live:** `plugins.hook_callback_timeout = 120`. - **Verified effective:** no `_pre_tool_call` timeout has been logged since **08:21:03** — re-checked at the end of the fix window and again at audit time (2026-11-09 00:0x). The **58th** line is still the last. - **Deliberately NOT done:** rewriting the plugin's caching layer. The caches work; the budget was the binding constraint. **Next lever is `state.db` checkpoint/retention discipline**, not more caching. ## Consequence — the same signature landed in the cron ledger The 2026-11-08 03:00 *wiki-stocktake-maintenance* slot (`1a8e90ea2271`, 03:01:11 → 04:36:08) closed **`failed`** on a **new terminal signature for that job**: RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text. Send `continue` to let it summarize. *Working hypothesis, not proven:* a `pre_tool_call` block holding a tool result open is exactly what this error describes, and it sits inside the same 14 h timeout window. It is **consistent**, not established — the error string does not name the hook. ## Impact | item | effect | |---|---| | *Mnemosyne Health Monitor* `ea79a2107e36` | 00:01:20 → **08:29:05** (8 h 27 m) and 12:00:51 → **18:00:42** (5 h 59 m 51 s) — its two slowest slots on record, both delivered | | *wiki-stocktake-maintenance* `1a8e90ea2271` | 03:01:11 → 04:36:08 `failed`; **`failure_streak` 3**; a **third** unalerted `detected` incident row | | *FreshRSS Midday Briefing* `ecc1fc13655a` | 12:00:50 → **14:10:43** (2 h 09 m 53 s) | | *Tech & AI Newsletter Digest* `85edf6772bca` | an admission-read `database is locked` incident (`85edf6_0877ca1e0eac`) at **07:23:02**, **resolved** at 08:06:54 when the run completed | | every interactive tool call | slow, then suppressed — the user-visible symptom that opened the investigation | ## Verification evidence ~/.hermes/logs/errors.log (58 timeout lines; last 2026-11-08 08:21:03) ~/.hermes/config.yaml (plugins.hook_callback_timeout: 120) ~/.hermes/plugins/conduit_push/__init__.py:119,257 (the only pre_tool_call registration) ~/.hermes/plugins/conduit_push/session_titles.py:21 (CACHE_SECONDS = 30.0) ~/.hermes/cron/executions.db, cron_incidents, jobs.json (run windows, streak, incident rows) ## Open items - **`state.db` is 1.89 GB with 2,901 sessions / 197,682 messages and a live 40 MB WAL** — it is the load source behind the spikes. Checkpoint and retention policy are still undecided ([[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]). - Confirm whether the stocktake's `No reply: … tool result was still pending` failure is really the same fault; if it recurs while the hook budget is 120 s, this hypothesis is dead. - Decide whether a skipped `pre_tool_call` should **fail loudly** rather than silently degrade tool dispatch. ## Follow-up (2026-11-10 nightly audit) - **The fix held; the cascade did not recur.** `errors.log` (1,035 retained lines) contains **no `pre_tool_call` timeout entry at all**. Its only two hook lines are benign skips — `on_session_end` 2026-11-09 22:58:35 and `pre_llm_call` 2026-11-10 00:02:54 (*"skipped after previous timeout or while still running"*). - **No 11-09 run showed the cascade signature.** Outside the two long-runners documented separately (stocktake **9 h 10 m**, Job Radar Stage 1 **9 h 09 m** — both stalled in *tool iteration*, not in a hook), the longest 11-09 execution was **74 min**, then 40 min and 11 min. - **2026-11-09 was the first `failed`-free cron day since 2026-11-05** (11-06: 1 failed, 11-07: 3, 11-08: 1, 11-09: **0 of 174**), and unclosed incident rows went 3 → 0. - **Residual:** the two long-runners above are the same 9-hour-turn environment this bug leaves behind, and one (`wiki-stocktake-maintenance`) is pinned to `free-stack` — a combo recorded broken on 2026-08-14. The job-iteration budget stays in force. ## Recurrence (2026-11-14 nightly audit) — the raised budget did not hold **Reopened.** The `pre_tool_call` hook overran its **raised 120 s** budget again at **2026-11-14 00:05:40**, on the wiki-audit job's own tool calls. The 30 -> 120 s fix of 2026-11-08 is therefore **proven insufficient at current load**, and this page is no longer `resolved`. | time (AWST) | line | |---|---| | 2026-11-14 00:05:40,043 | `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` | | 2026-11-14 00:05:43 -> 00:06:38 | **10** x `skipped after previous timeout or while still running`, ~6 s apart | - **Source:** `~/.hermes/logs/errors.log` (single-source count; `agent.log` carries the same lines). By day: **2026-11-12 = 0 timeouts / 0 skips; 2026-11-13 = 0 / 0; 2026-11-14 = 1 / 10.** It is the **first `pre_tool_call` timeout in the retained log since the 2026-11-08 cascade** — the 2026-11-08 lines themselves have since **rotated out** of `errors.log`, so this page's original 58-timeout figure still stands on the 2026-11-09 read and was **not re-verifiable** this window. - **Cost this time:** this audit's first `skill_view` and `terminal` calls were **refused outright** by the plugin (first-hand evidence), with dispatch recovering on its own by ~00:06:40 — roughly **one minute** of hard suppression. It is a **single-event recurrence, not the 18-hour cascade**. **No cron run failed:** 2026-11-13 settled **173 `completed` / 0 `failed`**. - **Load is the variable, and it is now ~2x the level recorded when the fault was diagnosed:** `load average **12.08 / 13.02 / 12.33**` on **2 vCPU** (2026-11-08 read: **5.89**). `state.db` has grown **1.89 GB -> 2.04 GB** (**2,036,908,032 B**; **2,977 sessions / 207,577 messages**, was 2,901 / 197,682), and the root filesystem is **88 % used**. - **The earlier open hypothesis is now settled.** *"If it recurs while the hook budget is 120 s, this hypothesis is dead"* — it recurred, so the load trigger is **confirmed** and there is **no headroom left in the timeout**: at load ~12 the callback needs >120 s. **Re-raising the budget would only mask the fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]). ## See also - [[systems/conduit]] - [[systems/hermes-agent]] - [[systems/scheduled-tasks]] - [[systems/mnemosyne]] - [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] - [[current-state]]