Files
agent-estate-wiki/incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
T

166 lines
11 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)
type: incident
status: active
created: 2026-11-09
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-20
confidence: high
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
---
# The `conduit_push` `pre_tool_call` hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)
**Summary.** From **2026-11-07 14:14:03** to **2026-11-08 08:21:03** the `conduit_push` plugin's
**sole `pre_tool_call` callback** overran Hermes's **30 s hook budget** and was skipped. Each skip
suppresses the callback for a window, so the *pre-tool* side of tool dispatch degraded from *slow* to
*blocked* — this is the fault that ate the 2026-11-08 morning and stretched that day's cron runs to
**6 h** (*Mnemosyne Health Monitor* 12:00:51 → 18:00:42) and **1 h 13 m** (*FreshRSS Midday* 12:00:50 →
14:10:43). Fixed **2026-11-08 13:35 AWST** by raising `plugins.hook_callback_timeout` **30 → 120**.
## Evidence (live reads, 2026-11-08 23:5x – 2026-11-09 00:1x AWST)
grep -c "callback _pre_tool_call timed out" ~/.hermes/logs/errors.log -> 58
# 2026-11-07: 12 lines 2026-11-08: 46 lines
# first 2026-11-07 14:14:03,075 last 2026-11-08 08:21:03,863
grep -c "skipped after previous timeout" ~/.hermes/logs/*.log -> 89
The exact pair of lines (`~/.hermes/logs/errors.log`):
2026-11-08 08:19:40,942 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call skipped after previous timeout or while still running
2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
**The register (correction to a plausible-but-wrong diagnosis).** `conduit_push/__init__.py:119` is the
**only** `pre_tool_call` registration on the host, and the plugin's caches are already in place —
`session_titles.py:21 CACHE_SECONDS = 30.0`. Timed in-process by the 2026-11-08 session:
**3.1 s cold, 0.00 s warm**. The *"the hook re-opens a 1.87 GB state.db per call, ~9 s each"* theory was
therefore **wrong** — warm calls were free. The timeouts track **load spikes** on the 2-core CT460
(load average **5.89** at 13:27:50; `state.db` carrying a **40 MB WAL** checkpointing on the same spindle),
not per-call database opens.
## Fix applied (2026-11-08 13:35 AWST)
~/.hermes/config.yaml
plugins:
hook_callback_timeout: 120 # was the built-in 30 s default
- **Verified live:** `plugins.hook_callback_timeout = 120`.
- **Verified effective:** no `_pre_tool_call` timeout has been logged since **08:21:03** — re-checked at
the end of the fix window and again at audit time (2026-11-09 00:0x). The **58th** line is still the last.
- **Deliberately NOT done:** rewriting the plugin's caching layer. The caches work; the budget was the
binding constraint. **Next lever is `state.db` checkpoint/retention discipline**, not more caching.
## Consequence — the same signature landed in the cron ledger
The 2026-11-08 03:00 *wiki-stocktake-maintenance* slot (`1a8e90ea2271`, 03:01:11 → 04:36:08) closed
**`failed`** on a **new terminal signature for that job**:
RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model
produced no follow-up text. Send `continue` to let it summarize.
*Working hypothesis, not proven:* a `pre_tool_call` block holding a tool result open is exactly what this
error describes, and it sits inside the same 14 h timeout window. It is **consistent**, not established —
the error string does not name the hook.
## Impact
| item | effect |
|---|---|
| *Mnemosyne Health Monitor* `ea79a2107e36` | 00:01:20 → **08:29:05** (8 h 27 m) and 12:00:51 → **18:00:42** (5 h 59 m 51 s) — its two slowest slots on record, both delivered |
| *wiki-stocktake-maintenance* `1a8e90ea2271` | 03:01:11 → 04:36:08 `failed`; **`failure_streak` 3**; a **third** unalerted `detected` incident row |
| *FreshRSS Midday Briefing* `ecc1fc13655a` | 12:00:50 → **14:10:43** (2 h 09 m 53 s) |
| *Tech & AI Newsletter Digest* `85edf6772bca` | an admission-read `database is locked` incident (`85edf6_0877ca1e0eac`) at **07:23:02**, **resolved** at 08:06:54 when the run completed |
| every interactive tool call | slow, then suppressed — the user-visible symptom that opened the investigation |
## Verification evidence
~/.hermes/logs/errors.log (58 timeout lines; last 2026-11-08 08:21:03)
~/.hermes/config.yaml (plugins.hook_callback_timeout: 120)
~/.hermes/plugins/conduit_push/__init__.py:119,257 (the only pre_tool_call registration)
~/.hermes/plugins/conduit_push/session_titles.py:21 (CACHE_SECONDS = 30.0)
~/.hermes/cron/executions.db, cron_incidents, jobs.json (run windows, streak, incident rows)
## Open items
- **`state.db` is 1.89 GB with 2,901 sessions / 197,682 messages and a live 40 MB WAL** — it is the load
source behind the spikes. Checkpoint and retention policy are still undecided
([[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
- Confirm whether the stocktake's `No reply: … tool result was still pending` failure is really the same
fault; if it recurs while the hook budget is 120 s, this hypothesis is dead.
- Decide whether a skipped `pre_tool_call` should **fail loudly** rather than silently degrade tool dispatch.
## Follow-up (2026-11-10 nightly audit)
- **The fix held; the cascade did not recur.** `errors.log` (1,035 retained lines) contains **no `pre_tool_call` timeout entry at all**. Its only two hook lines are benign skips — `on_session_end` 2026-11-09 22:58:35 and `pre_llm_call` 2026-11-10 00:02:54 (*"skipped after previous timeout or while still running"*).
- **No 11-09 run showed the cascade signature.** Outside the two long-runners documented separately (stocktake **9 h 10 m**, Job Radar Stage 1 **9 h 09 m** — both stalled in *tool iteration*, not in a hook), the longest 11-09 execution was **74 min**, then 40 min and 11 min.
- **2026-11-09 was the first `failed`-free cron day since 2026-11-05** (11-06: 1 failed, 11-07: 3, 11-08: 1, 11-09: **0 of 174**), and unclosed incident rows went 3 → 0.
- **Residual:** the two long-runners above are the same 9-hour-turn environment this bug leaves behind, and one (`wiki-stocktake-maintenance`) is pinned to `free-stack` — a combo recorded broken on 2026-08-14. The job-iteration budget stays in force.
## Recurrence (2026-11-14 nightly audit) — the raised budget did not hold
**Reopened.** The `pre_tool_call` hook overran its **raised 120 s** budget again at **2026-11-14 00:05:40**,
on the wiki-audit job's own tool calls. The 30 -> 120 s fix of 2026-11-08 is therefore **proven
insufficient at current load**, and this page is no longer `resolved`.
| time (AWST) | line |
|---|---|
| 2026-11-14 00:05:40,043 | `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` |
| 2026-11-14 00:05:43 -> 00:06:38 | **10** x `skipped after previous timeout or while still running`, ~6 s apart |
- **Source:** `~/.hermes/logs/errors.log` (single-source count; `agent.log` carries the same lines).
By day: **2026-11-12 = 0 timeouts / 0 skips; 2026-11-13 = 0 / 0; 2026-11-14 = 1 / 10.** It is the
**first `pre_tool_call` timeout in the retained log since the 2026-11-08 cascade** — the 2026-11-08
lines themselves have since **rotated out** of `errors.log`, so this page's original 58-timeout figure
still stands on the 2026-11-09 read and was **not re-verifiable** this window.
- **Cost this time:** this audit's first `skill_view` and `terminal` calls were **refused outright** by the
plugin (first-hand evidence), with dispatch recovering on its own by ~00:06:40 — roughly **one minute**
of hard suppression. It is a **single-event recurrence, not the 18-hour cascade**. **No cron run failed:**
2026-11-13 settled **173 `completed` / 0 `failed`**.
- **Load is the variable, and it is now ~2x the level recorded when the fault was diagnosed:**
`load average **12.08 / 13.02 / 12.33**` on **2 vCPU** (2026-11-08 read: **5.89**). `state.db` has grown
**1.89 GB -> 2.04 GB** (**2,036,908,032 B**; **2,977 sessions / 207,577 messages**, was 2,901 / 197,682),
and the root filesystem is **88 % used**.
- **The earlier open hypothesis is now settled.** *"If it recurs while the hook budget is 120 s, this
hypothesis is dead"* — it recurred, so the load trigger is **confirmed** and there is **no headroom left
in the timeout**: at load ~12 the callback needs >120 s. **Re-raising the budget would only mask the
fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and
reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
## Recurrence 2 (2026-11-15) — eleven timeouts in two clusters, 10 refusals
Recorded by the **2026-11-17 nightly audit** while backing up the 11-15 window: the **2026-11-15 audit
captured none of this**. Exact `errors.log` counts for 11-15:
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` —
**5** in 01:07:17 → 01:25:43 and **6** in 12:18:15 → 12:48:40 (a two-cluster day, unlike 11-14's one).
- **10** tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
— hour 01: `read_file` ×2 (134.17 s and 0.43 s), `execute_code` ×4, `patch` ×1, `read_file` ×1
(121.03 s); hour 12: `execute_code` ×1. Call-level blocks reach **120.63 s**, and the affected sessions
are `local-989b3642…` (01:xx) and `local-b6a1490f…` (12:xx).
- 11-14, for comparison: **1** timeout and **4** refusals, plus **4** plugin load failures.
- **11-16 and 11-17: 0 timeouts, 0 refusals, 0 load failures.** The episode did not carry over.
The loader half of the same 11-15 storm — **52** failed plugin loads, **519** discarded registrations
(`conduit_push` 474 of them) and **2** `pre_llm_call` timeouts — is now on its own page,
[[incidents/2026-11-15-plugin-load-timeout-storm]]. Its 12:18–12:49 cluster is the useful detail here:
**a hook-timeout cluster can occur with no accompanying loader failure**, so the two share a cause class
rather than a mechanism.
**Effect on the trigger hypothesis recorded above.** Amplitude is not a simple function of load average:
the 11-15 episode ran at a load the audit cannot measure retroactively, while the 11-17 read
(**6.19 / 6.64 / 7.48**) sits *below* the 11-14 figure of 12.08 with **no** recurrence — and a *larger*
`state.db` (2.09 GB, WAL 40 MB, `/` at 88 %). Keep a **load-average threshold out of this page's claim**;
keep **`state.db` size, disk headroom and the gateway's own timing budgets** in it.
## See also
- [[systems/conduit]]
- [[systems/hermes-agent]]
- [[systems/scheduled-tasks]]
- [[systems/mnemosyne]]
- [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
- [[current-state]]