Files
agent-estate-wiki/incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md

11 KiB
Raw Permalink Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08) incident active 2026-11-09 2026-11-17 2026-11-17 2026-11-20 high
cron
plugin
conduit
hook
timeout
state-db
load
watchdog
health-monitor
live-hermes-errors-log
live-hermes-agent-log
live-config-yaml
plugin-tree
live-state-db
live-executions-db
live-cron-incidents
live-jobs-json

The conduit_push pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)

Summary. From 2026-11-07 14:14:03 to 2026-11-08 08:21:03 the conduit_push plugin's sole pre_tool_call callback overran Hermes's 30 s hook budget and was skipped. Each skip suppresses the callback for a window, so the pre-tool side of tool dispatch degraded from slow to blocked — this is the fault that ate the 2026-11-08 morning and stretched that day's cron runs to 6 h (Mnemosyne Health Monitor 12:00:51 → 18:00:42) and 1 h 13 m (FreshRSS Midday 12:00:50 → 14:10:43). Fixed 2026-11-08 13:35 AWST by raising plugins.hook_callback_timeout 30 → 120.

Evidence (live reads, 2026-11-08 23:5x – 2026-11-09 00:1x AWST)

grep -c "callback _pre_tool_call timed out" ~/.hermes/logs/errors.log      -> 58
#  2026-11-07: 12 lines    2026-11-08: 46 lines
#  first 2026-11-07 14:14:03,075   last 2026-11-08 08:21:03,863
grep -c "skipped after previous timeout" ~/.hermes/logs/*.log              -> 89

The exact pair of lines (~/.hermes/logs/errors.log):

2026-11-08 08:19:40,942 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call skipped after previous timeout or while still running
2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping

The register (correction to a plausible-but-wrong diagnosis). conduit_push/__init__.py:119 is the only pre_tool_call registration on the host, and the plugin's caches are already in place — session_titles.py:21 CACHE_SECONDS = 30.0. Timed in-process by the 2026-11-08 session: 3.1 s cold, 0.00 s warm. The "the hook re-opens a 1.87 GB state.db per call, ~9 s each" theory was therefore wrong — warm calls were free. The timeouts track load spikes on the 2-core CT460 (load average 5.89 at 13:27:50; state.db carrying a 40 MB WAL checkpointing on the same spindle), not per-call database opens.

Fix applied (2026-11-08 13:35 AWST)

~/.hermes/config.yaml
plugins:
  hook_callback_timeout: 120        # was the built-in 30 s default
  • Verified live: plugins.hook_callback_timeout = 120.
  • Verified effective: no _pre_tool_call timeout has been logged since 08:21:03 — re-checked at the end of the fix window and again at audit time (2026-11-09 00:0x). The 58th line is still the last.
  • Deliberately NOT done: rewriting the plugin's caching layer. The caches work; the budget was the binding constraint. Next lever is state.db checkpoint/retention discipline, not more caching.

Consequence — the same signature landed in the cron ledger

The 2026-11-08 03:00 wiki-stocktake-maintenance slot (1a8e90ea2271, 03:01:11 → 04:36:08) closed failed on a new terminal signature for that job:

RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model
produced no follow-up text. Send `continue` to let it summarize.

Working hypothesis, not proven: a pre_tool_call block holding a tool result open is exactly what this error describes, and it sits inside the same 14 h timeout window. It is consistent, not established — the error string does not name the hook.

Impact

item effect
Mnemosyne Health Monitor ea79a2107e36 00:01:20 → 08:29:05 (8 h 27 m) and 12:00:51 → 18:00:42 (5 h 59 m 51 s) — its two slowest slots on record, both delivered
wiki-stocktake-maintenance 1a8e90ea2271 03:01:11 → 04:36:08 failed; failure_streak 3; a third unalerted detected incident row
FreshRSS Midday Briefing ecc1fc13655a 12:00:50 → 14:10:43 (2 h 09 m 53 s)
Tech & AI Newsletter Digest 85edf6772bca an admission-read database is locked incident (85edf6_0877ca1e0eac) at 07:23:02, resolved at 08:06:54 when the run completed
every interactive tool call slow, then suppressed — the user-visible symptom that opened the investigation

Verification evidence

~/.hermes/logs/errors.log                                 (58 timeout lines; last 2026-11-08 08:21:03)
~/.hermes/config.yaml                                     (plugins.hook_callback_timeout: 120)
~/.hermes/plugins/conduit_push/__init__.py:119,257        (the only pre_tool_call registration)
~/.hermes/plugins/conduit_push/session_titles.py:21       (CACHE_SECONDS = 30.0)
~/.hermes/cron/executions.db, cron_incidents, jobs.json   (run windows, streak, incident rows)

Open items

  • state.db is 1.89 GB with 2,901 sessions / 197,682 messages and a live 40 MB WAL — it is the load source behind the spikes. Checkpoint and retention policy are still undecided (incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts).
  • Confirm whether the stocktake's No reply: … tool result was still pending failure is really the same fault; if it recurs while the hook budget is 120 s, this hypothesis is dead.
  • Decide whether a skipped pre_tool_call should fail loudly rather than silently degrade tool dispatch.

Follow-up (2026-11-10 nightly audit)

  • The fix held; the cascade did not recur. errors.log (1,035 retained lines) contains no pre_tool_call timeout entry at all. Its only two hook lines are benign skips — on_session_end 2026-11-09 22:58:35 and pre_llm_call 2026-11-10 00:02:54 ("skipped after previous timeout or while still running").
  • No 11-09 run showed the cascade signature. Outside the two long-runners documented separately (stocktake 9 h 10 m, Job Radar Stage 1 9 h 09 m — both stalled in tool iteration, not in a hook), the longest 11-09 execution was 74 min, then 40 min and 11 min.
  • 2026-11-09 was the first failed-free cron day since 2026-11-05 (11-06: 1 failed, 11-07: 3, 11-08: 1, 11-09: 0 of 174), and unclosed incident rows went 3 → 0.
  • Residual: the two long-runners above are the same 9-hour-turn environment this bug leaves behind, and one (wiki-stocktake-maintenance) is pinned to free-stack — a combo recorded broken on 2026-08-14. The job-iteration budget stays in force.

Recurrence (2026-11-14 nightly audit) — the raised budget did not hold

Reopened. The pre_tool_call hook overran its raised 120 s budget again at 2026-11-14 00:05:40, on the wiki-audit job's own tool calls. The 30 -> 120 s fix of 2026-11-08 is therefore proven insufficient at current load, and this page is no longer resolved.

time (AWST) line
2026-11-14 00:05:40,043 Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping
2026-11-14 00:05:43 -> 00:06:38 10 x skipped after previous timeout or while still running, ~6 s apart
  • Source: ~/.hermes/logs/errors.log (single-source count; agent.log carries the same lines). By day: 2026-11-12 = 0 timeouts / 0 skips; 2026-11-13 = 0 / 0; 2026-11-14 = 1 / 10. It is the first pre_tool_call timeout in the retained log since the 2026-11-08 cascade — the 2026-11-08 lines themselves have since rotated out of errors.log, so this page's original 58-timeout figure still stands on the 2026-11-09 read and was not re-verifiable this window.
  • Cost this time: this audit's first skill_view and terminal calls were refused outright by the plugin (first-hand evidence), with dispatch recovering on its own by ~00:06:40 — roughly one minute of hard suppression. It is a single-event recurrence, not the 18-hour cascade. No cron run failed: 2026-11-13 settled 173 completed / 0 failed.
  • Load is the variable, and it is now ~2x the level recorded when the fault was diagnosed: load average **12.08 / 13.02 / 12.33** on 2 vCPU (2026-11-08 read: 5.89). state.db has grown 1.89 GB -> 2.04 GB (2,036,908,032 B; 2,977 sessions / 207,577 messages, was 2,901 / 197,682), and the root filesystem is 88 % used.
  • The earlier open hypothesis is now settled. "If it recurs while the hook budget is 120 s, this hypothesis is dead" — it recurred, so the load trigger is confirmed and there is no headroom left in the timeout: at load ~12 the callback needs >120 s. Re-raising the budget would only mask the fault. The remaining levers are unchanged and now urgent — state.db checkpoint/retention and reducing host load (see incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts).

Recurrence 2 (2026-11-15) — eleven timeouts in two clusters, 10 refusals

Recorded by the 2026-11-17 nightly audit while backing up the 11-15 window: the 2026-11-15 audit captured none of this. Exact errors.log counts for 11-15:

  • 11 × Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping — 5 in 01:07:17 → 01:25:43 and 6 in 12:18:15 → 12:48:40 (a two-cluster day, unlike 11-14's one).
  • 10 tool calls returned {"error": "pre_tool_call plugin callback timed out or is still running"} — hour 01: read_file ×2 (134.17 s and 0.43 s), execute_code ×4, patch ×1, read_file ×1 (121.03 s); hour 12: execute_code ×1. Call-level blocks reach 120.63 s, and the affected sessions are local-989b3642… (01:xx) and local-b6a1490f… (12:xx).
  • 11-14, for comparison: 1 timeout and 4 refusals, plus 4 plugin load failures.
  • 11-16 and 11-17: 0 timeouts, 0 refusals, 0 load failures. The episode did not carry over.

The loader half of the same 11-15 storm — 52 failed plugin loads, 519 discarded registrations (conduit_push 474 of them) and 2 pre_llm_call timeouts — is now on its own page, incidents/2026-11-15-plugin-load-timeout-storm. Its 12:18–12:49 cluster is the useful detail here: a hook-timeout cluster can occur with no accompanying loader failure, so the two share a cause class rather than a mechanism.

Effect on the trigger hypothesis recorded above. Amplitude is not a simple function of load average: the 11-15 episode ran at a load the audit cannot measure retroactively, while the 11-17 read (6.19 / 6.64 / 7.48) sits below the 11-14 figure of 12.08 with no recurrence — and a larger state.db (2.09 GB, WAL 40 MB, / at 88 %). Keep a load-average threshold out of this page's claim; keep state.db size, disk headroom and the gateway's own timing budgets in it.

See also