5.8 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08) | incident | resolved | 2026-11-09 | 2026-11-09 | 2026-11-09 | 2026-11-16 | high |
|
|
The conduit_push pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)
Summary. From 2026-11-07 14:14:03 to 2026-11-08 08:21:03 the conduit_push plugin's
sole pre_tool_call callback overran Hermes's 30 s hook budget and was skipped. Each skip
suppresses the callback for a window, so the pre-tool side of tool dispatch degraded from slow to
blocked — this is the fault that ate the 2026-11-08 morning and stretched that day's cron runs to
6 h (Mnemosyne Health Monitor 12:00:51 → 18:00:42) and 1 h 13 m (FreshRSS Midday 12:00:50 →
14:10:43). Fixed 2026-11-08 13:35 AWST by raising plugins.hook_callback_timeout 30 → 120.
Evidence (live reads, 2026-11-08 23:5x – 2026-11-09 00:1x AWST)
grep -c "callback _pre_tool_call timed out" ~/.hermes/logs/errors.log -> 58
# 2026-11-07: 12 lines 2026-11-08: 46 lines
# first 2026-11-07 14:14:03,075 last 2026-11-08 08:21:03,863
grep -c "skipped after previous timeout" ~/.hermes/logs/*.log -> 89
The exact pair of lines (~/.hermes/logs/errors.log):
2026-11-08 08:19:40,942 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call skipped after previous timeout or while still running
2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
The register (correction to a plausible-but-wrong diagnosis). conduit_push/__init__.py:119 is the
only pre_tool_call registration on the host, and the plugin's caches are already in place —
session_titles.py:21 CACHE_SECONDS = 30.0. Timed in-process by the 2026-11-08 session:
3.1 s cold, 0.00 s warm. The "the hook re-opens a 1.87 GB state.db per call, ~9 s each" theory was
therefore wrong — warm calls were free. The timeouts track load spikes on the 2-core CT460
(load average 5.89 at 13:27:50; state.db carrying a 40 MB WAL checkpointing on the same spindle),
not per-call database opens.
Fix applied (2026-11-08 13:35 AWST)
~/.hermes/config.yaml
plugins:
hook_callback_timeout: 120 # was the built-in 30 s default
- Verified live:
plugins.hook_callback_timeout = 120. - Verified effective: no
_pre_tool_calltimeout has been logged since 08:21:03 — re-checked at the end of the fix window and again at audit time (2026-11-09 00:0x). The 58th line is still the last. - Deliberately NOT done: rewriting the plugin's caching layer. The caches work; the budget was the
binding constraint. Next lever is
state.dbcheckpoint/retention discipline, not more caching.
Consequence — the same signature landed in the cron ledger
The 2026-11-08 03:00 wiki-stocktake-maintenance slot (1a8e90ea2271, 03:01:11 → 04:36:08) closed
failed on a new terminal signature for that job:
RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model
produced no follow-up text. Send `continue` to let it summarize.
Working hypothesis, not proven: a pre_tool_call block holding a tool result open is exactly what this
error describes, and it sits inside the same 14 h timeout window. It is consistent, not established —
the error string does not name the hook.
Impact
| item | effect |
|---|---|
Mnemosyne Health Monitor ea79a2107e36 |
00:01:20 → 08:29:05 (8 h 27 m) and 12:00:51 → 18:00:42 (5 h 59 m 51 s) — its two slowest slots on record, both delivered |
wiki-stocktake-maintenance 1a8e90ea2271 |
03:01:11 → 04:36:08 failed; failure_streak 3; a third unalerted detected incident row |
FreshRSS Midday Briefing ecc1fc13655a |
12:00:50 → 14:10:43 (2 h 09 m 53 s) |
Tech & AI Newsletter Digest 85edf6772bca |
an admission-read database is locked incident (85edf6_0877ca1e0eac) at 07:23:02, resolved at 08:06:54 when the run completed |
| every interactive tool call | slow, then suppressed — the user-visible symptom that opened the investigation |
Verification evidence
~/.hermes/logs/errors.log (58 timeout lines; last 2026-11-08 08:21:03)
~/.hermes/config.yaml (plugins.hook_callback_timeout: 120)
~/.hermes/plugins/conduit_push/__init__.py:119,257 (the only pre_tool_call registration)
~/.hermes/plugins/conduit_push/session_titles.py:21 (CACHE_SECONDS = 30.0)
~/.hermes/cron/executions.db, cron_incidents, jobs.json (run windows, streak, incident rows)
Open items
state.dbis 1.89 GB with 2,901 sessions / 197,682 messages and a live 40 MB WAL — it is the load source behind the spikes. Checkpoint and retention policy are still undecided (incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts).- Confirm whether the stocktake's
No reply: … tool result was still pendingfailure is really the same fault; if it recurs while the hook budget is 120 s, this hypothesis is dead. - Decide whether a skipped
pre_tool_callshould fail loudly rather than silently degrade tool dispatch.