9.1 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08) | incident | active | 2026-11-09 | 2026-11-14 | 2026-11-14 | 2026-11-17 | high |
|
|
The conduit_push pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)
Summary. From 2026-11-07 14:14:03 to 2026-11-08 08:21:03 the conduit_push plugin's
sole pre_tool_call callback overran Hermes's 30 s hook budget and was skipped. Each skip
suppresses the callback for a window, so the pre-tool side of tool dispatch degraded from slow to
blocked — this is the fault that ate the 2026-11-08 morning and stretched that day's cron runs to
6 h (Mnemosyne Health Monitor 12:00:51 → 18:00:42) and 1 h 13 m (FreshRSS Midday 12:00:50 →
14:10:43). Fixed 2026-11-08 13:35 AWST by raising plugins.hook_callback_timeout 30 → 120.
Evidence (live reads, 2026-11-08 23:5x – 2026-11-09 00:1x AWST)
grep -c "callback _pre_tool_call timed out" ~/.hermes/logs/errors.log -> 58
# 2026-11-07: 12 lines 2026-11-08: 46 lines
# first 2026-11-07 14:14:03,075 last 2026-11-08 08:21:03,863
grep -c "skipped after previous timeout" ~/.hermes/logs/*.log -> 89
The exact pair of lines (~/.hermes/logs/errors.log):
2026-11-08 08:19:40,942 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call skipped after previous timeout or while still running
2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
The register (correction to a plausible-but-wrong diagnosis). conduit_push/__init__.py:119 is the
only pre_tool_call registration on the host, and the plugin's caches are already in place —
session_titles.py:21 CACHE_SECONDS = 30.0. Timed in-process by the 2026-11-08 session:
3.1 s cold, 0.00 s warm. The "the hook re-opens a 1.87 GB state.db per call, ~9 s each" theory was
therefore wrong — warm calls were free. The timeouts track load spikes on the 2-core CT460
(load average 5.89 at 13:27:50; state.db carrying a 40 MB WAL checkpointing on the same spindle),
not per-call database opens.
Fix applied (2026-11-08 13:35 AWST)
~/.hermes/config.yaml
plugins:
hook_callback_timeout: 120 # was the built-in 30 s default
- Verified live:
plugins.hook_callback_timeout = 120. - Verified effective: no
_pre_tool_calltimeout has been logged since 08:21:03 — re-checked at the end of the fix window and again at audit time (2026-11-09 00:0x). The 58th line is still the last. - Deliberately NOT done: rewriting the plugin's caching layer. The caches work; the budget was the
binding constraint. Next lever is
state.dbcheckpoint/retention discipline, not more caching.
Consequence — the same signature landed in the cron ledger
The 2026-11-08 03:00 wiki-stocktake-maintenance slot (1a8e90ea2271, 03:01:11 → 04:36:08) closed
failed on a new terminal signature for that job:
RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model
produced no follow-up text. Send `continue` to let it summarize.
Working hypothesis, not proven: a pre_tool_call block holding a tool result open is exactly what this
error describes, and it sits inside the same 14 h timeout window. It is consistent, not established —
the error string does not name the hook.
Impact
| item | effect |
|---|---|
Mnemosyne Health Monitor ea79a2107e36 |
00:01:20 → 08:29:05 (8 h 27 m) and 12:00:51 → 18:00:42 (5 h 59 m 51 s) — its two slowest slots on record, both delivered |
wiki-stocktake-maintenance 1a8e90ea2271 |
03:01:11 → 04:36:08 failed; failure_streak 3; a third unalerted detected incident row |
FreshRSS Midday Briefing ecc1fc13655a |
12:00:50 → 14:10:43 (2 h 09 m 53 s) |
Tech & AI Newsletter Digest 85edf6772bca |
an admission-read database is locked incident (85edf6_0877ca1e0eac) at 07:23:02, resolved at 08:06:54 when the run completed |
| every interactive tool call | slow, then suppressed — the user-visible symptom that opened the investigation |
Verification evidence
~/.hermes/logs/errors.log (58 timeout lines; last 2026-11-08 08:21:03)
~/.hermes/config.yaml (plugins.hook_callback_timeout: 120)
~/.hermes/plugins/conduit_push/__init__.py:119,257 (the only pre_tool_call registration)
~/.hermes/plugins/conduit_push/session_titles.py:21 (CACHE_SECONDS = 30.0)
~/.hermes/cron/executions.db, cron_incidents, jobs.json (run windows, streak, incident rows)
Open items
state.dbis 1.89 GB with 2,901 sessions / 197,682 messages and a live 40 MB WAL — it is the load source behind the spikes. Checkpoint and retention policy are still undecided (incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts).- Confirm whether the stocktake's
No reply: … tool result was still pendingfailure is really the same fault; if it recurs while the hook budget is 120 s, this hypothesis is dead. - Decide whether a skipped
pre_tool_callshould fail loudly rather than silently degrade tool dispatch.
Follow-up (2026-11-10 nightly audit)
- The fix held; the cascade did not recur.
errors.log(1,035 retained lines) contains nopre_tool_calltimeout entry at all. Its only two hook lines are benign skips —on_session_end2026-11-09 22:58:35 andpre_llm_call2026-11-10 00:02:54 ("skipped after previous timeout or while still running"). - No 11-09 run showed the cascade signature. Outside the two long-runners documented separately (stocktake 9 h 10 m, Job Radar Stage 1 9 h 09 m — both stalled in tool iteration, not in a hook), the longest 11-09 execution was 74 min, then 40 min and 11 min.
- 2026-11-09 was the first
failed-free cron day since 2026-11-05 (11-06: 1 failed, 11-07: 3, 11-08: 1, 11-09: 0 of 174), and unclosed incident rows went 3 → 0. - Residual: the two long-runners above are the same 9-hour-turn environment this bug leaves behind, and one (
wiki-stocktake-maintenance) is pinned tofree-stack— a combo recorded broken on 2026-08-14. The job-iteration budget stays in force.
Recurrence (2026-11-14 nightly audit) — the raised budget did not hold
Reopened. The pre_tool_call hook overran its raised 120 s budget again at 2026-11-14 00:05:40,
on the wiki-audit job's own tool calls. The 30 -> 120 s fix of 2026-11-08 is therefore proven
insufficient at current load, and this page is no longer resolved.
| time (AWST) | line |
|---|---|
| 2026-11-14 00:05:40,043 | Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping |
| 2026-11-14 00:05:43 -> 00:06:38 | 10 x skipped after previous timeout or while still running, ~6 s apart |
- Source:
~/.hermes/logs/errors.log(single-source count;agent.logcarries the same lines). By day: 2026-11-12 = 0 timeouts / 0 skips; 2026-11-13 = 0 / 0; 2026-11-14 = 1 / 10. It is the firstpre_tool_calltimeout in the retained log since the 2026-11-08 cascade — the 2026-11-08 lines themselves have since rotated out oferrors.log, so this page's original 58-timeout figure still stands on the 2026-11-09 read and was not re-verifiable this window. - Cost this time: this audit's first
skill_viewandterminalcalls were refused outright by the plugin (first-hand evidence), with dispatch recovering on its own by ~00:06:40 — roughly one minute of hard suppression. It is a single-event recurrence, not the 18-hour cascade. No cron run failed: 2026-11-13 settled 173completed/ 0failed. - Load is the variable, and it is now ~2x the level recorded when the fault was diagnosed:
load average **12.08 / 13.02 / 12.33**on 2 vCPU (2026-11-08 read: 5.89).state.dbhas grown 1.89 GB -> 2.04 GB (2,036,908,032 B; 2,977 sessions / 207,577 messages, was 2,901 / 197,682), and the root filesystem is 88 % used. - The earlier open hypothesis is now settled. "If it recurs while the hook budget is 120 s, this
hypothesis is dead" — it recurred, so the load trigger is confirmed and there is no headroom left
in the timeout: at load ~12 the callback needs >120 s. Re-raising the budget would only mask the
fault. The remaining levers are unchanged and now urgent —
state.dbcheckpoint/retention and reducing host load (see incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts).