audit 2026-11-16: memory stall 62h54m (counts unchanged, cron_incidents 3->5 unclosed, consolidator zero-item streak 120); NEW incident — 11-15 plugin load-timeout storm (52 failed loads, 519 dropped registrations incl. conduit_push 474, 11 pre_tool_call + 2 pre_llm_call hook timeouts, 10 refusals) never recorded by the 11-15 audit; hook page Recurrence 2 (load average removed as predictor); fleet 29 jobs, 11-16 ledger 175 rows 173/2, deliveries 11/11

This commit is contained in:
hermes
2026-11-17 00:36:27 +08:00
parent 4eee02b9a2
commit e5612c8a8d
9 changed files with 299 additions and 22 deletions
@@ -3,9 +3,9 @@ title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026
type: incident
status: active
created: 2026-11-09
updated: 2026-11-14
verified_on: 2026-11-14
review_after: 2026-11-17
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-20
confidence: high
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
@@ -128,6 +128,33 @@ insufficient at current load**, and this page is no longer `resolved`.
fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and
reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
## Recurrence 2 (2026-11-15) — eleven timeouts in two clusters, 10 refusals
Recorded by the **2026-11-17 nightly audit** while backing up the 11-15 window: the **2026-11-15 audit
captured none of this**. Exact `errors.log` counts for 11-15:
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` —
**5** in 01:07:17 → 01:25:43 and **6** in 12:18:15 → 12:48:40 (a two-cluster day, unlike 11-14's one).
- **10** tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
— hour 01: `read_file` ×2 (134.17 s and 0.43 s), `execute_code` ×4, `patch` ×1, `read_file` ×1
(121.03 s); hour 12: `execute_code` ×1. Call-level blocks reach **120.63 s**, and the affected sessions
are `local-989b3642…` (01:xx) and `local-b6a1490f…` (12:xx).
- 11-14, for comparison: **1** timeout and **4** refusals, plus **4** plugin load failures.
- **11-16 and 11-17: 0 timeouts, 0 refusals, 0 load failures.** The episode did not carry over.
The loader half of the same 11-15 storm — **52** failed plugin loads, **519** discarded registrations
(`conduit_push` 474 of them) and **2** `pre_llm_call` timeouts — is now on its own page,
[[incidents/2026-11-15-plugin-load-timeout-storm]]. Its 12:18–12:49 cluster is the useful detail here:
**a hook-timeout cluster can occur with no accompanying loader failure**, so the two share a cause class
rather than a mechanism.
**Effect on the trigger hypothesis recorded above.** Amplitude is not a simple function of load average:
the 11-15 episode ran at a load the audit cannot measure retroactively, while the 11-17 read
(**6.19 / 6.64 / 7.48**) sits *below* the 11-14 figure of 12.08 with **no** recurrence — and a *larger*
`state.db` (2.09 GB, WAL 40 MB, `/` at 88 %). Keep a **load-average threshold out of this page's claim**;
keep **`state.db` size, disk headroom and the gateway's own timing budgets** in it.
## See also
- [[systems/conduit]]