audit 2026-11-16: memory stall 62h54m (counts unchanged, cron_incidents 3->5 unclosed, consolidator zero-item streak 120); NEW incident — 11-15 plugin load-timeout storm (52 failed loads, 519 dropped registrations incl. conduit_push 474, 11 pre_tool_call + 2 pre_llm_call hook timeouts, 10 refusals) never recorded by the 11-15 audit; hook page Recurrence 2 (load average removed as predictor); fleet 29 jobs, 11-16 ledger 175 rows 173/2, deliveries 11/11
This commit is contained in:
@@ -3,9 +3,9 @@ title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-09
|
||||
updated: 2026-11-14
|
||||
verified_on: 2026-11-14
|
||||
review_after: 2026-11-17
|
||||
updated: 2026-11-17
|
||||
verified_on: 2026-11-17
|
||||
review_after: 2026-11-20
|
||||
confidence: high
|
||||
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
|
||||
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
|
||||
@@ -128,6 +128,33 @@ insufficient at current load**, and this page is no longer `resolved`.
|
||||
fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and
|
||||
reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
|
||||
|
||||
## Recurrence 2 (2026-11-15) — eleven timeouts in two clusters, 10 refusals
|
||||
|
||||
Recorded by the **2026-11-17 nightly audit** while backing up the 11-15 window: the **2026-11-15 audit
|
||||
captured none of this**. Exact `errors.log` counts for 11-15:
|
||||
|
||||
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` —
|
||||
**5** in 01:07:17 → 01:25:43 and **6** in 12:18:15 → 12:48:40 (a two-cluster day, unlike 11-14's one).
|
||||
- **10** tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
|
||||
— hour 01: `read_file` ×2 (134.17 s and 0.43 s), `execute_code` ×4, `patch` ×1, `read_file` ×1
|
||||
(121.03 s); hour 12: `execute_code` ×1. Call-level blocks reach **120.63 s**, and the affected sessions
|
||||
are `local-989b3642…` (01:xx) and `local-b6a1490f…` (12:xx).
|
||||
- 11-14, for comparison: **1** timeout and **4** refusals, plus **4** plugin load failures.
|
||||
- **11-16 and 11-17: 0 timeouts, 0 refusals, 0 load failures.** The episode did not carry over.
|
||||
|
||||
The loader half of the same 11-15 storm — **52** failed plugin loads, **519** discarded registrations
|
||||
(`conduit_push` 474 of them) and **2** `pre_llm_call` timeouts — is now on its own page,
|
||||
[[incidents/2026-11-15-plugin-load-timeout-storm]]. Its 12:18–12:49 cluster is the useful detail here:
|
||||
**a hook-timeout cluster can occur with no accompanying loader failure**, so the two share a cause class
|
||||
rather than a mechanism.
|
||||
|
||||
**Effect on the trigger hypothesis recorded above.** Amplitude is not a simple function of load average:
|
||||
the 11-15 episode ran at a load the audit cannot measure retroactively, while the 11-17 read
|
||||
(**6.19 / 6.64 / 7.48**) sits *below* the 11-14 figure of 12.08 with **no** recurrence — and a *larger*
|
||||
`state.db` (2.09 GB, WAL 40 MB, `/` at 88 %). Keep a **load-average threshold out of this page's claim**;
|
||||
keep **`state.db` size, disk headroom and the gateway's own timing budgets** in it.
|
||||
|
||||
|
||||
## See also
|
||||
|
||||
- [[systems/conduit]]
|
||||
|
||||
Reference in New Issue
Block a user