audit(2026-11-13): third zero-failed day and a fully clean fleet; the Uber Eats fix clears the last non-clean cron row — and the pre_tool_call hook timeout recurred at load 12.08 (incident reopened resolved -> active, state.db 2.04 GB)

This commit is contained in:
Tony0410
2026-11-14 00:27:05 +08:00
parent a0ec3a1ec5
commit b0e4a54f8a
5 changed files with 133 additions and 15 deletions
@@ -1,11 +1,11 @@
---
title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)
type: incident
status: resolved
status: active
created: 2026-11-09
updated: 2026-11-09
verified_on: 2026-11-09
review_after: 2026-11-16
updated: 2026-11-14
verified_on: 2026-11-14
review_after: 2026-11-17
confidence: high
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
@@ -98,6 +98,36 @@ the error string does not name the hook.
- **2026-11-09 was the first `failed`-free cron day since 2026-11-05** (11-06: 1 failed, 11-07: 3, 11-08: 1, 11-09: **0 of 174**), and unclosed incident rows went 3 → 0.
- **Residual:** the two long-runners above are the same 9-hour-turn environment this bug leaves behind, and one (`wiki-stocktake-maintenance`) is pinned to `free-stack` — a combo recorded broken on 2026-08-14. The job-iteration budget stays in force.
## Recurrence (2026-11-14 nightly audit) — the raised budget did not hold
**Reopened.** The `pre_tool_call` hook overran its **raised 120 s** budget again at **2026-11-14 00:05:40**,
on the wiki-audit job's own tool calls. The 30 -> 120 s fix of 2026-11-08 is therefore **proven
insufficient at current load**, and this page is no longer `resolved`.
| time (AWST) | line |
|---|---|
| 2026-11-14 00:05:40,043 | `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` |
| 2026-11-14 00:05:43 -> 00:06:38 | **10** x `skipped after previous timeout or while still running`, ~6 s apart |
- **Source:** `~/.hermes/logs/errors.log` (single-source count; `agent.log` carries the same lines).
By day: **2026-11-12 = 0 timeouts / 0 skips; 2026-11-13 = 0 / 0; 2026-11-14 = 1 / 10.** It is the
**first `pre_tool_call` timeout in the retained log since the 2026-11-08 cascade** — the 2026-11-08
lines themselves have since **rotated out** of `errors.log`, so this page's original 58-timeout figure
still stands on the 2026-11-09 read and was **not re-verifiable** this window.
- **Cost this time:** this audit's first `skill_view` and `terminal` calls were **refused outright** by the
plugin (first-hand evidence), with dispatch recovering on its own by ~00:06:40 — roughly **one minute**
of hard suppression. It is a **single-event recurrence, not the 18-hour cascade**. **No cron run failed:**
2026-11-13 settled **173 `completed` / 0 `failed`**.
- **Load is the variable, and it is now ~2x the level recorded when the fault was diagnosed:**
`load average **12.08 / 13.02 / 12.33**` on **2 vCPU** (2026-11-08 read: **5.89**). `state.db` has grown
**1.89 GB -> 2.04 GB** (**2,036,908,032 B**; **2,977 sessions / 207,577 messages**, was 2,901 / 197,682),
and the root filesystem is **88 % used**.
- **The earlier open hypothesis is now settled.** *"If it recurs while the hook budget is 120 s, this
hypothesis is dead"* — it recurred, so the load trigger is **confirmed** and there is **no headroom left
in the timeout**: at load ~12 the callback needs >120 s. **Re-raising the budget would only mask the
fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and
reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
## See also
- [[systems/conduit]]