audit 2026-11-16: memory stall 62h54m (counts unchanged, cron_incidents 3->5 unclosed, consolidator zero-item streak 120); NEW incident — 11-15 plugin load-timeout storm (52 failed loads, 519 dropped registrations incl. conduit_push 474, 11 pre_tool_call + 2 pre_llm_call hook timeouts, 10 refusals) never recorded by the 11-15 audit; hook page Recurrence 2 (load average removed as predictor); fleet 29 jobs, 11-16 ledger 175 rows 173/2, deliveries 11/11
This commit is contained in:
@@ -0,0 +1,117 @@
|
||||
---
|
||||
title: Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
|
||||
type: incident
|
||||
status: unresolved
|
||||
created: 2026-11-17
|
||||
updated: 2026-11-17
|
||||
verified_on: 2026-11-17
|
||||
review_after: 2026-11-24
|
||||
confidence: high
|
||||
tags: [hermes, plugin, conduit, timeout, infrastructure, incident, load]
|
||||
sources: [live-errors-log, live-config-yaml, live-gateway-log, live-cron-jobs-json]
|
||||
---
|
||||
|
||||
# Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
|
||||
|
||||
**Not observed on 2026-11-16 or 2026-11-17 — 0 lines of either signature on both days.** No fix was
|
||||
applied; the storm stopped as abruptly as it started, and the loader budget that produced it is
|
||||
unchanged, so recurrence is possible. Found by the **2026-11-17 nightly audit**, which was filling the
|
||||
11-15 window's gaps; the **2026-11-15 audit recorded none of this**. Every count below is an exact
|
||||
`errors.log` grep, not an estimate.
|
||||
|
||||
## What the log records (2026-11-15)
|
||||
|
||||
- **52** × `Failed to load plugin '<name>': load timed out after 10s (import + register() never returned)`
|
||||
- **519** × `Plugin '<name>' called register_{tool,hook,skill,platform}() after its load timed out; ignored`
|
||||
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping`
|
||||
- **2** × `Hook 'pre_llm_call' callback _pre_llm_call timed out after 120s — skipping` (06:05:33, 08:41:41)
|
||||
- **10** × tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
|
||||
|
||||
Across the log's retained window (`errors.log` starts **2026-11-09 13:18**; `.1` / `.2` rotations hold
|
||||
older lines):
|
||||
|
||||
| Day | Plugin load failures | Registrations dropped | `pre_tool_call` timeouts | `pre_llm_call` timeouts | Tool refusals |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 2026-11-12 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 2026-11-13 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 2026-11-14 | 4 | 0 | 1 | 1 | 4 |
|
||||
| **2026-11-15** | **52** | **519** | **11** | **2** | **10** |
|
||||
| 2026-11-16 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 2026-11-17 | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
### By plugin (2026-11-15)
|
||||
|
||||
| Plugin | Failed loads | Registrations dropped |
|
||||
| --- | --- | --- |
|
||||
| `hermes-hq-push` | 26 | 0 |
|
||||
| `conduit_push` | 9 | **474** |
|
||||
| `homeassistant` | 7 | 34 |
|
||||
| `browser-browser-use` | 3 | 3 |
|
||||
| `basic` | 2 | 2 |
|
||||
| `cadu-secrets-vault` | 1 | 4 |
|
||||
| `telegram-platform` | 1 | 1 |
|
||||
| `nous` | 1 | 0 |
|
||||
| `drain` | 1 | 0 |
|
||||
| `browser-browserbase` | 1 | 1 |
|
||||
| **Total** | **52** | **519** |
|
||||
|
||||
## The dropped registrations are the more serious half
|
||||
|
||||
A failed load is a warning line. The `register_*()` lines that follow are **function loss**: the loader
|
||||
gives each plugin a **10 s** budget for `import + register()`, and when that budget expires the plugin's
|
||||
late registrations are **discarded, not honoured**. `conduit_push` alone lost **474** registrations across
|
||||
**eight** load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47) — i.e. for those
|
||||
processes the Conduit tools, skills and hooks were **not registered at all**, silently. See
|
||||
[[systems/conduit]].
|
||||
|
||||
## Clustering
|
||||
|
||||
These arrive in bursts, not as a trickle. 18 distinct registration-drop groups on 11-15; load failures
|
||||
spread over hours 01, 04–11 and 13:
|
||||
|
||||
- **01:05–01:32** — `hermes-hq-push` (01:05, 01:17, 01:32) and `homeassistant` (01:17) fail to load,
|
||||
`conduit_push` fails at 01:32:39 and drops ~52 registrations. Inside the same 20 minutes
|
||||
`pre_tool_call` times out **5×** (01:07:17, 01:16:11, 01:19:36, 01:22:39, 01:25:43) and **9** tool calls
|
||||
are refused in the same session (`local-989b3642…`).
|
||||
- **04:46–04:47** — the day's largest burst: `browser-browser-use`, `conduit_push` ×2, `hermes-hq-push` ×2
|
||||
and `homeassistant` all fail within ~55 s; **118** registrations discarded.
|
||||
- **06:31–06:48, 07:06, 08:16, 08:46, 10:47** — repeated `conduit_push`/`hermes-hq-push` failures, each
|
||||
dropping ~52–56 registrations.
|
||||
- **12:18–12:49** — a **second** `pre_tool_call` cluster (6 timeouts + 1 refusal) with **no load failures**
|
||||
in hours 11–12 bar one at 11:49 — so the hook half of this page's family can fire without a loader
|
||||
failure alongside it.
|
||||
|
||||
## What this is — and is not
|
||||
|
||||
- **Not a misconfiguration.** `plugins.hook_callback_timeout: 120` is set in `config.yaml` and **no**
|
||||
plugin-load-timeout key exists there — the 10 s loader budget is a **code default**. The failures are a
|
||||
**saturation symptom** on this host, not a wrong setting.
|
||||
- **Same window as the hook cascade.** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
recurred on 11-15 with the 11 `pre_tool_call` timeouts and 10 refusals counted above; the runtime deps
|
||||
guard also blew its **600 s** budget that day (see
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]]). Several of the estate's own timing budgets were
|
||||
exceeded in the same 24 h.
|
||||
- **Adjacency, not proof.** Nothing ties the three to a single cause. What they share is a host under
|
||||
sustained IO/scheduler pressure while `state.db` had grown past **2 GB** with a **40 MB** WAL (`/` at
|
||||
88 %). But load average at the 2026-11-17 audit read **6.19 / 6.64 / 7.48** — *below* the 11-14 reading
|
||||
of 12.08 — with **no** recurrence of this class on 11-16/11-17, so **load average alone does not
|
||||
predict it**.
|
||||
- **Not host restarts.** `gateway.log` has **0** startup/shutdown markers on 11-15, and `uptime` at the
|
||||
audit was 6 d 2 h (host last up **2026-11-10 ~22:04**) — so the ~18 load events are **in-process plugin
|
||||
(re)loads**. Plugin registration output in `agent.log` appears at agent-process start (e.g. the
|
||||
2026-11-16 00:00:00 block), which is why a job starting during saturation can run with plugins missing.
|
||||
|
||||
## Open question
|
||||
|
||||
Why did `import + register()` exceed 10 s for whole plugins, and does the loader fail **closed or open**
|
||||
for the caller? Today it fails **open** — the job continues, silently without those tools. A check that
|
||||
asserts **plugin registration** rather than only job exit status would surface this class; the same idea
|
||||
applies to the hook callback budget. **Proposed, not applied** (it changes Hermes runtime behaviour, so
|
||||
it is Anthony's call).
|
||||
|
||||
## See also
|
||||
|
||||
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — the companion hook-callback cascade
|
||||
- [[systems/conduit]] — what the 474 dropped `conduit_push` registrations cost
|
||||
- [[systems/hermes-agent]] — plugin loading, config and timeouts
|
||||
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — same-window runtime pressure
|
||||
Reference in New Issue
Block a user