Files
agent-estate-wiki/incidents/2026-11-15-plugin-load-timeout-storm.md
T

118 lines
6.3 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
type: incident
status: unresolved
created: 2026-11-17
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-24
confidence: high
tags: [hermes, plugin, conduit, timeout, infrastructure, incident, load]
sources: [live-errors-log, live-config-yaml, live-gateway-log, live-cron-jobs-json]
---
# Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
**Not observed on 2026-11-16 or 2026-11-17 — 0 lines of either signature on both days.** No fix was
applied; the storm stopped as abruptly as it started, and the loader budget that produced it is
unchanged, so recurrence is possible. Found by the **2026-11-17 nightly audit**, which was filling the
11-15 window's gaps; the **2026-11-15 audit recorded none of this**. Every count below is an exact
`errors.log` grep, not an estimate.
## What the log records (2026-11-15)
- **52** × `Failed to load plugin '<name>': load timed out after 10s (import + register() never returned)`
- **519** × `Plugin '<name>' called register_{tool,hook,skill,platform}() after its load timed out; ignored`
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping`
- **2** × `Hook 'pre_llm_call' callback _pre_llm_call timed out after 120s — skipping` (06:05:33, 08:41:41)
- **10** × tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
Across the log's retained window (`errors.log` starts **2026-11-09 13:18**; `.1` / `.2` rotations hold
older lines):
| Day | Plugin load failures | Registrations dropped | `pre_tool_call` timeouts | `pre_llm_call` timeouts | Tool refusals |
| --- | --- | --- | --- | --- | --- |
| 2026-11-12 | 0 | 0 | 0 | 0 | 0 |
| 2026-11-13 | 0 | 0 | 0 | 0 | 0 |
| 2026-11-14 | 4 | 0 | 1 | 1 | 4 |
| **2026-11-15** | **52** | **519** | **11** | **2** | **10** |
| 2026-11-16 | 0 | 0 | 0 | 0 | 0 |
| 2026-11-17 | 0 | 0 | 0 | 0 | 0 |
### By plugin (2026-11-15)
| Plugin | Failed loads | Registrations dropped |
| --- | --- | --- |
| `hermes-hq-push` | 26 | 0 |
| `conduit_push` | 9 | **474** |
| `homeassistant` | 7 | 34 |
| `browser-browser-use` | 3 | 3 |
| `basic` | 2 | 2 |
| `cadu-secrets-vault` | 1 | 4 |
| `telegram-platform` | 1 | 1 |
| `nous` | 1 | 0 |
| `drain` | 1 | 0 |
| `browser-browserbase` | 1 | 1 |
| **Total** | **52** | **519** |
## The dropped registrations are the more serious half
A failed load is a warning line. The `register_*()` lines that follow are **function loss**: the loader
gives each plugin a **10 s** budget for `import + register()`, and when that budget expires the plugin's
late registrations are **discarded, not honoured**. `conduit_push` alone lost **474** registrations across
**eight** load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47) — i.e. for those
processes the Conduit tools, skills and hooks were **not registered at all**, silently. See
[[systems/conduit]].
## Clustering
These arrive in bursts, not as a trickle. 18 distinct registration-drop groups on 11-15; load failures
spread over hours 01, 04–11 and 13:
- **01:05–01:32** — `hermes-hq-push` (01:05, 01:17, 01:32) and `homeassistant` (01:17) fail to load,
`conduit_push` fails at 01:32:39 and drops ~52 registrations. Inside the same 20 minutes
`pre_tool_call` times out **5×** (01:07:17, 01:16:11, 01:19:36, 01:22:39, 01:25:43) and **9** tool calls
are refused in the same session (`local-989b3642…`).
- **04:46–04:47** — the day's largest burst: `browser-browser-use`, `conduit_push` ×2, `hermes-hq-push` ×2
and `homeassistant` all fail within ~55 s; **118** registrations discarded.
- **06:31–06:48, 07:06, 08:16, 08:46, 10:47** — repeated `conduit_push`/`hermes-hq-push` failures, each
dropping ~52–56 registrations.
- **12:18–12:49** — a **second** `pre_tool_call` cluster (6 timeouts + 1 refusal) with **no load failures**
in hours 11–12 bar one at 11:49 — so the hook half of this page's family can fire without a loader
failure alongside it.
## What this is — and is not
- **Not a misconfiguration.** `plugins.hook_callback_timeout: 120` is set in `config.yaml` and **no**
plugin-load-timeout key exists there — the 10 s loader budget is a **code default**. The failures are a
**saturation symptom** on this host, not a wrong setting.
- **Same window as the hook cascade.** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
recurred on 11-15 with the 11 `pre_tool_call` timeouts and 10 refusals counted above; the runtime deps
guard also blew its **600 s** budget that day (see
[[incidents/2026-11-14-mnemosyne-capture-stall]]). Several of the estate's own timing budgets were
exceeded in the same 24 h.
- **Adjacency, not proof.** Nothing ties the three to a single cause. What they share is a host under
sustained IO/scheduler pressure while `state.db` had grown past **2 GB** with a **40 MB** WAL (`/` at
88 %). But load average at the 2026-11-17 audit read **6.19 / 6.64 / 7.48** — *below* the 11-14 reading
of 12.08 — with **no** recurrence of this class on 11-16/11-17, so **load average alone does not
predict it**.
- **Not host restarts.** `gateway.log` has **0** startup/shutdown markers on 11-15, and `uptime` at the
audit was 6 d 2 h (host last up **2026-11-10 ~22:04**) — so the ~18 load events are **in-process plugin
(re)loads**. Plugin registration output in `agent.log` appears at agent-process start (e.g. the
2026-11-16 00:00:00 block), which is why a job starting during saturation can run with plugins missing.
## Open question
Why did `import + register()` exceed 10 s for whole plugins, and does the loader fail **closed or open**
for the caller? Today it fails **open** — the job continues, silently without those tools. A check that
asserts **plugin registration** rather than only job exit status would surface this class; the same idea
applies to the hook callback budget. **Proposed, not applied** (it changes Hermes runtime behaviour, so
it is Anthony's call).
## See also
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — the companion hook-callback cascade
- [[systems/conduit]] — what the 474 dropped `conduit_push` registrations cost
- [[systems/hermes-agent]] — plugin loading, config and timeouts
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — same-window runtime pressure