118 lines
6.3 KiB
Markdown
118 lines
6.3 KiB
Markdown
---
|
||
title: Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
|
||
type: incident
|
||
status: unresolved
|
||
created: 2026-11-17
|
||
updated: 2026-11-17
|
||
verified_on: 2026-11-17
|
||
review_after: 2026-11-24
|
||
confidence: high
|
||
tags: [hermes, plugin, conduit, timeout, infrastructure, incident, load]
|
||
sources: [live-errors-log, live-config-yaml, live-gateway-log, live-cron-jobs-json]
|
||
---
|
||
|
||
# Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
|
||
|
||
**Not observed on 2026-11-16 or 2026-11-17 — 0 lines of either signature on both days.** No fix was
|
||
applied; the storm stopped as abruptly as it started, and the loader budget that produced it is
|
||
unchanged, so recurrence is possible. Found by the **2026-11-17 nightly audit**, which was filling the
|
||
11-15 window's gaps; the **2026-11-15 audit recorded none of this**. Every count below is an exact
|
||
`errors.log` grep, not an estimate.
|
||
|
||
## What the log records (2026-11-15)
|
||
|
||
- **52** × `Failed to load plugin '<name>': load timed out after 10s (import + register() never returned)`
|
||
- **519** × `Plugin '<name>' called register_{tool,hook,skill,platform}() after its load timed out; ignored`
|
||
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping`
|
||
- **2** × `Hook 'pre_llm_call' callback _pre_llm_call timed out after 120s — skipping` (06:05:33, 08:41:41)
|
||
- **10** × tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
|
||
|
||
Across the log's retained window (`errors.log` starts **2026-11-09 13:18**; `.1` / `.2` rotations hold
|
||
older lines):
|
||
|
||
| Day | Plugin load failures | Registrations dropped | `pre_tool_call` timeouts | `pre_llm_call` timeouts | Tool refusals |
|
||
| --- | --- | --- | --- | --- | --- |
|
||
| 2026-11-12 | 0 | 0 | 0 | 0 | 0 |
|
||
| 2026-11-13 | 0 | 0 | 0 | 0 | 0 |
|
||
| 2026-11-14 | 4 | 0 | 1 | 1 | 4 |
|
||
| **2026-11-15** | **52** | **519** | **11** | **2** | **10** |
|
||
| 2026-11-16 | 0 | 0 | 0 | 0 | 0 |
|
||
| 2026-11-17 | 0 | 0 | 0 | 0 | 0 |
|
||
|
||
### By plugin (2026-11-15)
|
||
|
||
| Plugin | Failed loads | Registrations dropped |
|
||
| --- | --- | --- |
|
||
| `hermes-hq-push` | 26 | 0 |
|
||
| `conduit_push` | 9 | **474** |
|
||
| `homeassistant` | 7 | 34 |
|
||
| `browser-browser-use` | 3 | 3 |
|
||
| `basic` | 2 | 2 |
|
||
| `cadu-secrets-vault` | 1 | 4 |
|
||
| `telegram-platform` | 1 | 1 |
|
||
| `nous` | 1 | 0 |
|
||
| `drain` | 1 | 0 |
|
||
| `browser-browserbase` | 1 | 1 |
|
||
| **Total** | **52** | **519** |
|
||
|
||
## The dropped registrations are the more serious half
|
||
|
||
A failed load is a warning line. The `register_*()` lines that follow are **function loss**: the loader
|
||
gives each plugin a **10 s** budget for `import + register()`, and when that budget expires the plugin's
|
||
late registrations are **discarded, not honoured**. `conduit_push` alone lost **474** registrations across
|
||
**eight** load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47) — i.e. for those
|
||
processes the Conduit tools, skills and hooks were **not registered at all**, silently. See
|
||
[[systems/conduit]].
|
||
|
||
## Clustering
|
||
|
||
These arrive in bursts, not as a trickle. 18 distinct registration-drop groups on 11-15; load failures
|
||
spread over hours 01, 04–11 and 13:
|
||
|
||
- **01:05–01:32** — `hermes-hq-push` (01:05, 01:17, 01:32) and `homeassistant` (01:17) fail to load,
|
||
`conduit_push` fails at 01:32:39 and drops ~52 registrations. Inside the same 20 minutes
|
||
`pre_tool_call` times out **5×** (01:07:17, 01:16:11, 01:19:36, 01:22:39, 01:25:43) and **9** tool calls
|
||
are refused in the same session (`local-989b3642…`).
|
||
- **04:46–04:47** — the day's largest burst: `browser-browser-use`, `conduit_push` ×2, `hermes-hq-push` ×2
|
||
and `homeassistant` all fail within ~55 s; **118** registrations discarded.
|
||
- **06:31–06:48, 07:06, 08:16, 08:46, 10:47** — repeated `conduit_push`/`hermes-hq-push` failures, each
|
||
dropping ~52–56 registrations.
|
||
- **12:18–12:49** — a **second** `pre_tool_call` cluster (6 timeouts + 1 refusal) with **no load failures**
|
||
in hours 11–12 bar one at 11:49 — so the hook half of this page's family can fire without a loader
|
||
failure alongside it.
|
||
|
||
## What this is — and is not
|
||
|
||
- **Not a misconfiguration.** `plugins.hook_callback_timeout: 120` is set in `config.yaml` and **no**
|
||
plugin-load-timeout key exists there — the 10 s loader budget is a **code default**. The failures are a
|
||
**saturation symptom** on this host, not a wrong setting.
|
||
- **Same window as the hook cascade.** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||
recurred on 11-15 with the 11 `pre_tool_call` timeouts and 10 refusals counted above; the runtime deps
|
||
guard also blew its **600 s** budget that day (see
|
||
[[incidents/2026-11-14-mnemosyne-capture-stall]]). Several of the estate's own timing budgets were
|
||
exceeded in the same 24 h.
|
||
- **Adjacency, not proof.** Nothing ties the three to a single cause. What they share is a host under
|
||
sustained IO/scheduler pressure while `state.db` had grown past **2 GB** with a **40 MB** WAL (`/` at
|
||
88 %). But load average at the 2026-11-17 audit read **6.19 / 6.64 / 7.48** — *below* the 11-14 reading
|
||
of 12.08 — with **no** recurrence of this class on 11-16/11-17, so **load average alone does not
|
||
predict it**.
|
||
- **Not host restarts.** `gateway.log` has **0** startup/shutdown markers on 11-15, and `uptime` at the
|
||
audit was 6 d 2 h (host last up **2026-11-10 ~22:04**) — so the ~18 load events are **in-process plugin
|
||
(re)loads**. Plugin registration output in `agent.log` appears at agent-process start (e.g. the
|
||
2026-11-16 00:00:00 block), which is why a job starting during saturation can run with plugins missing.
|
||
|
||
## Open question
|
||
|
||
Why did `import + register()` exceed 10 s for whole plugins, and does the loader fail **closed or open**
|
||
for the caller? Today it fails **open** — the job continues, silently without those tools. A check that
|
||
asserts **plugin registration** rather than only job exit status would surface this class; the same idea
|
||
applies to the hook callback budget. **Proposed, not applied** (it changes Hermes runtime behaviour, so
|
||
it is Anthony's call).
|
||
|
||
## See also
|
||
|
||
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — the companion hook-callback cascade
|
||
- [[systems/conduit]] — what the 474 dropped `conduit_push` registrations cost
|
||
- [[systems/hermes-agent]] — plugin loading, config and timeouts
|
||
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — same-window runtime pressure
|