6.3 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15) | incident | unresolved | 2026-11-17 | 2026-11-17 | 2026-11-17 | 2026-11-24 | high |
|
|
Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
Not observed on 2026-11-16 or 2026-11-17 — 0 lines of either signature on both days. No fix was
applied; the storm stopped as abruptly as it started, and the loader budget that produced it is
unchanged, so recurrence is possible. Found by the 2026-11-17 nightly audit, which was filling the
11-15 window's gaps; the 2026-11-15 audit recorded none of this. Every count below is an exact
errors.log grep, not an estimate.
What the log records (2026-11-15)
- 52 ×
Failed to load plugin '<name>': load timed out after 10s (import + register() never returned) - 519 ×
Plugin '<name>' called register_{tool,hook,skill,platform}() after its load timed out; ignored - 11 ×
Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping - 2 ×
Hook 'pre_llm_call' callback _pre_llm_call timed out after 120s — skipping(06:05:33, 08:41:41) - 10 × tool calls returned
{"error": "pre_tool_call plugin callback timed out or is still running"}
Across the log's retained window (errors.log starts 2026-11-09 13:18; .1 / .2 rotations hold
older lines):
| Day | Plugin load failures | Registrations dropped | pre_tool_call timeouts |
pre_llm_call timeouts |
Tool refusals |
|---|---|---|---|---|---|
| 2026-11-12 | 0 | 0 | 0 | 0 | 0 |
| 2026-11-13 | 0 | 0 | 0 | 0 | 0 |
| 2026-11-14 | 4 | 0 | 1 | 1 | 4 |
| 2026-11-15 | 52 | 519 | 11 | 2 | 10 |
| 2026-11-16 | 0 | 0 | 0 | 0 | 0 |
| 2026-11-17 | 0 | 0 | 0 | 0 | 0 |
By plugin (2026-11-15)
| Plugin | Failed loads | Registrations dropped |
|---|---|---|
hermes-hq-push |
26 | 0 |
conduit_push |
9 | 474 |
homeassistant |
7 | 34 |
browser-browser-use |
3 | 3 |
basic |
2 | 2 |
cadu-secrets-vault |
1 | 4 |
telegram-platform |
1 | 1 |
nous |
1 | 0 |
drain |
1 | 0 |
browser-browserbase |
1 | 1 |
| Total | 52 | 519 |
The dropped registrations are the more serious half
A failed load is a warning line. The register_*() lines that follow are function loss: the loader
gives each plugin a 10 s budget for import + register(), and when that budget expires the plugin's
late registrations are discarded, not honoured. conduit_push alone lost 474 registrations across
eight load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47) — i.e. for those
processes the Conduit tools, skills and hooks were not registered at all, silently. See
systems/conduit.
Clustering
These arrive in bursts, not as a trickle. 18 distinct registration-drop groups on 11-15; load failures spread over hours 01, 04–11 and 13:
- 01:05–01:32 —
hermes-hq-push(01:05, 01:17, 01:32) andhomeassistant(01:17) fail to load,conduit_pushfails at 01:32:39 and drops ~52 registrations. Inside the same 20 minutespre_tool_calltimes out 5× (01:07:17, 01:16:11, 01:19:36, 01:22:39, 01:25:43) and 9 tool calls are refused in the same session (local-989b3642…). - 04:46–04:47 — the day's largest burst:
browser-browser-use,conduit_push×2,hermes-hq-push×2 andhomeassistantall fail within ~55 s; 118 registrations discarded. - 06:31–06:48, 07:06, 08:16, 08:46, 10:47 — repeated
conduit_push/hermes-hq-pushfailures, each dropping ~52–56 registrations. - 12:18–12:49 — a second
pre_tool_callcluster (6 timeouts + 1 refusal) with no load failures in hours 11–12 bar one at 11:49 — so the hook half of this page's family can fire without a loader failure alongside it.
What this is — and is not
- Not a misconfiguration.
plugins.hook_callback_timeout: 120is set inconfig.yamland no plugin-load-timeout key exists there — the 10 s loader budget is a code default. The failures are a saturation symptom on this host, not a wrong setting. - Same window as the hook cascade. incidents/2026-11-08-pre-tool-call-hook-timeout-cascade
recurred on 11-15 with the 11
pre_tool_calltimeouts and 10 refusals counted above; the runtime deps guard also blew its 600 s budget that day (see incidents/2026-11-14-mnemosyne-capture-stall). Several of the estate's own timing budgets were exceeded in the same 24 h. - Adjacency, not proof. Nothing ties the three to a single cause. What they share is a host under
sustained IO/scheduler pressure while
state.dbhad grown past 2 GB with a 40 MB WAL (/at 88 %). But load average at the 2026-11-17 audit read 6.19 / 6.64 / 7.48 — below the 11-14 reading of 12.08 — with no recurrence of this class on 11-16/11-17, so load average alone does not predict it. - Not host restarts.
gateway.loghas 0 startup/shutdown markers on 11-15, anduptimeat the audit was 6 d 2 h (host last up 2026-11-10 ~22:04) — so the ~18 load events are in-process plugin (re)loads. Plugin registration output inagent.logappears at agent-process start (e.g. the 2026-11-16 00:00:00 block), which is why a job starting during saturation can run with plugins missing.
Open question
Why did import + register() exceed 10 s for whole plugins, and does the loader fail closed or open
for the caller? Today it fails open — the job continues, silently without those tools. A check that
asserts plugin registration rather than only job exit status would surface this class; the same idea
applies to the hook callback budget. Proposed, not applied (it changes Hermes runtime behaviour, so
it is Anthony's call).
See also
- incidents/2026-11-08-pre-tool-call-hook-timeout-cascade — the companion hook-callback cascade
- systems/conduit — what the 474 dropped
conduit_pushregistrations cost - systems/hermes-agent — plugin loading, config and timeouts
- incidents/2026-11-14-mnemosyne-capture-stall — same-window runtime pressure