--- title: Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15) type: incident status: unresolved created: 2026-11-17 updated: 2026-11-17 verified_on: 2026-11-17 review_after: 2026-11-24 confidence: high tags: [hermes, plugin, conduit, timeout, infrastructure, incident, load] sources: [live-errors-log, live-config-yaml, live-gateway-log, live-cron-jobs-json] --- # Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15) **Not observed on 2026-11-16 or 2026-11-17 — 0 lines of either signature on both days.** No fix was applied; the storm stopped as abruptly as it started, and the loader budget that produced it is unchanged, so recurrence is possible. Found by the **2026-11-17 nightly audit**, which was filling the 11-15 window's gaps; the **2026-11-15 audit recorded none of this**. Every count below is an exact `errors.log` grep, not an estimate. ## What the log records (2026-11-15) - **52** × `Failed to load plugin '': load timed out after 10s (import + register() never returned)` - **519** × `Plugin '' called register_{tool,hook,skill,platform}() after its load timed out; ignored` - **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` - **2** × `Hook 'pre_llm_call' callback _pre_llm_call timed out after 120s — skipping` (06:05:33, 08:41:41) - **10** × tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}` Across the log's retained window (`errors.log` starts **2026-11-09 13:18**; `.1` / `.2` rotations hold older lines): | Day | Plugin load failures | Registrations dropped | `pre_tool_call` timeouts | `pre_llm_call` timeouts | Tool refusals | | --- | --- | --- | --- | --- | --- | | 2026-11-12 | 0 | 0 | 0 | 0 | 0 | | 2026-11-13 | 0 | 0 | 0 | 0 | 0 | | 2026-11-14 | 4 | 0 | 1 | 1 | 4 | | **2026-11-15** | **52** | **519** | **11** | **2** | **10** | | 2026-11-16 | 0 | 0 | 0 | 0 | 0 | | 2026-11-17 | 0 | 0 | 0 | 0 | 0 | ### By plugin (2026-11-15) | Plugin | Failed loads | Registrations dropped | | --- | --- | --- | | `hermes-hq-push` | 26 | 0 | | `conduit_push` | 9 | **474** | | `homeassistant` | 7 | 34 | | `browser-browser-use` | 3 | 3 | | `basic` | 2 | 2 | | `cadu-secrets-vault` | 1 | 4 | | `telegram-platform` | 1 | 1 | | `nous` | 1 | 0 | | `drain` | 1 | 0 | | `browser-browserbase` | 1 | 1 | | **Total** | **52** | **519** | ## The dropped registrations are the more serious half A failed load is a warning line. The `register_*()` lines that follow are **function loss**: the loader gives each plugin a **10 s** budget for `import + register()`, and when that budget expires the plugin's late registrations are **discarded, not honoured**. `conduit_push` alone lost **474** registrations across **eight** load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47) — i.e. for those processes the Conduit tools, skills and hooks were **not registered at all**, silently. See [[systems/conduit]]. ## Clustering These arrive in bursts, not as a trickle. 18 distinct registration-drop groups on 11-15; load failures spread over hours 01, 04–11 and 13: - **01:05–01:32** — `hermes-hq-push` (01:05, 01:17, 01:32) and `homeassistant` (01:17) fail to load, `conduit_push` fails at 01:32:39 and drops ~52 registrations. Inside the same 20 minutes `pre_tool_call` times out **5×** (01:07:17, 01:16:11, 01:19:36, 01:22:39, 01:25:43) and **9** tool calls are refused in the same session (`local-989b3642…`). - **04:46–04:47** — the day's largest burst: `browser-browser-use`, `conduit_push` ×2, `hermes-hq-push` ×2 and `homeassistant` all fail within ~55 s; **118** registrations discarded. - **06:31–06:48, 07:06, 08:16, 08:46, 10:47** — repeated `conduit_push`/`hermes-hq-push` failures, each dropping ~52–56 registrations. - **12:18–12:49** — a **second** `pre_tool_call` cluster (6 timeouts + 1 refusal) with **no load failures** in hours 11–12 bar one at 11:49 — so the hook half of this page's family can fire without a loader failure alongside it. ## What this is — and is not - **Not a misconfiguration.** `plugins.hook_callback_timeout: 120` is set in `config.yaml` and **no** plugin-load-timeout key exists there — the 10 s loader budget is a **code default**. The failures are a **saturation symptom** on this host, not a wrong setting. - **Same window as the hook cascade.** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] recurred on 11-15 with the 11 `pre_tool_call` timeouts and 10 refusals counted above; the runtime deps guard also blew its **600 s** budget that day (see [[incidents/2026-11-14-mnemosyne-capture-stall]]). Several of the estate's own timing budgets were exceeded in the same 24 h. - **Adjacency, not proof.** Nothing ties the three to a single cause. What they share is a host under sustained IO/scheduler pressure while `state.db` had grown past **2 GB** with a **40 MB** WAL (`/` at 88 %). But load average at the 2026-11-17 audit read **6.19 / 6.64 / 7.48** — *below* the 11-14 reading of 12.08 — with **no** recurrence of this class on 11-16/11-17, so **load average alone does not predict it**. - **Not host restarts.** `gateway.log` has **0** startup/shutdown markers on 11-15, and `uptime` at the audit was 6 d 2 h (host last up **2026-11-10 ~22:04**) — so the ~18 load events are **in-process plugin (re)loads**. Plugin registration output in `agent.log` appears at agent-process start (e.g. the 2026-11-16 00:00:00 block), which is why a job starting during saturation can run with plugins missing. ## Open question Why did `import + register()` exceed 10 s for whole plugins, and does the loader fail **closed or open** for the caller? Today it fails **open** — the job continues, silently without those tools. A check that asserts **plugin registration** rather than only job exit status would surface this class; the same idea applies to the hook callback budget. **Proposed, not applied** (it changes Hermes runtime behaviour, so it is Anthony's call). ## See also - [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — the companion hook-callback cascade - [[systems/conduit]] — what the 474 dropped `conduit_push` registrations cost - [[systems/hermes-agent]] — plugin loading, config and timeouts - [[incidents/2026-11-14-mnemosyne-capture-stall]] — same-window runtime pressure