Files
agent-estate-wiki/incidents/2026-11-15-plugin-load-timeout-storm.md
T

6.3 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15) incident unresolved 2026-11-17 2026-11-17 2026-11-17 2026-11-24 high
hermes
plugin
conduit
timeout
infrastructure
incident
load
live-errors-log
live-config-yaml
live-gateway-log
live-cron-jobs-json

Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)

Not observed on 2026-11-16 or 2026-11-17 — 0 lines of either signature on both days. No fix was applied; the storm stopped as abruptly as it started, and the loader budget that produced it is unchanged, so recurrence is possible. Found by the 2026-11-17 nightly audit, which was filling the 11-15 window's gaps; the 2026-11-15 audit recorded none of this. Every count below is an exact errors.log grep, not an estimate.

What the log records (2026-11-15)

  • 52 × Failed to load plugin '<name>': load timed out after 10s (import + register() never returned)
  • 519 × Plugin '<name>' called register_{tool,hook,skill,platform}() after its load timed out; ignored
  • 11 × Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping
  • 2 × Hook 'pre_llm_call' callback _pre_llm_call timed out after 120s — skipping (06:05:33, 08:41:41)
  • 10 × tool calls returned {"error": "pre_tool_call plugin callback timed out or is still running"}

Across the log's retained window (errors.log starts 2026-11-09 13:18; .1 / .2 rotations hold older lines):

Day Plugin load failures Registrations dropped pre_tool_call timeouts pre_llm_call timeouts Tool refusals
2026-11-12 0 0 0 0 0
2026-11-13 0 0 0 0 0
2026-11-14 4 0 1 1 4
2026-11-15 52 519 11 2 10
2026-11-16 0 0 0 0 0
2026-11-17 0 0 0 0 0

By plugin (2026-11-15)

Plugin Failed loads Registrations dropped
hermes-hq-push 26 0
conduit_push 9 474
homeassistant 7 34
browser-browser-use 3 3
basic 2 2
cadu-secrets-vault 1 4
telegram-platform 1 1
nous 1 0
drain 1 0
browser-browserbase 1 1
Total 52 519

The dropped registrations are the more serious half

A failed load is a warning line. The register_*() lines that follow are function loss: the loader gives each plugin a 10 s budget for import + register(), and when that budget expires the plugin's late registrations are discarded, not honoured. conduit_push alone lost 474 registrations across eight load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47) — i.e. for those processes the Conduit tools, skills and hooks were not registered at all, silently. See systems/conduit.

Clustering

These arrive in bursts, not as a trickle. 18 distinct registration-drop groups on 11-15; load failures spread over hours 01, 04–11 and 13:

  • 01:05–01:32 — hermes-hq-push (01:05, 01:17, 01:32) and homeassistant (01:17) fail to load, conduit_push fails at 01:32:39 and drops ~52 registrations. Inside the same 20 minutes pre_tool_call times out 5× (01:07:17, 01:16:11, 01:19:36, 01:22:39, 01:25:43) and 9 tool calls are refused in the same session (local-989b3642…).
  • 04:46–04:47 — the day's largest burst: browser-browser-use, conduit_push ×2, hermes-hq-push ×2 and homeassistant all fail within ~55 s; 118 registrations discarded.
  • 06:31–06:48, 07:06, 08:16, 08:46, 10:47 — repeated conduit_push/hermes-hq-push failures, each dropping ~52–56 registrations.
  • 12:18–12:49 — a second pre_tool_call cluster (6 timeouts + 1 refusal) with no load failures in hours 11–12 bar one at 11:49 — so the hook half of this page's family can fire without a loader failure alongside it.

What this is — and is not

  • Not a misconfiguration. plugins.hook_callback_timeout: 120 is set in config.yaml and no plugin-load-timeout key exists there — the 10 s loader budget is a code default. The failures are a saturation symptom on this host, not a wrong setting.
  • Same window as the hook cascade. incidents/2026-11-08-pre-tool-call-hook-timeout-cascade recurred on 11-15 with the 11 pre_tool_call timeouts and 10 refusals counted above; the runtime deps guard also blew its 600 s budget that day (see incidents/2026-11-14-mnemosyne-capture-stall). Several of the estate's own timing budgets were exceeded in the same 24 h.
  • Adjacency, not proof. Nothing ties the three to a single cause. What they share is a host under sustained IO/scheduler pressure while state.db had grown past 2 GB with a 40 MB WAL (/ at 88 %). But load average at the 2026-11-17 audit read 6.19 / 6.64 / 7.48 — below the 11-14 reading of 12.08 — with no recurrence of this class on 11-16/11-17, so load average alone does not predict it.
  • Not host restarts. gateway.log has 0 startup/shutdown markers on 11-15, and uptime at the audit was 6 d 2 h (host last up 2026-11-10 ~22:04) — so the ~18 load events are in-process plugin (re)loads. Plugin registration output in agent.log appears at agent-process start (e.g. the 2026-11-16 00:00:00 block), which is why a job starting during saturation can run with plugins missing.

Open question

Why did import + register() exceed 10 s for whole plugins, and does the loader fail closed or open for the caller? Today it fails open — the job continues, silently without those tools. A check that asserts plugin registration rather than only job exit status would surface this class; the same idea applies to the hook callback budget. Proposed, not applied (it changes Hermes runtime behaviour, so it is Anthony's call).

See also