audit 2026-11-16: memory stall 62h54m (counts unchanged, cron_incidents 3->5 unclosed, consolidator zero-item streak 120); NEW incident — 11-15 plugin load-timeout storm (52 failed loads, 519 dropped registrations incl. conduit_push 474, 11 pre_tool_call + 2 pre_llm_call hook timeouts, 10 refusals) never recorded by the 11-15 audit; hook page Recurrence 2 (load average removed as predictor); fleet 29 jobs, 11-16 ledger 175 rows 173/2, deliveries 11/11

This commit is contained in:
hermes
2026-11-17 00:36:27 +08:00
parent 4eee02b9a2
commit e5612c8a8d
9 changed files with 299 additions and 22 deletions
@@ -3,9 +3,9 @@ title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026
type: incident
status: active
created: 2026-11-09
updated: 2026-11-14
verified_on: 2026-11-14
review_after: 2026-11-17
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-20
confidence: high
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
@@ -128,6 +128,33 @@ insufficient at current load**, and this page is no longer `resolved`.
fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and
reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
## Recurrence 2 (2026-11-15) — eleven timeouts in two clusters, 10 refusals
Recorded by the **2026-11-17 nightly audit** while backing up the 11-15 window: the **2026-11-15 audit
captured none of this**. Exact `errors.log` counts for 11-15:
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` —
**5** in 01:07:17 → 01:25:43 and **6** in 12:18:15 → 12:48:40 (a two-cluster day, unlike 11-14's one).
- **10** tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
— hour 01: `read_file` ×2 (134.17 s and 0.43 s), `execute_code` ×4, `patch` ×1, `read_file` ×1
(121.03 s); hour 12: `execute_code` ×1. Call-level blocks reach **120.63 s**, and the affected sessions
are `local-989b3642…` (01:xx) and `local-b6a1490f…` (12:xx).
- 11-14, for comparison: **1** timeout and **4** refusals, plus **4** plugin load failures.
- **11-16 and 11-17: 0 timeouts, 0 refusals, 0 load failures.** The episode did not carry over.
The loader half of the same 11-15 storm — **52** failed plugin loads, **519** discarded registrations
(`conduit_push` 474 of them) and **2** `pre_llm_call` timeouts — is now on its own page,
[[incidents/2026-11-15-plugin-load-timeout-storm]]. Its 12:18–12:49 cluster is the useful detail here:
**a hook-timeout cluster can occur with no accompanying loader failure**, so the two share a cause class
rather than a mechanism.
**Effect on the trigger hypothesis recorded above.** Amplitude is not a simple function of load average:
the 11-15 episode ran at a load the audit cannot measure retroactively, while the 11-17 read
(**6.19 / 6.64 / 7.48**) sits *below* the 11-14 figure of 12.08 with **no** recurrence — and a *larger*
`state.db` (2.09 GB, WAL 40 MB, `/` at 88 %). Keep a **load-average threshold out of this page's claim**;
keep **`state.db` size, disk headroom and the gateway's own timing budgets** in it.
## See also
- [[systems/conduit]]
@@ -1,17 +1,17 @@
---
title: Mnemosyne capture stalled for 38+ hours (2026-11-14 09:33 AWST onward) — three unclosed alerts, freeze total on re-read
title: Mnemosyne capture stalled for 62+ hours (2026-11-14 09:33 AWST onward) — five unclosed alerts, freeze total on re-read
type: incident
status: active
created: 2026-11-15
updated: 2026-11-16
verified_on: 2026-11-16
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-18
confidence: high
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
---
# Mnemosyne capture stalled for 38+ hours (from 2026-11-14 09:33 AWST)
# Mnemosyne capture stalled for 62+ hours (from 2026-11-14 09:33 AWST)
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
@@ -110,6 +110,40 @@ proof of common cause** — but the guard and the capture path share the same ru
times out at 600 s belongs next to this stall rather than dismissed.
## Re-read 2026-11-17 00:27 AWST — stall **62 h 54 m**, **five** unclosed alerts, zero-item streak **120**
Third nightly read, sixth week of the same numbers changing only their counters. **The freeze is still
total.**
- **Counts identical for the fifth consecutive read:** `working_memory` **1130** / `episodic_memory`
**1249**; newest rows still `2026-11-14 09:32:59` AWST (working) and `09:33:34` AWST (episodic) —
**62 h 54 m** with **zero** writes.
- **`cron_incidents` 3 → 5 unclosed** (141 → **143** rows total). Both new rows are `ea79a2107e36`,
`failure_type: auth`, `state: alerted`, and again `first_seen_at == last_seen_at`:
| Incident id | `first_seen_at` | `alerted_at` |
| --- | --- | --- |
| `ea79a2_915d004a108a` | `2026-11-16T12:02:06.679532+08:00` | `2026-11-16T12:02:12.472265+08:00` |
| `ea79a2_0fab8d43db75` | `2026-11-17T00:03:37.624640+08:00` | `2026-11-17T00:03:46.151392+08:00` |
The signature-fragmentation mechanism below is holding exactly as described: **one fresh row per
firing, no escalation**, so the estate's unclosed count now grows **~2 per day** with nothing accumulating
on any single row.
- **The monitor is on a 5-slot failure streak** (`jobs.json`: `ea79a2107e36`, `last_status: error`,
`failure_streak: 5`). 11-16's ledger holds **2 failed rows of 175, both this job**
(00:01:33 → 00:03:22, 12:01:18 → 12:02:12); 11-17 so far 1 failed of 7 (00:01:39 → 00:03:46).
Every failure is `exit 1` with `STALE MEMORY PIPELINE: capture missed: us…` — the same gate.
- **The consolidator is alive and deliberately doing nothing.** `consolidation_log` latest row **4742**
at `2026-11-16T16:00:41.922076+00:00` (= 00:00:41 AWST), `items_consolidated: 0`, summary *"No-op
consolidation cycle — timestamp updated for watchdog"*. Last non-zero row: **4622** at
`2026-11-14T09:33:34` → **120 consecutive zero-item cycles** (was 74 at the 11-16 read). The cycles
still fire ~every 31 min; there is simply **nothing new to consolidate**, which is the memory-side
reason the stall cannot self-heal.
- **Still no cause, still no fix, still `status: active`.** The 11-16 read's two conclusions stand: the
failure is **total**, not a trickle, and it affects **all** write paths, not just turn capture.
`review_after` 2026-11-18 — this page is re-read at the next audit, and the monitor keeps alerting.
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
@@ -0,0 +1,117 @@
---
title: Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
type: incident
status: unresolved
created: 2026-11-17
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-24
confidence: high
tags: [hermes, plugin, conduit, timeout, infrastructure, incident, load]
sources: [live-errors-log, live-config-yaml, live-gateway-log, live-cron-jobs-json]
---
# Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
**Not observed on 2026-11-16 or 2026-11-17 — 0 lines of either signature on both days.** No fix was
applied; the storm stopped as abruptly as it started, and the loader budget that produced it is
unchanged, so recurrence is possible. Found by the **2026-11-17 nightly audit**, which was filling the
11-15 window's gaps; the **2026-11-15 audit recorded none of this**. Every count below is an exact
`errors.log` grep, not an estimate.
## What the log records (2026-11-15)
- **52** × `Failed to load plugin '<name>': load timed out after 10s (import + register() never returned)`
- **519** × `Plugin '<name>' called register_{tool,hook,skill,platform}() after its load timed out; ignored`
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping`
- **2** × `Hook 'pre_llm_call' callback _pre_llm_call timed out after 120s — skipping` (06:05:33, 08:41:41)
- **10** × tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
Across the log's retained window (`errors.log` starts **2026-11-09 13:18**; `.1` / `.2` rotations hold
older lines):
| Day | Plugin load failures | Registrations dropped | `pre_tool_call` timeouts | `pre_llm_call` timeouts | Tool refusals |
| --- | --- | --- | --- | --- | --- |
| 2026-11-12 | 0 | 0 | 0 | 0 | 0 |
| 2026-11-13 | 0 | 0 | 0 | 0 | 0 |
| 2026-11-14 | 4 | 0 | 1 | 1 | 4 |
| **2026-11-15** | **52** | **519** | **11** | **2** | **10** |
| 2026-11-16 | 0 | 0 | 0 | 0 | 0 |
| 2026-11-17 | 0 | 0 | 0 | 0 | 0 |
### By plugin (2026-11-15)
| Plugin | Failed loads | Registrations dropped |
| --- | --- | --- |
| `hermes-hq-push` | 26 | 0 |
| `conduit_push` | 9 | **474** |
| `homeassistant` | 7 | 34 |
| `browser-browser-use` | 3 | 3 |
| `basic` | 2 | 2 |
| `cadu-secrets-vault` | 1 | 4 |
| `telegram-platform` | 1 | 1 |
| `nous` | 1 | 0 |
| `drain` | 1 | 0 |
| `browser-browserbase` | 1 | 1 |
| **Total** | **52** | **519** |
## The dropped registrations are the more serious half
A failed load is a warning line. The `register_*()` lines that follow are **function loss**: the loader
gives each plugin a **10 s** budget for `import + register()`, and when that budget expires the plugin's
late registrations are **discarded, not honoured**. `conduit_push` alone lost **474** registrations across
**eight** load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47) — i.e. for those
processes the Conduit tools, skills and hooks were **not registered at all**, silently. See
[[systems/conduit]].
## Clustering
These arrive in bursts, not as a trickle. 18 distinct registration-drop groups on 11-15; load failures
spread over hours 01, 04–11 and 13:
- **01:05–01:32** — `hermes-hq-push` (01:05, 01:17, 01:32) and `homeassistant` (01:17) fail to load,
`conduit_push` fails at 01:32:39 and drops ~52 registrations. Inside the same 20 minutes
`pre_tool_call` times out **5×** (01:07:17, 01:16:11, 01:19:36, 01:22:39, 01:25:43) and **9** tool calls
are refused in the same session (`local-989b3642…`).
- **04:46–04:47** — the day's largest burst: `browser-browser-use`, `conduit_push` ×2, `hermes-hq-push` ×2
and `homeassistant` all fail within ~55 s; **118** registrations discarded.
- **06:31–06:48, 07:06, 08:16, 08:46, 10:47** — repeated `conduit_push`/`hermes-hq-push` failures, each
dropping ~52–56 registrations.
- **12:18–12:49** — a **second** `pre_tool_call` cluster (6 timeouts + 1 refusal) with **no load failures**
in hours 11–12 bar one at 11:49 — so the hook half of this page's family can fire without a loader
failure alongside it.
## What this is — and is not
- **Not a misconfiguration.** `plugins.hook_callback_timeout: 120` is set in `config.yaml` and **no**
plugin-load-timeout key exists there — the 10 s loader budget is a **code default**. The failures are a
**saturation symptom** on this host, not a wrong setting.
- **Same window as the hook cascade.** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
recurred on 11-15 with the 11 `pre_tool_call` timeouts and 10 refusals counted above; the runtime deps
guard also blew its **600 s** budget that day (see
[[incidents/2026-11-14-mnemosyne-capture-stall]]). Several of the estate's own timing budgets were
exceeded in the same 24 h.
- **Adjacency, not proof.** Nothing ties the three to a single cause. What they share is a host under
sustained IO/scheduler pressure while `state.db` had grown past **2 GB** with a **40 MB** WAL (`/` at
88 %). But load average at the 2026-11-17 audit read **6.19 / 6.64 / 7.48** — *below* the 11-14 reading
of 12.08 — with **no** recurrence of this class on 11-16/11-17, so **load average alone does not
predict it**.
- **Not host restarts.** `gateway.log` has **0** startup/shutdown markers on 11-15, and `uptime` at the
audit was 6 d 2 h (host last up **2026-11-10 ~22:04**) — so the ~18 load events are **in-process plugin
(re)loads**. Plugin registration output in `agent.log` appears at agent-process start (e.g. the
2026-11-16 00:00:00 block), which is why a job starting during saturation can run with plugins missing.
## Open question
Why did `import + register()` exceed 10 s for whole plugins, and does the loader fail **closed or open**
for the caller? Today it fails **open** — the job continues, silently without those tools. A check that
asserts **plugin registration** rather than only job exit status would surface this class; the same idea
applies to the hook callback budget. **Proposed, not applied** (it changes Hermes runtime behaviour, so
it is Anthony's call).
## See also
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — the companion hook-callback cascade
- [[systems/conduit]] — what the 474 dropped `conduit_push` registrations cost
- [[systems/hermes-agent]] — plugin loading, config and timeouts
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — same-window runtime pressure