audit 2026-11-16: memory stall 62h54m (counts unchanged, cron_incidents 3->5 unclosed, consolidator zero-item streak 120); NEW incident — 11-15 plugin load-timeout storm (52 failed loads, 519 dropped registrations incl. conduit_push 474, 11 pre_tool_call + 2 pre_llm_call hook timeouts, 10 refusals) never recorded by the 11-15 audit; hook page Recurrence 2 (load average removed as predictor); fleet 29 jobs, 11-16 ledger 175 rows 173/2, deliveries 11/11
This commit is contained in:
@@ -3,9 +3,9 @@ title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-09
|
||||
updated: 2026-11-14
|
||||
verified_on: 2026-11-14
|
||||
review_after: 2026-11-17
|
||||
updated: 2026-11-17
|
||||
verified_on: 2026-11-17
|
||||
review_after: 2026-11-20
|
||||
confidence: high
|
||||
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
|
||||
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
|
||||
@@ -128,6 +128,33 @@ insufficient at current load**, and this page is no longer `resolved`.
|
||||
fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and
|
||||
reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
|
||||
|
||||
## Recurrence 2 (2026-11-15) — eleven timeouts in two clusters, 10 refusals
|
||||
|
||||
Recorded by the **2026-11-17 nightly audit** while backing up the 11-15 window: the **2026-11-15 audit
|
||||
captured none of this**. Exact `errors.log` counts for 11-15:
|
||||
|
||||
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` —
|
||||
**5** in 01:07:17 → 01:25:43 and **6** in 12:18:15 → 12:48:40 (a two-cluster day, unlike 11-14's one).
|
||||
- **10** tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
|
||||
— hour 01: `read_file` ×2 (134.17 s and 0.43 s), `execute_code` ×4, `patch` ×1, `read_file` ×1
|
||||
(121.03 s); hour 12: `execute_code` ×1. Call-level blocks reach **120.63 s**, and the affected sessions
|
||||
are `local-989b3642…` (01:xx) and `local-b6a1490f…` (12:xx).
|
||||
- 11-14, for comparison: **1** timeout and **4** refusals, plus **4** plugin load failures.
|
||||
- **11-16 and 11-17: 0 timeouts, 0 refusals, 0 load failures.** The episode did not carry over.
|
||||
|
||||
The loader half of the same 11-15 storm — **52** failed plugin loads, **519** discarded registrations
|
||||
(`conduit_push` 474 of them) and **2** `pre_llm_call` timeouts — is now on its own page,
|
||||
[[incidents/2026-11-15-plugin-load-timeout-storm]]. Its 12:18–12:49 cluster is the useful detail here:
|
||||
**a hook-timeout cluster can occur with no accompanying loader failure**, so the two share a cause class
|
||||
rather than a mechanism.
|
||||
|
||||
**Effect on the trigger hypothesis recorded above.** Amplitude is not a simple function of load average:
|
||||
the 11-15 episode ran at a load the audit cannot measure retroactively, while the 11-17 read
|
||||
(**6.19 / 6.64 / 7.48**) sits *below* the 11-14 figure of 12.08 with **no** recurrence — and a *larger*
|
||||
`state.db` (2.09 GB, WAL 40 MB, `/` at 88 %). Keep a **load-average threshold out of this page's claim**;
|
||||
keep **`state.db` size, disk headroom and the gateway's own timing budgets** in it.
|
||||
|
||||
|
||||
## See also
|
||||
|
||||
- [[systems/conduit]]
|
||||
|
||||
@@ -1,17 +1,17 @@
|
||||
---
|
||||
title: Mnemosyne capture stalled for 38+ hours (2026-11-14 09:33 AWST onward) — three unclosed alerts, freeze total on re-read
|
||||
title: Mnemosyne capture stalled for 62+ hours (2026-11-14 09:33 AWST onward) — five unclosed alerts, freeze total on re-read
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-15
|
||||
updated: 2026-11-16
|
||||
verified_on: 2026-11-16
|
||||
updated: 2026-11-17
|
||||
verified_on: 2026-11-17
|
||||
review_after: 2026-11-18
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
|
||||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
|
||||
---
|
||||
|
||||
# Mnemosyne capture stalled for 38+ hours (from 2026-11-14 09:33 AWST)
|
||||
# Mnemosyne capture stalled for 62+ hours (from 2026-11-14 09:33 AWST)
|
||||
|
||||
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
|
||||
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
|
||||
@@ -110,6 +110,40 @@ proof of common cause** — but the guard and the capture path share the same ru
|
||||
times out at 600 s belongs next to this stall rather than dismissed.
|
||||
|
||||
|
||||
## Re-read 2026-11-17 00:27 AWST — stall **62 h 54 m**, **five** unclosed alerts, zero-item streak **120**
|
||||
|
||||
Third nightly read, sixth week of the same numbers changing only their counters. **The freeze is still
|
||||
total.**
|
||||
|
||||
- **Counts identical for the fifth consecutive read:** `working_memory` **1130** / `episodic_memory`
|
||||
**1249**; newest rows still `2026-11-14 09:32:59` AWST (working) and `09:33:34` AWST (episodic) —
|
||||
**62 h 54 m** with **zero** writes.
|
||||
- **`cron_incidents` 3 → 5 unclosed** (141 → **143** rows total). Both new rows are `ea79a2107e36`,
|
||||
`failure_type: auth`, `state: alerted`, and again `first_seen_at == last_seen_at`:
|
||||
|
||||
| Incident id | `first_seen_at` | `alerted_at` |
|
||||
| --- | --- | --- |
|
||||
| `ea79a2_915d004a108a` | `2026-11-16T12:02:06.679532+08:00` | `2026-11-16T12:02:12.472265+08:00` |
|
||||
| `ea79a2_0fab8d43db75` | `2026-11-17T00:03:37.624640+08:00` | `2026-11-17T00:03:46.151392+08:00` |
|
||||
|
||||
The signature-fragmentation mechanism below is holding exactly as described: **one fresh row per
|
||||
firing, no escalation**, so the estate's unclosed count now grows **~2 per day** with nothing accumulating
|
||||
on any single row.
|
||||
- **The monitor is on a 5-slot failure streak** (`jobs.json`: `ea79a2107e36`, `last_status: error`,
|
||||
`failure_streak: 5`). 11-16's ledger holds **2 failed rows of 175, both this job**
|
||||
(00:01:33 → 00:03:22, 12:01:18 → 12:02:12); 11-17 so far 1 failed of 7 (00:01:39 → 00:03:46).
|
||||
Every failure is `exit 1` with `STALE MEMORY PIPELINE: capture missed: us…` — the same gate.
|
||||
- **The consolidator is alive and deliberately doing nothing.** `consolidation_log` latest row **4742**
|
||||
at `2026-11-16T16:00:41.922076+00:00` (= 00:00:41 AWST), `items_consolidated: 0`, summary *"No-op
|
||||
consolidation cycle — timestamp updated for watchdog"*. Last non-zero row: **4622** at
|
||||
`2026-11-14T09:33:34` → **120 consecutive zero-item cycles** (was 74 at the 11-16 read). The cycles
|
||||
still fire ~every 31 min; there is simply **nothing new to consolidate**, which is the memory-side
|
||||
reason the stall cannot self-heal.
|
||||
- **Still no cause, still no fix, still `status: active`.** The 11-16 read's two conclusions stand: the
|
||||
failure is **total**, not a trickle, and it affects **all** write paths, not just turn capture.
|
||||
`review_after` 2026-11-18 — this page is re-read at the next audit, and the monitor keeps alerting.
|
||||
|
||||
|
||||
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
||||
|
||||
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
title: Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
|
||||
type: incident
|
||||
status: unresolved
|
||||
created: 2026-11-17
|
||||
updated: 2026-11-17
|
||||
verified_on: 2026-11-17
|
||||
review_after: 2026-11-24
|
||||
confidence: high
|
||||
tags: [hermes, plugin, conduit, timeout, infrastructure, incident, load]
|
||||
sources: [live-errors-log, live-config-yaml, live-gateway-log, live-cron-jobs-json]
|
||||
---
|
||||
|
||||
# Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
|
||||
|
||||
**Not observed on 2026-11-16 or 2026-11-17 — 0 lines of either signature on both days.** No fix was
|
||||
applied; the storm stopped as abruptly as it started, and the loader budget that produced it is
|
||||
unchanged, so recurrence is possible. Found by the **2026-11-17 nightly audit**, which was filling the
|
||||
11-15 window's gaps; the **2026-11-15 audit recorded none of this**. Every count below is an exact
|
||||
`errors.log` grep, not an estimate.
|
||||
|
||||
## What the log records (2026-11-15)
|
||||
|
||||
- **52** × `Failed to load plugin '<name>': load timed out after 10s (import + register() never returned)`
|
||||
- **519** × `Plugin '<name>' called register_{tool,hook,skill,platform}() after its load timed out; ignored`
|
||||
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping`
|
||||
- **2** × `Hook 'pre_llm_call' callback _pre_llm_call timed out after 120s — skipping` (06:05:33, 08:41:41)
|
||||
- **10** × tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
|
||||
|
||||
Across the log's retained window (`errors.log` starts **2026-11-09 13:18**; `.1` / `.2` rotations hold
|
||||
older lines):
|
||||
|
||||
| Day | Plugin load failures | Registrations dropped | `pre_tool_call` timeouts | `pre_llm_call` timeouts | Tool refusals |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 2026-11-12 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 2026-11-13 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 2026-11-14 | 4 | 0 | 1 | 1 | 4 |
|
||||
| **2026-11-15** | **52** | **519** | **11** | **2** | **10** |
|
||||
| 2026-11-16 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 2026-11-17 | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
### By plugin (2026-11-15)
|
||||
|
||||
| Plugin | Failed loads | Registrations dropped |
|
||||
| --- | --- | --- |
|
||||
| `hermes-hq-push` | 26 | 0 |
|
||||
| `conduit_push` | 9 | **474** |
|
||||
| `homeassistant` | 7 | 34 |
|
||||
| `browser-browser-use` | 3 | 3 |
|
||||
| `basic` | 2 | 2 |
|
||||
| `cadu-secrets-vault` | 1 | 4 |
|
||||
| `telegram-platform` | 1 | 1 |
|
||||
| `nous` | 1 | 0 |
|
||||
| `drain` | 1 | 0 |
|
||||
| `browser-browserbase` | 1 | 1 |
|
||||
| **Total** | **52** | **519** |
|
||||
|
||||
## The dropped registrations are the more serious half
|
||||
|
||||
A failed load is a warning line. The `register_*()` lines that follow are **function loss**: the loader
|
||||
gives each plugin a **10 s** budget for `import + register()`, and when that budget expires the plugin's
|
||||
late registrations are **discarded, not honoured**. `conduit_push` alone lost **474** registrations across
|
||||
**eight** load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47) — i.e. for those
|
||||
processes the Conduit tools, skills and hooks were **not registered at all**, silently. See
|
||||
[[systems/conduit]].
|
||||
|
||||
## Clustering
|
||||
|
||||
These arrive in bursts, not as a trickle. 18 distinct registration-drop groups on 11-15; load failures
|
||||
spread over hours 01, 04–11 and 13:
|
||||
|
||||
- **01:05–01:32** — `hermes-hq-push` (01:05, 01:17, 01:32) and `homeassistant` (01:17) fail to load,
|
||||
`conduit_push` fails at 01:32:39 and drops ~52 registrations. Inside the same 20 minutes
|
||||
`pre_tool_call` times out **5×** (01:07:17, 01:16:11, 01:19:36, 01:22:39, 01:25:43) and **9** tool calls
|
||||
are refused in the same session (`local-989b3642…`).
|
||||
- **04:46–04:47** — the day's largest burst: `browser-browser-use`, `conduit_push` ×2, `hermes-hq-push` ×2
|
||||
and `homeassistant` all fail within ~55 s; **118** registrations discarded.
|
||||
- **06:31–06:48, 07:06, 08:16, 08:46, 10:47** — repeated `conduit_push`/`hermes-hq-push` failures, each
|
||||
dropping ~52–56 registrations.
|
||||
- **12:18–12:49** — a **second** `pre_tool_call` cluster (6 timeouts + 1 refusal) with **no load failures**
|
||||
in hours 11–12 bar one at 11:49 — so the hook half of this page's family can fire without a loader
|
||||
failure alongside it.
|
||||
|
||||
## What this is — and is not
|
||||
|
||||
- **Not a misconfiguration.** `plugins.hook_callback_timeout: 120` is set in `config.yaml` and **no**
|
||||
plugin-load-timeout key exists there — the 10 s loader budget is a **code default**. The failures are a
|
||||
**saturation symptom** on this host, not a wrong setting.
|
||||
- **Same window as the hook cascade.** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
recurred on 11-15 with the 11 `pre_tool_call` timeouts and 10 refusals counted above; the runtime deps
|
||||
guard also blew its **600 s** budget that day (see
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]]). Several of the estate's own timing budgets were
|
||||
exceeded in the same 24 h.
|
||||
- **Adjacency, not proof.** Nothing ties the three to a single cause. What they share is a host under
|
||||
sustained IO/scheduler pressure while `state.db` had grown past **2 GB** with a **40 MB** WAL (`/` at
|
||||
88 %). But load average at the 2026-11-17 audit read **6.19 / 6.64 / 7.48** — *below* the 11-14 reading
|
||||
of 12.08 — with **no** recurrence of this class on 11-16/11-17, so **load average alone does not
|
||||
predict it**.
|
||||
- **Not host restarts.** `gateway.log` has **0** startup/shutdown markers on 11-15, and `uptime` at the
|
||||
audit was 6 d 2 h (host last up **2026-11-10 ~22:04**) — so the ~18 load events are **in-process plugin
|
||||
(re)loads**. Plugin registration output in `agent.log` appears at agent-process start (e.g. the
|
||||
2026-11-16 00:00:00 block), which is why a job starting during saturation can run with plugins missing.
|
||||
|
||||
## Open question
|
||||
|
||||
Why did `import + register()` exceed 10 s for whole plugins, and does the loader fail **closed or open**
|
||||
for the caller? Today it fails **open** — the job continues, silently without those tools. A check that
|
||||
asserts **plugin registration** rather than only job exit status would surface this class; the same idea
|
||||
applies to the hook callback budget. **Proposed, not applied** (it changes Hermes runtime behaviour, so
|
||||
it is Anthony's call).
|
||||
|
||||
## See also
|
||||
|
||||
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — the companion hook-callback cascade
|
||||
- [[systems/conduit]] — what the 474 dropped `conduit_push` registrations cost
|
||||
- [[systems/hermes-agent]] — plugin loading, config and timeouts
|
||||
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — same-window runtime pressure
|
||||
Reference in New Issue
Block a user