audit 2026-11-16: memory stall 62h54m (counts unchanged, cron_incidents 3->5 unclosed, consolidator zero-item streak 120); NEW incident — 11-15 plugin load-timeout storm (52 failed loads, 519 dropped registrations incl. conduit_push 474, 11 pre_tool_call + 2 pre_llm_call hook timeouts, 10 refusals) never recorded by the 11-15 audit; hook page Recurrence 2 (load average removed as predictor); fleet 29 jobs, 11-16 ledger 175 rows 173/2, deliveries 11/11

This commit is contained in:
hermes
2026-11-17 00:36:27 +08:00
parent 4eee02b9a2
commit e5612c8a8d
9 changed files with 299 additions and 22 deletions
@@ -1,17 +1,17 @@
---
title: Mnemosyne capture stalled for 38+ hours (2026-11-14 09:33 AWST onward) — three unclosed alerts, freeze total on re-read
title: Mnemosyne capture stalled for 62+ hours (2026-11-14 09:33 AWST onward) — five unclosed alerts, freeze total on re-read
type: incident
status: active
created: 2026-11-15
updated: 2026-11-16
verified_on: 2026-11-16
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-18
confidence: high
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
---
# Mnemosyne capture stalled for 38+ hours (from 2026-11-14 09:33 AWST)
# Mnemosyne capture stalled for 62+ hours (from 2026-11-14 09:33 AWST)
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
@@ -110,6 +110,40 @@ proof of common cause** — but the guard and the capture path share the same ru
times out at 600 s belongs next to this stall rather than dismissed.
## Re-read 2026-11-17 00:27 AWST — stall **62 h 54 m**, **five** unclosed alerts, zero-item streak **120**
Third nightly read, sixth week of the same numbers changing only their counters. **The freeze is still
total.**
- **Counts identical for the fifth consecutive read:** `working_memory` **1130** / `episodic_memory`
**1249**; newest rows still `2026-11-14 09:32:59` AWST (working) and `09:33:34` AWST (episodic) —
**62 h 54 m** with **zero** writes.
- **`cron_incidents` 3 → 5 unclosed** (141 → **143** rows total). Both new rows are `ea79a2107e36`,
`failure_type: auth`, `state: alerted`, and again `first_seen_at == last_seen_at`:
| Incident id | `first_seen_at` | `alerted_at` |
| --- | --- | --- |
| `ea79a2_915d004a108a` | `2026-11-16T12:02:06.679532+08:00` | `2026-11-16T12:02:12.472265+08:00` |
| `ea79a2_0fab8d43db75` | `2026-11-17T00:03:37.624640+08:00` | `2026-11-17T00:03:46.151392+08:00` |
The signature-fragmentation mechanism below is holding exactly as described: **one fresh row per
firing, no escalation**, so the estate's unclosed count now grows **~2 per day** with nothing accumulating
on any single row.
- **The monitor is on a 5-slot failure streak** (`jobs.json`: `ea79a2107e36`, `last_status: error`,
`failure_streak: 5`). 11-16's ledger holds **2 failed rows of 175, both this job**
(00:01:33 → 00:03:22, 12:01:18 → 12:02:12); 11-17 so far 1 failed of 7 (00:01:39 → 00:03:46).
Every failure is `exit 1` with `STALE MEMORY PIPELINE: capture missed: us…` — the same gate.
- **The consolidator is alive and deliberately doing nothing.** `consolidation_log` latest row **4742**
at `2026-11-16T16:00:41.922076+00:00` (= 00:00:41 AWST), `items_consolidated: 0`, summary *"No-op
consolidation cycle — timestamp updated for watchdog"*. Last non-zero row: **4622** at
`2026-11-14T09:33:34` → **120 consecutive zero-item cycles** (was 74 at the 11-16 read). The cycles
still fire ~every 31 min; there is simply **nothing new to consolidate**, which is the memory-side
reason the stall cannot self-heal.
- **Still no cause, still no fix, still `status: active`.** The 11-16 read's two conclusions stand: the
failure is **total**, not a trickle, and it affects **all** write paths, not just turn capture.
`review_after` 2026-11-18 — this page is re-read at the next audit, and the monitor keeps alerting.
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.