--- title: Mnemosyne capture stalled for ~15 hours (2026-11-14 09:33 AWST onward) — health monitor escalated 00:03 AWST 2026-11-15 type: incident status: active created: 2026-11-15 updated: 2026-11-15 verified_on: 2026-11-15 review_after: 2026-11-17 confidence: high tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident] sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db] --- # Mnemosyne capture stalled for ~15 hours (from 2026-11-14 09:33 AWST) **Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**, and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state **`alerted`**, **unclosed**). Found by the **2026-11-14 nightly audit**; not caused by anything in that audit's window (it started the previous morning and simply had not been noticed, because the monitor's own earlier reading of the same condition was classified informational — see below). ## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST) - Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e. **2026-11-14 09:32:59 AWST** (the row's own `timestamp` field reads `2026-11-14T09:32:59`). - Newest `episodic_memory.created_at`: **`2026-11-14 01:33:34`** (09:33:34 AWST). **Both** write paths stopped inside the same minute. - Last 8 `working_memory` writes, all on 11-14 (UTC): `01:01:20, 01:02:39, 01:02:40, 01:03:28, 01:04:19, 01:32:58, 01:32:59, 01:32:59` — then **nothing for ~15 hours**. - Rows by day (UTC `created_at`): **11-09: 11, 11-10: 10, 11-11: 8, 11-12: 37, 11-13: 14, 11-14: 15**. So the pipeline was *low-volume but alive* before the stall, and the stall is not a normal quiet stretch. - Totals at audit time: **working 1130 / episodic 1249**, DB **27 MB**. - Gap: at the 00:03:35 alert the newest capture was **14 h 30 m 36 s** old; at this audit's read (~00:40) **≈15 h 7 m**. ## The alert (and its mislabelling) ``` 🚨 Mnemosyne Health Monitor — exception(s): STALE MEMORY PIPELINE: capture missed: user message 4.64157453 ago (desktop) has no working_memory row (last capture 2026-11-14 01:32:59) ``` - `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`, `first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`. Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`. - ⚠ **The `auth` label describes the classifier, not the fault — now evidenced, not just suspected.** The failure is a *staleness* check and the page records no auth error text. This job's **twelve earlier `cron_incidents` rows** read `agent` ×3, `timeout` ×1, `unknown` ×8 (`ea79a2_649d8ac46e8a`, `_f17b45759db0`, `_573d17e10fc1`, `_6fb70b032305`, `_54d6b1f0f58a`, `_747b27556fd0`, `_c3ab2d784ba3`, `_a4661554a3c2`, `_d93c3acdc4f2`, `_9197df32320b`, `_111e67a3d83c`, `_f317fcd843ae`), and the estate-wide table is **53 `unknown` / 47 `agent` / 15 `timeout` / 14 `script` / 5 `auth`**. `agent` is also impossible here: the job runs `no_agent` (plain script). Treat the field as opaque routing metadata. - ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a ~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own log line below. ## What the earlier slots actually saw — correction, re-derived 2026-11-15 01:2x AWST **An earlier revision of this page got this backwards.** It claimed the **12:00:52 AWST** run had "already seen the same hole and called it benign" (`capture delay -589.5 min`), giving the estate ~12 h of unalarmed lead time, and blamed a "sign, not magnitude" threshold. **Both claims are withdrawn.** The `-589.5 min` line is not the 12:00 run's — it sits inside the **00:03:35 run's own output block**. Verified by line number in `~/.hermes/logs/mnemosyne-health-monitor.log` (**1103 lines**): | line | content | | --- | --- | | 884 | `=== 2026-11-14 00:09:12 AWST health monitor (no_agent) ===` | | 900 | `ℹ️ last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min` | | 957 | `=== 2026-11-14 12:01:52 AWST health monitor (no_agent) ===` | | 973 | `ℹ️ last user msg 2026-11-14 01:03 UTC (ios) capture delay 29.4 min` | | 1030 | `=== 2026-11-15 00:03:35 AWST health monitor (no_agent) ===` | | 1046 | `ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min` | | 1047 | `❌ STALE MEMORY PIPELINE: …` | Two further corrections to the earlier revision: - **There was no 18:03 run.** The job's schedule is `0 */12 * * *`; the only 2026-11-14 slots are 00:09:12 and 12:01:52 (both exit 0, delivery `suppressed`). The "exited 0 at 12:00 and 18:03" sentence is withdrawn — 18:03 belongs to some other job. - **The detection chain was not broken.** `mnemosyne_freshness_gate.py` fires on **magnitude** (`delay_min < -CAPTURE_LAG_MIN`, `MNEMOSYNE_CAPTURE_LAG_MIN` default **120** min) — there is no sign-only test to blame. The 12:00:52 measurement of **+29.4 min** is the delay of the newest message *that had been captured* (the 09:03 AWST `ios` turn, captured 09:32:59). The gate computes against the newest **uncaptured** message and only reports a problem when one exists; with the conversation idle from 09:03 to 19:22, every user message still had a capture row, so there was **nothing to detect**. The first message to postdate the stall was the **19:22 AWST desktop** turn (`11:22 UTC`, the same message the 00:03 alert names); the next scheduled slot after it was 00:03:35, which escalated (`-589.5 min` = 09:32:59 − 19:22). **Corrected statement of the gap:** the pipeline died at **09:32:59 AWST**; it became *detectable* at **19:22 AWST** (first uncaptured message) and was *detected* at **00:03:35** by the next slot — a detection latency of 4 h 41 m after detectability, driven by cadence, not by a broken threshold. **What was genuinely unread sits in the consolidation counters:** the cycles ran every ~31 min and returned **0 items for 29 consecutive runs** (streak opening `2026-11-14T02:04:42+00:00` = **10:04:42 AWST**; last productive cycles 4622/4621/4620 at `09:33:34` / `09:33:16` / `09:01:40`). Every run printed `Last consolidation: … (0 items)` — including the "healthy" 12:00:52 one — but the gate tests consolidation **age** only (`> 36 h`, default), never the item count. **An empty-but-current cycle is the loudest signal available and the only one nothing watches.** ## Impact - ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST** (the last user message the monitor could see), all Telegram traffic, and every cron session in the window. Recall over that period will answer from nothing. - The **23:30 AWST sleep/consolidation cycle ran and found 0 items** to consolidate (`Last consolidation: 2026-11-14T15:30:51 UTC (0 items)`), so consolidation is *not* compensating. - Nothing else in the estate broke: **2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` / 1 `running`** in `executions.db`, deliveries **4 / 4 `delivered`**, and no other cron row went red. This is a **memory-path incident only**. The one exception is the monitor's **own** job: it is now the fleet's **single non-clean row** (`last_status: error`, `failure_streak` 1, previously `ok` / 0) and **`cron_incidents` went 0 → 1 unclosed** (138 rows total). **2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (this job), 1 delivery `delivered`. ## Not the known config killer The previously recorded capture-killer on this host is a **stringified list/dict** written into `~/.hermes/config.yaml` by `hermes config set`. Checked and **clean**: - `config.yaml` **mtime 2026-11-13 18:03** — the file was not modified during or before the stall began. - A full YAML walk found **no** scalar that begins with `[` or `{`; `memory.mnemosyne.*` parses as proper lists (`ignore_patterns: []`, `skip_contexts`, `sync_roles`). ## Open / unverified - **Root cause: not established.** No capture-path error lines appear in `agent.log` after 09:33 — the writes simply stop. Candidates to test next: a stuck capture worker or watermark, the `pulse`/heartbeat traffic that ran 12:03–19:22, session-side `memory_enabled` / `mnemosyne.profile_isolation` interaction, or the sleep cycle's `consolidation_claimed_at` state. - ✅ **Resolved — the 00:03 escalation DID deliver.** `deliveries.db` holds exactly one 2026-11-15 row: **`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked; the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]). - Whether capture resumed after this audit's read (the monitor's next slot is the natural check). ## Related - [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence - [[systems/memory-backend]] — memory backend settings - [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] — the still-open secrets-at-rest page for the same database - [[systems/scheduled-tasks]] — the fleet's execution ledger