144 lines
9.2 KiB
Markdown
144 lines
9.2 KiB
Markdown
---
|
||
title: Mnemosyne capture stalled for ~15 hours (2026-11-14 09:33 AWST onward) — health monitor escalated 00:03 AWST 2026-11-15
|
||
type: incident
|
||
status: active
|
||
created: 2026-11-15
|
||
updated: 2026-11-15
|
||
verified_on: 2026-11-15
|
||
review_after: 2026-11-17
|
||
confidence: high
|
||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident]
|
||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db]
|
||
---
|
||
|
||
# Mnemosyne capture stalled for ~15 hours (from 2026-11-14 09:33 AWST)
|
||
|
||
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
|
||
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
|
||
(exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state **`alerted`**, **unclosed**).
|
||
|
||
Found by the **2026-11-14 nightly audit**; not caused by anything in that audit's window (it started
|
||
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
|
||
the same condition was classified informational — see below).
|
||
|
||
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
||
|
||
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
||
**2026-11-14 09:32:59 AWST** (the row's own `timestamp` field reads `2026-11-14T09:32:59`).
|
||
- Newest `episodic_memory.created_at`: **`2026-11-14 01:33:34`** (09:33:34 AWST). **Both** write paths
|
||
stopped inside the same minute.
|
||
- Last 8 `working_memory` writes, all on 11-14 (UTC): `01:01:20, 01:02:39, 01:02:40, 01:03:28,
|
||
01:04:19, 01:32:58, 01:32:59, 01:32:59` — then **nothing for ~15 hours**.
|
||
- Rows by day (UTC `created_at`): **11-09: 11, 11-10: 10, 11-11: 8, 11-12: 37, 11-13: 14, 11-14: 15**.
|
||
So the pipeline was *low-volume but alive* before the stall, and the stall is not a normal quiet stretch.
|
||
- Totals at audit time: **working 1130 / episodic 1249**, DB **27 MB**.
|
||
- Gap: at the 00:03:35 alert the newest capture was **14 h 30 m 36 s** old; at this audit's read
|
||
(~00:40) **≈15 h 7 m**.
|
||
|
||
## The alert (and its mislabelling)
|
||
|
||
```
|
||
🚨 Mnemosyne Health Monitor — exception(s):
|
||
|
||
STALE MEMORY PIPELINE: capture missed: user message 4.64157453 ago (desktop)
|
||
has no working_memory row (last capture 2026-11-14 01:32:59)
|
||
```
|
||
- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`,
|
||
`first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`.
|
||
Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`.
|
||
- ⚠ **The `auth` label describes the classifier, not the fault — now evidenced, not just suspected.** The
|
||
failure is a *staleness* check and the page records no auth error text. This job's **twelve earlier
|
||
`cron_incidents` rows** read `agent` ×3, `timeout` ×1, `unknown` ×8 (`ea79a2_649d8ac46e8a`,
|
||
`_f17b45759db0`, `_573d17e10fc1`, `_6fb70b032305`, `_54d6b1f0f58a`, `_747b27556fd0`, `_c3ab2d784ba3`,
|
||
`_a4661554a3c2`, `_d93c3acdc4f2`, `_9197df32320b`, `_111e67a3d83c`, `_f317fcd843ae`), and the estate-wide
|
||
table is **53 `unknown` / 47 `agent` / 15 `timeout` / 14 `script` / 5 `auth`**. `agent` is also impossible
|
||
here: the job runs `no_agent` (plain script). Treat the field as opaque routing metadata.
|
||
- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a
|
||
~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own
|
||
log line below.
|
||
|
||
## What the earlier slots actually saw — correction, re-derived 2026-11-15 01:2x AWST
|
||
|
||
**An earlier revision of this page got this backwards.** It claimed the **12:00:52 AWST** run had
|
||
"already seen the same hole and called it benign" (`capture delay -589.5 min`), giving the estate ~12 h
|
||
of unalarmed lead time, and blamed a "sign, not magnitude" threshold. **Both claims are withdrawn.** The
|
||
`-589.5 min` line is not the 12:00 run's — it sits inside the **00:03:35 run's own output block**.
|
||
|
||
Verified by line number in `~/.hermes/logs/mnemosyne-health-monitor.log` (**1103 lines**):
|
||
|
||
| line | content |
|
||
| --- | --- |
|
||
| 884 | `=== 2026-11-14 00:09:12 AWST health monitor (no_agent) ===` |
|
||
| 900 | `ℹ️ last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min` |
|
||
| 957 | `=== 2026-11-14 12:01:52 AWST health monitor (no_agent) ===` |
|
||
| 973 | `ℹ️ last user msg 2026-11-14 01:03 UTC (ios) capture delay 29.4 min` |
|
||
| 1030 | `=== 2026-11-15 00:03:35 AWST health monitor (no_agent) ===` |
|
||
| 1046 | `ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min` |
|
||
| 1047 | `❌ STALE MEMORY PIPELINE: …` |
|
||
|
||
Two further corrections to the earlier revision:
|
||
|
||
- **There was no 18:03 run.** The job's schedule is `0 */12 * * *`; the only 2026-11-14 slots are
|
||
00:09:12 and 12:01:52 (both exit 0, delivery `suppressed`). The "exited 0 at 12:00 and 18:03"
|
||
sentence is withdrawn — 18:03 belongs to some other job.
|
||
- **The detection chain was not broken.** `mnemosyne_freshness_gate.py` fires on **magnitude**
|
||
(`delay_min < -CAPTURE_LAG_MIN`, `MNEMOSYNE_CAPTURE_LAG_MIN` default **120** min) — there is no
|
||
sign-only test to blame. The 12:00:52 measurement of **+29.4 min** is the delay of the newest message
|
||
*that had been captured* (the 09:03 AWST `ios` turn, captured 09:32:59). The gate computes against the
|
||
newest **uncaptured** message and only reports a problem when one exists; with the conversation idle
|
||
from 09:03 to 19:22, every user message still had a capture row, so there was **nothing to detect**.
|
||
The first message to postdate the stall was the **19:22 AWST desktop** turn (`11:22 UTC`, the same
|
||
message the 00:03 alert names); the next scheduled slot after it was 00:03:35, which escalated
|
||
(`-589.5 min` = 09:32:59 − 19:22).
|
||
|
||
**Corrected statement of the gap:** the pipeline died at **09:32:59 AWST**; it became *detectable* at
|
||
**19:22 AWST** (first uncaptured message) and was *detected* at **00:03:35** by the next slot — a
|
||
detection latency of 4 h 41 m after detectability, driven by cadence, not by a broken threshold.
|
||
**What was genuinely unread sits in the consolidation counters:** the cycles ran every ~31 min and
|
||
returned **0 items for 29 consecutive runs** (streak opening `2026-11-14T02:04:42+00:00` = **10:04:42
|
||
AWST**; last productive cycles 4622/4621/4620 at `09:33:34` / `09:33:16` / `09:01:40`). Every run
|
||
printed `Last consolidation: … (0 items)` — including the "healthy" 12:00:52 one — but the gate tests
|
||
consolidation **age** only (`> 36 h`, default), never the item count. **An empty-but-current cycle is
|
||
the loudest signal available and the only one nothing watches.**
|
||
|
||
## Impact
|
||
|
||
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
|
||
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
|
||
window. Recall over that period will answer from nothing.
|
||
- The **23:30 AWST sleep/consolidation cycle ran and found 0 items** to consolidate
|
||
(`Last consolidation: 2026-11-14T15:30:51 UTC (0 items)`), so consolidation is *not* compensating.
|
||
- Nothing else in the estate broke: **2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` /
|
||
1 `running`** in `executions.db`, deliveries **4 / 4 `delivered`**, and no other cron row went red. This is a
|
||
**memory-path incident only**. The one exception is the monitor's **own** job: it is now the fleet's
|
||
**single non-clean row** (`last_status: error`, `failure_streak` 1, previously `ok` / 0) and
|
||
**`cron_incidents` went 0 → 1 unclosed** (138 rows total). **2026-11-15 so far: 8 rows — 7
|
||
`completed` / 1 `failed`** (this job), 1 delivery `delivered`.
|
||
|
||
## Not the known config killer
|
||
|
||
The previously recorded capture-killer on this host is a **stringified list/dict** written into
|
||
`~/.hermes/config.yaml` by `hermes config set`. Checked and **clean**:
|
||
- `config.yaml` **mtime 2026-11-13 18:03** — the file was not modified during or before the stall began.
|
||
- A full YAML walk found **no** scalar that begins with `[` or `{`; `memory.mnemosyne.*` parses as
|
||
proper lists (`ignore_patterns: []`, `skip_contexts`, `sync_roles`).
|
||
|
||
## Open / unverified
|
||
|
||
- **Root cause: not established.** No capture-path error lines appear in `agent.log` after 09:33 —
|
||
the writes simply stop. Candidates to test next: a stuck capture worker or watermark, the
|
||
`pulse`/heartbeat traffic that ran 12:03–19:22, session-side `memory_enabled` /
|
||
`mnemosyne.profile_isolation` interaction, or the sleep cycle's `consolidation_claimed_at` state.
|
||
- ✅ **Resolved — the 00:03 escalation DID deliver.** `deliveries.db` holds exactly one 2026-11-15 row:
|
||
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
|
||
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
|
||
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
|
||
- Whether capture resumed after this audit's read (the monitor's next slot is the natural check).
|
||
|
||
## Related
|
||
|
||
- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence
|
||
- [[systems/memory-backend]] — memory backend settings
|
||
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] — the still-open secrets-at-rest page for the same database
|
||
- [[systems/scheduled-tasks]] — the fleet's execution ledger
|