Files
agent-estate-wiki/incidents/2026-11-14-mnemosyne-capture-stall.md
T
Tony0410 4a6e86948d wiki: 2026-11-14 nightly audit + corrections — Phonon TTS on CT200, Mnemosyne capture stall open
- new: systems/phonon-tts.md, incidents/2026-11-14-mnemosyne-capture-stall.md
- CT200 inventory re-verified: 64 non-seeder running / 72 total (delta = the TTS trio, no removals)
- fleet ledger: 11-14 = 167 rows / 166 completed / 0 failed; first red last_status since 11-13
  (Mnemosyne Health Monitor, error, streak 1); cron_incidents 0 -> 1 unclosed (138 rows)
- corrections pass: withdrew the 'unidentified extra container' claim; closed the alert-delivery
  question (delivered 00:03:39, for_failure=1)
- updated: index.md, current-state.md, log.md, systems/scheduled-tasks.md, systems/mnemosyne.md,
  infrastructure/docker-services.md
2026-11-15 00:42:01 +08:00

109 lines
6.3 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Mnemosyne capture stalled for ~15 hours (2026-11-14 09:33 AWST onward) — health monitor escalated 00:03 AWST 2026-11-15
type: incident
status: active
created: 2026-11-15
updated: 2026-11-15
verified_on: 2026-11-15
review_after: 2026-11-17
confidence: high
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident]
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db]
---
# Mnemosyne capture stalled for ~15 hours (from 2026-11-14 09:33 AWST)
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
(exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state **`alerted`**, **unclosed**).
Found by the **2026-11-14 nightly audit**; not caused by anything in that audit's window (it started
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
the same condition was classified informational — see below).
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
**2026-11-14 09:32:59 AWST** (the row's own `timestamp` field reads `2026-11-14T09:32:59`).
- Newest `episodic_memory.created_at`: **`2026-11-14 01:33:34`** (09:33:34 AWST). **Both** write paths
stopped inside the same minute.
- Last 8 `working_memory` writes, all on 11-14 (UTC): `01:01:20, 01:02:39, 01:02:40, 01:03:28,
01:04:19, 01:32:58, 01:32:59, 01:32:59` — then **nothing for ~15 hours**.
- Rows by day (UTC `created_at`): **11-09: 11, 11-10: 10, 11-11: 8, 11-12: 37, 11-13: 14, 11-14: 15**.
So the pipeline was *low-volume but alive* before the stall, and the stall is not a normal quiet stretch.
- Totals at audit time: **working 1130 / episodic 1249**, DB **27 MB**.
- Gap: at the 00:03:35 alert the newest capture was **14 h 30 m 36 s** old; at this audit's read
(~00:40) **≈15 h 7 m**.
## The alert (and its mislabelling)
```
🚨 Mnemosyne Health Monitor — exception(s):
STALE MEMORY PIPELINE: capture missed: user message 4.64157453 ago (desktop)
has no working_memory row (last capture 2026-11-14 01:32:59)
```
- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`,
`first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`.
Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`.
- ⚠ **The `auth` label looks wrong** — the failure is a *staleness* check, not a credential fault, and
the page records no auth error text. **Observation, not verified diagnosis.**
- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a
~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own
log line below.
## The earlier, quieter reading — 12:00:52 AWST 2026-11-14
`~/.hermes/logs/mnemosyne-health-monitor.log` shows the **12:00:52 AWST** run already looking at the
same hole and calling it benign:
```
ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min
```
That is **9 h 49.5 m** of missing capture, logged as ℹ️ informational (the delay is **negative**, i.e.
capture sits *behind* the message clock — the sign convention makes a growing gap look small). The
job exited 0 at 12:00 and 18:03; it only escalated at 00:03 the next morning.
**So the fault had ~12 hours of lead time that produced no alarm.**
## Impact
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
window. Recall over that period will answer from nothing.
- The **23:30 AWST sleep/consolidation cycle ran and found 0 items** to consolidate
(`Last consolidation: 2026-11-14T15:30:51 UTC (0 items)`), so consolidation is *not* compensating.
- Nothing else in the estate broke: **2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` /
1 `running`** in `executions.db`, deliveries **4 / 4 `delivered`**, and no other cron row went red. This is a
**memory-path incident only**. The one exception is the monitor's **own** job: it is now the fleet's
**single non-clean row** (`last_status: error`, `failure_streak` 1, previously `ok` / 0) and
**`cron_incidents` went 0 → 1 unclosed** (138 rows total). **2026-11-15 so far: 8 rows — 7
`completed` / 1 `failed`** (this job), 1 delivery `delivered`.
## Not the known config killer
The previously recorded capture-killer on this host is a **stringified list/dict** written into
`~/.hermes/config.yaml` by `hermes config set`. Checked and **clean**:
- `config.yaml` **mtime 2026-11-13 18:03** — the file was not modified during or before the stall began.
- A full YAML walk found **no** scalar that begins with `[` or `{`; `memory.mnemosyne.*` parses as
proper lists (`ignore_patterns: []`, `skip_contexts`, `sync_roles`).
## Open / unverified
- **Root cause: not established.** No capture-path error lines appear in `agent.log` after 09:33 —
the writes simply stop. Candidates to test next: a stuck capture worker or watermark, the
`pulse`/heartbeat traffic that ran 12:03–19:22, session-side `memory_enabled` /
`mnemosyne.profile_isolation` interaction, or the sleep cycle's `consolidation_claimed_at` state.
- ✅ **Resolved — the 00:03 escalation DID deliver.** `deliveries.db` holds exactly one 2026-11-15 row:
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
- Whether capture resumed after this audit's read (the monitor's next slot is the natural check).
## Related
- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence
- [[systems/memory-backend]] — memory backend settings
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] — the still-open secrets-at-rest page for the same database
- [[systems/scheduled-tasks]] — the fleet's execution ledger