Files
agent-estate-wiki/incidents/2026-11-14-mnemosyne-capture-stall.md
T
Tony0410 4a6e86948d wiki: 2026-11-14 nightly audit + corrections — Phonon TTS on CT200, Mnemosyne capture stall open
- new: systems/phonon-tts.md, incidents/2026-11-14-mnemosyne-capture-stall.md
- CT200 inventory re-verified: 64 non-seeder running / 72 total (delta = the TTS trio, no removals)
- fleet ledger: 11-14 = 167 rows / 166 completed / 0 failed; first red last_status since 11-13
  (Mnemosyne Health Monitor, error, streak 1); cron_incidents 0 -> 1 unclosed (138 rows)
- corrections pass: withdrew the 'unidentified extra container' claim; closed the alert-delivery
  question (delivered 00:03:39, for_failure=1)
- updated: index.md, current-state.md, log.md, systems/scheduled-tasks.md, systems/mnemosyne.md,
  infrastructure/docker-services.md
2026-11-15 00:42:01 +08:00

6.3 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Mnemosyne capture stalled for ~15 hours (2026-11-14 09:33 AWST onward) — health monitor escalated 00:03 AWST 2026-11-15 incident active 2026-11-15 2026-11-15 2026-11-15 2026-11-17 high
mnemosyne
memory
capture
cron
health-monitor
desktop
pulse
incident
live-mnemosyne-db
live-cron-executions-db
live-cron-incidents
live-hermes-agent-log
live-mnemosyne-health-monitor-log
live-config-yaml
live-state-db

Mnemosyne capture stalled for ~15 hours (from 2026-11-14 09:33 AWST)

Open. Nothing has been written to Mnemosyne's capture tables since 2026-11-14 ~09:33 AWST, and on 2026-11-15 00:03:35 the Mnemosyne Health Monitor (ea79a2107e36) failed on it (exit 1, incident ea79a2_601037516abd, failure_type: auth, state alerted, unclosed).

Found by the 2026-11-14 nightly audit; not caused by anything in that audit's window (it started the previous morning and simply had not been noticed, because the monitor's own earlier reading of the same condition was classified informational — see below).

Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)

  • Newest working_memory.created_at: 2026-11-14 01:32:59 — stored UTC, i.e. 2026-11-14 09:32:59 AWST (the row's own timestamp field reads 2026-11-14T09:32:59).
  • Newest episodic_memory.created_at: 2026-11-14 01:33:34 (09:33:34 AWST). Both write paths stopped inside the same minute.
  • Last 8 working_memory writes, all on 11-14 (UTC): 01:01:20, 01:02:39, 01:02:40, 01:03:28, 01:04:19, 01:32:58, 01:32:59, 01:32:59 — then nothing for ~15 hours.
  • Rows by day (UTC created_at): 11-09: 11, 11-10: 10, 11-11: 8, 11-12: 37, 11-13: 14, 11-14: 15. So the pipeline was low-volume but alive before the stall, and the stall is not a normal quiet stretch.
  • Totals at audit time: working 1130 / episodic 1249, DB 27 MB.
  • Gap: at the 00:03:35 alert the newest capture was 14 h 30 m 36 s old; at this audit's read (~00:40) ≈15 h 7 m.

The alert (and its mislabelling)

🚨 Mnemosyne Health Monitor — exception(s):

STALE MEMORY PIPELINE: capture missed: user message 4.64157453 ago (desktop)
has no working_memory row (last capture 2026-11-14 01:32:59)
  • cron_incidents row ea79a2_601037516abd: state: alerted, failure_type: auth, first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00, closed_at: null. Output file: ~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md.
  • ⚠ The auth label looks wrong — the failure is a staleness check, not a credential fault, and the page records no auth error text. Observation, not verified diagnosis.
  • ⚠ The alert text prints the raw stored value (01:32:59, UTC) with no timezone, which reads as a ~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own log line below.

The earlier, quieter reading — 12:00:52 AWST 2026-11-14

~/.hermes/logs/mnemosyne-health-monitor.log shows the 12:00:52 AWST run already looking at the same hole and calling it benign:

ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min

That is 9 h 49.5 m of missing capture, logged as ℹ️ informational (the delay is negative, i.e. capture sits behind the message clock — the sign convention makes a growing gap look small). The job exited 0 at 12:00 and 18:03; it only escalated at 00:03 the next morning. So the fault had ~12 hours of lead time that produced no alarm.

Impact

  • ~15 hours of context is not in memory: the desktop/Bot Chat session that ran to 19:22 AWST (the last user message the monitor could see), all Telegram traffic, and every cron session in the window. Recall over that period will answer from nothing.
  • The 23:30 AWST sleep/consolidation cycle ran and found 0 items to consolidate (Last consolidation: 2026-11-14T15:30:51 UTC (0 items)), so consolidation is not compensating.
  • Nothing else in the estate broke: 2026-11-14 settled 167 rows — 166 completed / 0 failed / 1 running in executions.db, deliveries 4 / 4 delivered, and no other cron row went red. This is a memory-path incident only. The one exception is the monitor's own job: it is now the fleet's single non-clean row (last_status: error, failure_streak 1, previously ok / 0) and cron_incidents went 0 → 1 unclosed (138 rows total). 2026-11-15 so far: 8 rows — 7 completed / 1 failed (this job), 1 delivery delivered.

Not the known config killer

The previously recorded capture-killer on this host is a stringified list/dict written into ~/.hermes/config.yaml by hermes config set. Checked and clean:

  • config.yaml mtime 2026-11-13 18:03 — the file was not modified during or before the stall began.
  • A full YAML walk found no scalar that begins with [ or {; memory.mnemosyne.* parses as proper lists (ignore_patterns: [], skip_contexts, sync_roles).

Open / unverified

  • Root cause: not established. No capture-path error lines appear in agent.log after 09:33 — the writes simply stop. Candidates to test next: a stuck capture worker or watermark, the pulse/heartbeat traffic that ran 12:03–19:22, session-side memory_enabled / mnemosyne.profile_isolation interaction, or the sleep cycle's consolidation_claimed_at state.
  • ✅ Resolved — the 00:03 escalation DID deliver. deliveries.db holds exactly one 2026-11-15 row: 2026-11-15T00:03:39.052888+08:00, status: delivered, for_failure: 1 — i.e. 3 s after the incident row was raised and 5 s before its alerted_at (00:03:44.5). The notification path worked; the detection path did not (see the ℹ️-at-12:00 bullet on systems/scheduled-tasks).
  • Whether capture resumed after this audit's read (the monitor's next slot is the natural check).