- new: systems/phonon-tts.md, incidents/2026-11-14-mnemosyne-capture-stall.md - CT200 inventory re-verified: 64 non-seeder running / 72 total (delta = the TTS trio, no removals) - fleet ledger: 11-14 = 167 rows / 166 completed / 0 failed; first red last_status since 11-13 (Mnemosyne Health Monitor, error, streak 1); cron_incidents 0 -> 1 unclosed (138 rows) - corrections pass: withdrew the 'unidentified extra container' claim; closed the alert-delivery question (delivered 00:03:39, for_failure=1) - updated: index.md, current-state.md, log.md, systems/scheduled-tasks.md, systems/mnemosyne.md, infrastructure/docker-services.md
6.3 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mnemosyne capture stalled for ~15 hours (2026-11-14 09:33 AWST onward) — health monitor escalated 00:03 AWST 2026-11-15 | incident | active | 2026-11-15 | 2026-11-15 | 2026-11-15 | 2026-11-17 | high |
|
|
Mnemosyne capture stalled for ~15 hours (from 2026-11-14 09:33 AWST)
Open. Nothing has been written to Mnemosyne's capture tables since 2026-11-14 ~09:33 AWST,
and on 2026-11-15 00:03:35 the Mnemosyne Health Monitor (ea79a2107e36) failed on it
(exit 1, incident ea79a2_601037516abd, failure_type: auth, state alerted, unclosed).
Found by the 2026-11-14 nightly audit; not caused by anything in that audit's window (it started the previous morning and simply had not been noticed, because the monitor's own earlier reading of the same condition was classified informational — see below).
Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
- Newest
working_memory.created_at:2026-11-14 01:32:59— stored UTC, i.e. 2026-11-14 09:32:59 AWST (the row's owntimestampfield reads2026-11-14T09:32:59). - Newest
episodic_memory.created_at:2026-11-14 01:33:34(09:33:34 AWST). Both write paths stopped inside the same minute. - Last 8
working_memorywrites, all on 11-14 (UTC):01:01:20, 01:02:39, 01:02:40, 01:03:28, 01:04:19, 01:32:58, 01:32:59, 01:32:59— then nothing for ~15 hours. - Rows by day (UTC
created_at): 11-09: 11, 11-10: 10, 11-11: 8, 11-12: 37, 11-13: 14, 11-14: 15. So the pipeline was low-volume but alive before the stall, and the stall is not a normal quiet stretch. - Totals at audit time: working 1130 / episodic 1249, DB 27 MB.
- Gap: at the 00:03:35 alert the newest capture was 14 h 30 m 36 s old; at this audit's read (~00:40) ≈15 h 7 m.
The alert (and its mislabelling)
🚨 Mnemosyne Health Monitor — exception(s):
STALE MEMORY PIPELINE: capture missed: user message 4.64157453 ago (desktop)
has no working_memory row (last capture 2026-11-14 01:32:59)
cron_incidentsrowea79a2_601037516abd:state: alerted,failure_type: auth,first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00,closed_at: null. Output file:~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md.- ⚠ The
authlabel looks wrong — the failure is a staleness check, not a credential fault, and the page records no auth error text. Observation, not verified diagnosis. - ⚠ The alert text prints the raw stored value (
01:32:59, UTC) with no timezone, which reads as a ~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own log line below.
The earlier, quieter reading — 12:00:52 AWST 2026-11-14
~/.hermes/logs/mnemosyne-health-monitor.log shows the 12:00:52 AWST run already looking at the
same hole and calling it benign:
ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min
That is 9 h 49.5 m of missing capture, logged as ℹ️ informational (the delay is negative, i.e. capture sits behind the message clock — the sign convention makes a growing gap look small). The job exited 0 at 12:00 and 18:03; it only escalated at 00:03 the next morning. So the fault had ~12 hours of lead time that produced no alarm.
Impact
- ~15 hours of context is not in memory: the desktop/Bot Chat session that ran to 19:22 AWST (the last user message the monitor could see), all Telegram traffic, and every cron session in the window. Recall over that period will answer from nothing.
- The 23:30 AWST sleep/consolidation cycle ran and found 0 items to consolidate
(
Last consolidation: 2026-11-14T15:30:51 UTC (0 items)), so consolidation is not compensating. - Nothing else in the estate broke: 2026-11-14 settled 167 rows — 166
completed/ 0failed/ 1runninginexecutions.db, deliveries 4 / 4delivered, and no other cron row went red. This is a memory-path incident only. The one exception is the monitor's own job: it is now the fleet's single non-clean row (last_status: error,failure_streak1, previouslyok/ 0) andcron_incidentswent 0 → 1 unclosed (138 rows total). 2026-11-15 so far: 8 rows — 7completed/ 1failed(this job), 1 deliverydelivered.
Not the known config killer
The previously recorded capture-killer on this host is a stringified list/dict written into
~/.hermes/config.yaml by hermes config set. Checked and clean:
config.yamlmtime 2026-11-13 18:03 — the file was not modified during or before the stall began.- A full YAML walk found no scalar that begins with
[or{;memory.mnemosyne.*parses as proper lists (ignore_patterns: [],skip_contexts,sync_roles).
Open / unverified
- Root cause: not established. No capture-path error lines appear in
agent.logafter 09:33 — the writes simply stop. Candidates to test next: a stuck capture worker or watermark, thepulse/heartbeat traffic that ran 12:03–19:22, session-sidememory_enabled/mnemosyne.profile_isolationinteraction, or the sleep cycle'sconsolidation_claimed_atstate. - ✅ Resolved — the 00:03 escalation DID deliver.
deliveries.dbholds exactly one 2026-11-15 row:2026-11-15T00:03:39.052888+08:00,status: delivered,for_failure: 1— i.e. 3 s after the incident row was raised and 5 s before itsalerted_at(00:03:44.5). The notification path worked; the detection path did not (see the ℹ️-at-12:00 bullet on systems/scheduled-tasks). - Whether capture resumed after this audit's read (the monitor's next slot is the natural check).
Related
- systems/mnemosyne — Mnemosyne cron jobs and config evidence
- systems/memory-backend — memory backend settings
- incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts — the still-open secrets-at-rest page for the same database
- systems/scheduled-tasks — the fleet's execution ledger