- new: systems/phonon-tts.md, incidents/2026-11-14-mnemosyne-capture-stall.md - CT200 inventory re-verified: 64 non-seeder running / 72 total (delta = the TTS trio, no removals) - fleet ledger: 11-14 = 167 rows / 166 completed / 0 failed; first red last_status since 11-13 (Mnemosyne Health Monitor, error, streak 1); cron_incidents 0 -> 1 unclosed (138 rows) - corrections pass: withdrew the 'unidentified extra container' claim; closed the alert-delivery question (delivered 00:03:39, for_failure=1) - updated: index.md, current-state.md, log.md, systems/scheduled-tasks.md, systems/mnemosyne.md, infrastructure/docker-services.md
109 lines
6.3 KiB
Markdown
109 lines
6.3 KiB
Markdown
---
|
||
title: Mnemosyne capture stalled for ~15 hours (2026-11-14 09:33 AWST onward) — health monitor escalated 00:03 AWST 2026-11-15
|
||
type: incident
|
||
status: active
|
||
created: 2026-11-15
|
||
updated: 2026-11-15
|
||
verified_on: 2026-11-15
|
||
review_after: 2026-11-17
|
||
confidence: high
|
||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident]
|
||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db]
|
||
---
|
||
|
||
# Mnemosyne capture stalled for ~15 hours (from 2026-11-14 09:33 AWST)
|
||
|
||
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
|
||
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
|
||
(exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state **`alerted`**, **unclosed**).
|
||
|
||
Found by the **2026-11-14 nightly audit**; not caused by anything in that audit's window (it started
|
||
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
|
||
the same condition was classified informational — see below).
|
||
|
||
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
||
|
||
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
||
**2026-11-14 09:32:59 AWST** (the row's own `timestamp` field reads `2026-11-14T09:32:59`).
|
||
- Newest `episodic_memory.created_at`: **`2026-11-14 01:33:34`** (09:33:34 AWST). **Both** write paths
|
||
stopped inside the same minute.
|
||
- Last 8 `working_memory` writes, all on 11-14 (UTC): `01:01:20, 01:02:39, 01:02:40, 01:03:28,
|
||
01:04:19, 01:32:58, 01:32:59, 01:32:59` — then **nothing for ~15 hours**.
|
||
- Rows by day (UTC `created_at`): **11-09: 11, 11-10: 10, 11-11: 8, 11-12: 37, 11-13: 14, 11-14: 15**.
|
||
So the pipeline was *low-volume but alive* before the stall, and the stall is not a normal quiet stretch.
|
||
- Totals at audit time: **working 1130 / episodic 1249**, DB **27 MB**.
|
||
- Gap: at the 00:03:35 alert the newest capture was **14 h 30 m 36 s** old; at this audit's read
|
||
(~00:40) **≈15 h 7 m**.
|
||
|
||
## The alert (and its mislabelling)
|
||
|
||
```
|
||
🚨 Mnemosyne Health Monitor — exception(s):
|
||
|
||
STALE MEMORY PIPELINE: capture missed: user message 4.64157453 ago (desktop)
|
||
has no working_memory row (last capture 2026-11-14 01:32:59)
|
||
```
|
||
- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`,
|
||
`first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`.
|
||
Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`.
|
||
- ⚠ **The `auth` label looks wrong** — the failure is a *staleness* check, not a credential fault, and
|
||
the page records no auth error text. **Observation, not verified diagnosis.**
|
||
- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a
|
||
~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own
|
||
log line below.
|
||
|
||
## The earlier, quieter reading — 12:00:52 AWST 2026-11-14
|
||
|
||
`~/.hermes/logs/mnemosyne-health-monitor.log` shows the **12:00:52 AWST** run already looking at the
|
||
same hole and calling it benign:
|
||
|
||
```
|
||
ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min
|
||
```
|
||
|
||
That is **9 h 49.5 m** of missing capture, logged as ℹ️ informational (the delay is **negative**, i.e.
|
||
capture sits *behind* the message clock — the sign convention makes a growing gap look small). The
|
||
job exited 0 at 12:00 and 18:03; it only escalated at 00:03 the next morning.
|
||
**So the fault had ~12 hours of lead time that produced no alarm.**
|
||
|
||
## Impact
|
||
|
||
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
|
||
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
|
||
window. Recall over that period will answer from nothing.
|
||
- The **23:30 AWST sleep/consolidation cycle ran and found 0 items** to consolidate
|
||
(`Last consolidation: 2026-11-14T15:30:51 UTC (0 items)`), so consolidation is *not* compensating.
|
||
- Nothing else in the estate broke: **2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` /
|
||
1 `running`** in `executions.db`, deliveries **4 / 4 `delivered`**, and no other cron row went red. This is a
|
||
**memory-path incident only**. The one exception is the monitor's **own** job: it is now the fleet's
|
||
**single non-clean row** (`last_status: error`, `failure_streak` 1, previously `ok` / 0) and
|
||
**`cron_incidents` went 0 → 1 unclosed** (138 rows total). **2026-11-15 so far: 8 rows — 7
|
||
`completed` / 1 `failed`** (this job), 1 delivery `delivered`.
|
||
|
||
## Not the known config killer
|
||
|
||
The previously recorded capture-killer on this host is a **stringified list/dict** written into
|
||
`~/.hermes/config.yaml` by `hermes config set`. Checked and **clean**:
|
||
- `config.yaml` **mtime 2026-11-13 18:03** — the file was not modified during or before the stall began.
|
||
- A full YAML walk found **no** scalar that begins with `[` or `{`; `memory.mnemosyne.*` parses as
|
||
proper lists (`ignore_patterns: []`, `skip_contexts`, `sync_roles`).
|
||
|
||
## Open / unverified
|
||
|
||
- **Root cause: not established.** No capture-path error lines appear in `agent.log` after 09:33 —
|
||
the writes simply stop. Candidates to test next: a stuck capture worker or watermark, the
|
||
`pulse`/heartbeat traffic that ran 12:03–19:22, session-side `memory_enabled` /
|
||
`mnemosyne.profile_isolation` interaction, or the sleep cycle's `consolidation_claimed_at` state.
|
||
- ✅ **Resolved — the 00:03 escalation DID deliver.** `deliveries.db` holds exactly one 2026-11-15 row:
|
||
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
|
||
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
|
||
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
|
||
- Whether capture resumed after this audit's read (the monitor's next slot is the natural check).
|
||
|
||
## Related
|
||
|
||
- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence
|
||
- [[systems/memory-backend]] — memory backend settings
|
||
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] — the still-open secrets-at-rest page for the same database
|
||
- [[systems/scheduled-tasks]] — the fleet's execution ledger
|