16 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mnemosyne capture stalled for 38+ hours (2026-11-14 09:33 AWST onward) — three unclosed alerts, freeze total on re-read | incident | active | 2026-11-15 | 2026-11-16 | 2026-11-16 | 2026-11-18 | high |
|
|
Mnemosyne capture stalled for 38+ hours (from 2026-11-14 09:33 AWST)
Open. Nothing has been written to Mnemosyne's capture tables since 2026-11-14 ~09:33 AWST,
and on 2026-11-15 00:03:35 the Mnemosyne Health Monitor (ea79a2107e36) failed on it
(exit 1, incident ea79a2_601037516abd, failure_type: auth, state alerted, unclosed).
Found by the 2026-11-14 nightly audit; not caused by anything in that audit's window (it started the previous morning and simply had not been noticed, because the monitor's own earlier reading of the same condition was classified informational — see below).
Re-read 2026-11-16 00:18 AWST — still open, stall 38 h 46 m, and now three unclosed alerts
Six hours later, at the next nightly audit, nothing has moved. The freeze is total, the monitor has fired twice more, and the scheduler has minted further unclosed rows for the same fault.
-
The counts are frozen, not merely low.
working_memory1130 /episodic_memory1249 — the same two numbers as every earlier read of the stall (2026-11-15 ~00:4x and 12:03) — so this is not a quiet stretch or a degraded trickle. Zero writes in 38 h 46 m. -
The whole write path stopped, not just turn capture. Newest rows remain the three
[MODEL_REFRESH_PROPOSAL]maintenance writes at09:32:58/09:32:59AWST; the newest user-turn capture is2026-11-14 09:04:19 AWST. Maintenance outlived the last turn by ~28 min, then both stopped inside the same minute. -
cron_incidents1 → 3 unclosed (138 → 141 rows total). All three areea79a2107e36, allfailure_type: auth, allstate: alerted, and each hasfirst_seen_at == last_seen_at— a single observation, nolast_seen_ataccumulation, no escalation:Incident id first_seen_atalerted_atea79a2_601037516abd2026-11-15T00:03:35.815373+08:002026-11-15T00:03:44.559501+08:00ea79a2_5f817708701e2026-11-15T12:03:14.928500+08:002026-11-15T12:03:20.245081+08:00ea79a2_75303493d4842026-11-16T00:03:15.418001+08:002026-11-16T00:03:22.365399+08:00 -
⚠ New mechanism, evidenced: the incident signature embeds the elapsed-hours float. The three signatures differ only by
4.64157453/4.634034074444445/4.643481075555555— a value never equal twice — so the recurrence never matches the prior row. One durable fault therefore mints a fresh unclosed row every slot, and the estate's unclosed count grows ~1 per 12 h while no single row ever escalates. This job alone now has 15cron_incidentsrows (12 olderunknown, these 3auth). Proposed fix — NOT applied (it changes the monitor's incident path, so it is Anthony's call): normalise the value out of the signature — round it, or omit it — so the same condition re-opens the existing row instead of fragmenting.
The monitor's ≈4.64 h figure is NOT the stall — verified against the gate's own source
Every alert in this sequence prints a near-identical elapsed time while the stall grew from 14 h to
38.7 h. That number is not a measurement of the stall. mnemosyne_freshness_gate.py's
newest_user_message() selects the newest messages row with role='user' whose session
source NOT IN ('cron','subagent') (the NON_CONVERSATIONAL set, line 52) and returns that row's
source; the alert prints (now − that message) / 3600 h. So the figure is the age of the newest
conversational message — not the age of the newest capture.
The newest conversational messages on this box are Pulse turns recorded under source: desktop
(Pulse delivery check pXXXX-1 …, Pulse morning brief …), on a ~6 h cadence. Each alert's
≈4.64 h lands on the then-newest Pulse turn to within about a minute:
| Alert | Elapsed printed | Now ≈ | Newest conversational message it measured |
|---|---|---|---|
| 2026-11-15 00:03:35 | 4.64157 h |
00:00:54 | 2026-11-14 19:22:30 (desktop / Pulse) |
| 2026-11-15 12:03:14 | 4.63403 h |
12:01:25 | 2026-11-15 07:23:25 (desktop / Pulse) |
| 2026-11-16 00:03:15 | 4.64348 h |
00:01:33 | 2026-11-15 19:22:59 (desktop / Pulse) |
Why they all read ≈4.64: the 0 */12 slots happen to fire ~4.6 h after the most recent Pulse turn.
The near-constancy is a cadence coincidence and must not be read as "the stall is only ~4.6 h" —
the stall is 38 h 46 m. Same trap class as the naive-UTC note below: a number shaped like a duration
that is not one.
Positive proof the capture path is dead rather than merely idle: the identical class of message was
captured before the stall — the 2026-11-14 00:09:12 run logged
last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min. Four 2026-11-15 Pulse turns
(07:00:43, 07:23:25, 13:23:00, 19:22:59) now have no capture row at all.
The consolidator is alive and every cycle is a deliberate no-op
Latest consolidation_log row 4696 at 2026-11-15T16:14:14.934306+00:00 (= 2026-11-16 00:14:14
AWST), items_consolidated: 0, session_id: auto-consolidate-maintenance; cadence ~31 min
(15:12:13 → 15:43:08 → 16:14:14 UTC). New evidence: the row's summary_preview reads
No-op consolidation cycle — timestamp updated for watchdog — the cycle touches its own timestamp
to satisfy an age test, so a zero-item streak can run indefinitely while every age-based gate reads
"fresh". The streak is now ~74 consecutive zero-item cycles, opening
2026-11-14T02:04:42+00:00 (10:04:42 AWST). A zero-item-streak threshold — not an age threshold — is
what would have caught this at 10:04 on 11-14.
Adjacent fault in the same runtime — the watchdog lost a slot (2026-11-15 11:31)
96eb95ce12f2 (Mnemosyne Watchdog, every 15 min) failed on its 11:31:20.699 slot,
finished_at 2026-11-15T11:49:10.534610 — 17 m 49.8 s, exit 1:
runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds
A 600 s timeout inside a 15-minute slot overran two slots: the 11:45 slot was skipped (no row
at all) and the next slot, 12:01:25.120952, completed (→ 12:02:02.657231), so it self-recovered
with one slot lost. This is a new signature for this job — the August and October occurrences had the
guard quoting another job's error text. Across 11-15 → 11-16 it is the only failed row of its 97.
Alert delivered (deliveries.db 11:49:06.052341, for_failure: 1, delivered). Adjacency, not
proof of common cause — but the guard and the capture path share the same runtime, so a guard that
times out at 600 s belongs next to this stall rather than dismissed.
Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
- Newest
working_memory.created_at:2026-11-14 01:32:59— stored UTC, i.e. 2026-11-14 09:32:59 AWST (the row's owntimestampfield reads2026-11-14T09:32:59). - Newest
episodic_memory.created_at:2026-11-14 01:33:34(09:33:34 AWST). Both write paths stopped inside the same minute. - Last 8
working_memorywrites, all on 11-14 (UTC):01:01:20, 01:02:39, 01:02:40, 01:03:28, 01:04:19, 01:32:58, 01:32:59, 01:32:59— then nothing for ~15 hours. - Rows by day (UTC
created_at): 11-09: 11, 11-10: 10, 11-11: 8, 11-12: 37, 11-13: 14, 11-14: 15. So the pipeline was low-volume but alive before the stall, and the stall is not a normal quiet stretch. - Totals at audit time: working 1130 / episodic 1249, DB 27 MB.
- Gap: at the 00:03:35 alert the newest capture was 14 h 30 m 36 s old; at this audit's read (~00:40) ≈15 h 7 m.
The alert (and its mislabelling)
🚨 Mnemosyne Health Monitor — exception(s):
STALE MEMORY PIPELINE: capture missed: user message 4.64157453 ago (desktop)
has no working_memory row (last capture 2026-11-14 01:32:59)
cron_incidentsrowea79a2_601037516abd:state: alerted,failure_type: auth,first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00,closed_at: null. Output file:~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md.- ⚠ The
authlabel describes the classifier, not the fault — now evidenced, not just suspected. The failure is a staleness check and the page records no auth error text. This job's twelve earliercron_incidentsrows readagent×3,timeout×1,unknown×8 (ea79a2_649d8ac46e8a,_f17b45759db0,_573d17e10fc1,_6fb70b032305,_54d6b1f0f58a,_747b27556fd0,_c3ab2d784ba3,_a4661554a3c2,_d93c3acdc4f2,_9197df32320b,_111e67a3d83c,_f317fcd843ae), and the estate-wide table is 53unknown/ 47agent/ 15timeout/ 14script/ 5auth.agentis also impossible here: the job runsno_agent(plain script). Treat the field as opaque routing metadata. - ⚠ The alert text prints the raw stored value (
01:32:59, UTC) with no timezone, which reads as a ~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own log line below.
What the earlier slots actually saw — correction, re-derived 2026-11-15 01:2x AWST
An earlier revision of this page got this backwards. It claimed the 12:00:52 AWST run had
"already seen the same hole and called it benign" (capture delay -589.5 min), giving the estate ~12 h
of unalarmed lead time, and blamed a "sign, not magnitude" threshold. Both claims are withdrawn. The
-589.5 min line is not the 12:00 run's — it sits inside the 00:03:35 run's own output block.
Verified by line number in ~/.hermes/logs/mnemosyne-health-monitor.log (1103 lines):
| line | content |
|---|---|
| 884 | === 2026-11-14 00:09:12 AWST health monitor (no_agent) === |
| 900 | ℹ️ last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min |
| 957 | === 2026-11-14 12:01:52 AWST health monitor (no_agent) === |
| 973 | ℹ️ last user msg 2026-11-14 01:03 UTC (ios) capture delay 29.4 min |
| 1030 | === 2026-11-15 00:03:35 AWST health monitor (no_agent) === |
| 1046 | ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min |
| 1047 | ❌ STALE MEMORY PIPELINE: … |
Two further corrections to the earlier revision:
- There was no 18:03 run. The job's schedule is
0 */12 * * *; the only 2026-11-14 slots are 00:09:12 and 12:01:52 (both exit 0, deliverysuppressed). The "exited 0 at 12:00 and 18:03" sentence is withdrawn — 18:03 belongs to some other job. - The detection chain was not broken.
mnemosyne_freshness_gate.pyfires on magnitude (delay_min < -CAPTURE_LAG_MIN,MNEMOSYNE_CAPTURE_LAG_MINdefault 120 min) — there is no sign-only test to blame. The 12:00:52 measurement of +29.4 min is the delay of the newest message that had been captured (the 09:03 AWSTiosturn, captured 09:32:59). The gate computes against the newest uncaptured message and only reports a problem when one exists; with the conversation idle from 09:03 to 19:22, every user message still had a capture row, so there was nothing to detect. The first message to postdate the stall was the 19:22 AWST desktop turn (11:22 UTC, the same message the 00:03 alert names); the next scheduled slot after it was 00:03:35, which escalated (-589.5 min= 09:32:59 − 19:22).
Corrected statement of the gap: the pipeline died at 09:32:59 AWST; it became detectable at
19:22 AWST (first uncaptured message) and was detected at 00:03:35 by the next slot — a
detection latency of 4 h 41 m after detectability, driven by cadence, not by a broken threshold.
What was genuinely unread sits in the consolidation counters: the cycles ran every ~31 min and
returned 0 items for 29 consecutive runs (streak opening 2026-11-14T02:04:42+00:00 = 10:04:42
AWST; last productive cycles 4622/4621/4620 at 09:33:34 / 09:33:16 / 09:01:40). Every run
printed Last consolidation: … (0 items) — including the "healthy" 12:00:52 one — but the gate tests
consolidation age only (> 36 h, default), never the item count. An empty-but-current cycle is
the loudest signal available and the only one nothing watches.
Impact
Magnitude superseded 2026-11-16 00:18 AWST: 38 h 46 m uncaptured (was ~15 h), with two more unclosed alerts and four 2026-11-15 Pulse turns (
07:00:43,07:23:25,13:23:00,19:22:59) added to the uncaptured set. The ~15 h figures below stand as the 2026-11-15 readings.
- ~15 hours of context is not in memory: the desktop/Bot Chat session that ran to 19:22 AWST (the last user message the monitor could see), all Telegram traffic, and every cron session in the window. Recall over that period will answer from nothing.
- The 23:30 AWST sleep/consolidation cycle ran and found 0 items to consolidate
(
Last consolidation: 2026-11-14T15:30:51 UTC (0 items)), so consolidation is not compensating. - Nothing else in the estate broke: 2026-11-14 settled 167 rows — 166
completed/ 0failed/ 1runninginexecutions.db, deliveries 4 / 4delivered, and no other cron row went red. This is a memory-path incident only. The one exception is the monitor's own job: it is now the fleet's single non-clean row (last_status: error,failure_streak1, previouslyok/ 0) andcron_incidentswent 0 → 1 unclosed (138 rows total). 2026-11-15 so far: 8 rows — 7completed/ 1failed(this job), 1 deliverydelivered.
Not the known config killer
The previously recorded capture-killer on this host is a stringified list/dict written into
~/.hermes/config.yaml by hermes config set. Checked and clean:
config.yamlmtime 2026-11-13 18:03 — the file was not modified during or before the stall began.- A full YAML walk found no scalar that begins with
[or{;memory.mnemosyne.*parses as proper lists (ignore_patterns: [],skip_contexts,sync_roles).
Open / unverified
- Root cause: not established. No capture-path error lines appear in
agent.logafter 09:33 — the writes simply stop. Candidates to test next: a stuck capture worker or watermark, thepulse/heartbeat traffic that ran 12:03–19:22, session-sidememory_enabled/mnemosyne.profile_isolationinteraction, or the sleep cycle'sconsolidation_claimed_atstate. - ✅ Resolved — the 00:03 escalation DID deliver.
deliveries.dbholds exactly one 2026-11-15 row:2026-11-15T00:03:39.052888+08:00,status: delivered,for_failure: 1— i.e. 3 s after the incident row was raised and 5 s before itsalerted_at(00:03:44.5). The notification path worked; the detection path did not (see the ℹ️-at-12:00 bullet on systems/scheduled-tasks). - ✅ Answered 2026-11-16 00:18 — capture did NOT resume. Two further monitor slots failed on the
same condition (
12:03:14and2026-11-16 00:03:15) and the row counts are unchanged (working1130 /episodic1249). Closed as no; the underlying cause remains open. - The estate's only automated alarm is understating this incident. The alert's ≈4.64 h is a cadence artefact, not a stall duration — anyone reading the alert text alone would conclude the pipeline is ~4.6 h behind when it is 38 h 46 m. A stall-duration figure (newest capture vs now) would fix it.
Related
- systems/mnemosyne — Mnemosyne cron jobs and config evidence
- systems/memory-backend — memory backend settings
- incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts — the still-open secrets-at-rest page for the same database
- systems/scheduled-tasks — the fleet's execution ledger