Files
agent-estate-wiki/incidents/2026-11-14-mnemosyne-capture-stall.md
T
Tony0410 226767f3ec wiki audit 2026-11-20: three-window catch-up (11-17/11-18/11-19, both prior audit slots died)
- new: incidents/2026-11-18-wiki-audit-credit-outage (audit failed 11-18 + 11-19, deliveries suppressed, incident row alerted_at=None)
- new: incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm (115 consecutive slots, 59 unresolved conflicts vs threshold 50)
- corrected: incidents/2026-11-14-mnemosyne-capture-stall + systems/mnemosyne — freeze ENDED 2026-11-18 19:25 AWST (105h52m, no intervention); 'capture dead' reading was conversational silence (every interactive session captured)
- systems/scheduled-tasks: new Count bullet 11-17 173(171/2) / 11-18 172(153/19) / 11-19 176(78/98); fleet 29 (26/3); non-clean last_status 1->4; unclosed incidents 5->4; deliveries 24 (23 delivered / 1 failed)
- index.md feed + incidents + systems lines; log.md dated entry; current-state.md new section; frontmatter dates -> 2026-11-20
2026-11-20 01:02:44 +08:00

333 lines
23 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
type: incident
status: active
created: 2026-11-15
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
confidence: high
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
---
# Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
> ⚠ **2026-11-20 re-read — the freeze ended by itself at 2026-11-18 19:25 AWST (105 h 52 m total), and every
> interactive session inside the window was captured.** The "capture is dead" reading below is a *quiet estate*,
> not a broken hook. Read the newest re-read section before the earlier readings.
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
(exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state **`alerted`**, **unclosed**).
Found by the **2026-11-14 nightly audit**; not caused by anything in that audit's window (it started
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
the same condition was classified informational — see below).
## Re-read 2026-11-16 00:18 AWST — still open, stall 38 h 46 m, and now **three** unclosed alerts
Six hours later, at the next nightly audit, nothing has moved. The freeze is **total**, the monitor has
fired twice more, and the scheduler has minted further unclosed rows for the same fault.
- **The counts are frozen, not merely low.** `working_memory` **1130** / `episodic_memory` **1249** —
**the same two numbers as every earlier read of the stall** (2026-11-15 ~00:4x and 12:03) — so this is
not a quiet stretch or a degraded trickle. Zero writes in **38 h 46 m**.
- **The whole write path stopped, not just turn capture.** Newest rows remain the three
`[MODEL_REFRESH_PROPOSAL]` maintenance writes at **`09:32:58` / `09:32:59` AWST**; the newest
*user-turn* capture is **`2026-11-14 09:04:19 AWST`**. Maintenance outlived the last turn by ~28 min,
then both stopped inside the same minute.
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows total). All three are `ea79a2107e36`, all
`failure_type: auth`, all `state: alerted`, and **each has `first_seen_at == last_seen_at`** — a single
observation, no `last_seen_at` accumulation, no escalation:
| Incident id | `first_seen_at` | `alerted_at` |
| --- | --- | --- |
| `ea79a2_601037516abd` | `2026-11-15T00:03:35.815373+08:00` | `2026-11-15T00:03:44.559501+08:00` |
| `ea79a2_5f817708701e` | `2026-11-15T12:03:14.928500+08:00` | `2026-11-15T12:03:20.245081+08:00` |
| `ea79a2_75303493d484` | `2026-11-16T00:03:15.418001+08:00` | `2026-11-16T00:03:22.365399+08:00` |
- ⚠ **New mechanism, evidenced: the incident signature embeds the elapsed-hours float.** The three
signatures differ only by `4.64157453` / `4.634034074444445` / `4.643481075555555` — a value **never
equal twice** — so the recurrence never matches the prior row. One durable fault therefore **mints a
fresh unclosed row every slot**, and the estate's unclosed count grows ~1 per 12 h while no single row
ever escalates. This job alone now has **15** `cron_incidents` rows (12 older `unknown`, these 3
`auth`). **Proposed fix — NOT applied** (it changes the monitor's incident path, so it is Anthony's
call): normalise the value out of the signature — round it, or omit it — so the same condition
**re-opens the existing row** instead of fragmenting.
### The monitor's ≈4.64 h figure is NOT the stall — verified against the gate's own source
Every alert in this sequence prints a near-identical elapsed time while the stall grew from 14 h to
38.7 h. That number is not a measurement of the stall. `mnemosyne_freshness_gate.py`'s
`newest_user_message()` selects the newest `messages` row with `role='user'` whose session
`source NOT IN ('cron','subagent')` (the `NON_CONVERSATIONAL` set, line 52) and returns that row's
`source`; the alert prints **`(now − that message) / 3600` h**. So the figure is **the age of the newest
*conversational* message — not the age of the newest capture.**
The newest conversational messages on this box are **Pulse** turns recorded under `source: desktop`
(`Pulse delivery check pXXXX-1 …`, `Pulse morning brief …`), on a **~6 h cadence**. Each alert's
≈4.64 h lands on the then-newest Pulse turn to within about a minute:
| Alert | Elapsed printed | Now ≈ | Newest conversational message it measured |
| --- | --- | --- | --- |
| 2026-11-15 00:03:35 | `4.64157 h` | 00:00:54 | **2026-11-14 19:22:30** (desktop / Pulse) |
| 2026-11-15 12:03:14 | `4.63403 h` | 12:01:25 | **2026-11-15 07:23:25** (desktop / Pulse) |
| 2026-11-16 00:03:15 | `4.64348 h` | 00:01:33 | **2026-11-15 19:22:59** (desktop / Pulse) |
**Why they all read ≈4.64:** the `0 */12` slots happen to fire ~4.6 h after the most recent Pulse turn.
The near-constancy is a **cadence coincidence** and must **not** be read as "the stall is only ~4.6 h" —
the stall is **38 h 46 m**. Same trap class as the naive-UTC note below: a number shaped like a duration
that is not one.
**Positive proof the capture path is dead rather than merely idle:** the identical class of message *was*
captured before the stall — the 2026-11-14 00:09:12 run logged
`last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`. Four 2026-11-15 Pulse turns
(`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have **no capture row at all**.
### The consolidator is alive and every cycle is a deliberate no-op
Latest `consolidation_log` row **4696** at `2026-11-15T16:14:14.934306+00:00` (= **2026-11-16 00:14:14
AWST**), `items_consolidated: 0`, `session_id: auto-consolidate-maintenance`; cadence ~31 min
(`15:12:13` → `15:43:08` → `16:14:14` UTC). **New evidence:** the row's `summary_preview` reads
**`No-op consolidation cycle — timestamp updated for watchdog`** — the cycle **touches its own timestamp
to satisfy an age test**, so a zero-item streak can run indefinitely while every age-based gate reads
"fresh". The streak is now **~74 consecutive zero-item cycles**, opening
`2026-11-14T02:04:42+00:00` (10:04:42 AWST). **A zero-item-streak threshold — not an age threshold — is
what would have caught this at 10:04 on 11-14.**
### Adjacent fault in the same runtime — the watchdog lost a slot (2026-11-15 11:31)
`96eb95ce12f2` (**Mnemosyne Watchdog**, every 15 min) **failed** on its `11:31:20.699` slot,
`finished_at 2026-11-15T11:49:10.534610` — **17 m 49.8 s**, exit 1:
```
runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds
```
A **600 s timeout inside a 15-minute slot** overran two slots: the `11:45` slot was **skipped** (no row
at all) and the **next slot, `12:01:25.120952`, completed** (`→ 12:02:02.657231`), so it self-recovered
with one slot lost. This is a **new signature** for this job — the August and October occurrences had the
guard quoting *another* job's error text. Across 11-15 → 11-16 it is the only `failed` row of its 97.
Alert delivered (`deliveries.db` `11:49:06.052341`, `for_failure: 1`, `delivered`). **Adjacency, not
proof of common cause** — but the guard and the capture path share the same runtime, so a guard that
times out at 600 s belongs next to this stall rather than dismissed.
## Re-read 2026-11-17 00:27 AWST — stall **62 h 54 m**, **five** unclosed alerts, zero-item streak **120**
Third nightly read, sixth week of the same numbers changing only their counters. **The freeze is still
total.**
- **Counts identical for the fifth consecutive read:** `working_memory` **1130** / `episodic_memory`
**1249**; newest rows still `2026-11-14 09:32:59` AWST (working) and `09:33:34` AWST (episodic) —
**62 h 54 m** with **zero** writes.
- **`cron_incidents` 3 → 5 unclosed** (141 → **143** rows total). Both new rows are `ea79a2107e36`,
`failure_type: auth`, `state: alerted`, and again `first_seen_at == last_seen_at`:
| Incident id | `first_seen_at` | `alerted_at` |
| --- | --- | --- |
| `ea79a2_915d004a108a` | `2026-11-16T12:02:06.679532+08:00` | `2026-11-16T12:02:12.472265+08:00` |
| `ea79a2_0fab8d43db75` | `2026-11-17T00:03:37.624640+08:00` | `2026-11-17T00:03:46.151392+08:00` |
The signature-fragmentation mechanism below is holding exactly as described: **one fresh row per
firing, no escalation**, so the estate's unclosed count now grows **~2 per day** with nothing accumulating
on any single row.
- **The monitor is on a 5-slot failure streak** (`jobs.json`: `ea79a2107e36`, `last_status: error`,
`failure_streak: 5`). 11-16's ledger holds **2 failed rows of 175, both this job**
(00:01:33 → 00:03:22, 12:01:18 → 12:02:12); 11-17 so far 1 failed of 7 (00:01:39 → 00:03:46).
Every failure is `exit 1` with `STALE MEMORY PIPELINE: capture missed: us…` — the same gate.
- **The consolidator is alive and deliberately doing nothing.** `consolidation_log` latest row **4742**
at `2026-11-16T16:00:41.922076+00:00` (= 00:00:41 AWST), `items_consolidated: 0`, summary *"No-op
consolidation cycle — timestamp updated for watchdog"*. Last non-zero row: **4622** at
`2026-11-14T09:33:34` → **120 consecutive zero-item cycles** (was 74 at the 11-16 read). The cycles
still fire ~every 31 min; there is simply **nothing new to consolidate**, which is the memory-side
reason the stall cannot self-heal.
- **Still no cause, still no fix, still `status: active`.** The 11-16 read's two conclusions stand: the
failure is **total**, not a trickle, and it affects **all** write paths, not just turn capture.
`review_after` 2026-11-18 — this page is re-read at the next audit, and the monitor keeps alerting.
## Re-read 2026-11-20 00:5x AWST — the freeze ended by itself after **105 h 52 m**, and the "capture is dead" reading was wrong
**The capture freeze is over. The alarm is not.** Live reads of `mnemosyne.db` at 2026-11-20 00:0x–00:5x AWST:
| counter | 2026-11-17 00:27 | 2026-11-20 00:5x |
| --- | --- | --- |
| `working_memory` | 1130 | **1131** |
| `episodic_memory` | 1249 | **1250** |
| newest write | `2026-11-14 09:32:59` AWST | **`2026-11-18 19:25:03` AWST** (stored UTC `11:25:03`) |
The break coincides with nothing: no config edit, no restart and no repair is recorded against the capture path
in the window — the only nearby state change is the 11-18 19:18 desktop session itself. The single new
`working_memory` row carries `session_id: hermes_20261118_191808_5c9e08` and its content begins
`[USER] You are helping me migrate contex…`, which identifies the captured turn (the `desktop` session
**`20261118_191808_5c9e08`**, *"Helping migrate context AI assistant"*, 9 messages, started 19:18 AWST).
**Stall duration: 105 h 52 m** (2026-11-14 09:33 → 2026-11-18 19:25 AWST).
⚠ **The correction that matters — the window contained almost no conversation to capture.** `state.db` lists
**three** non-cron sessions since 2026-11-14 00:00 AWST: two `desktop` sessions on **11-14** (**08:35**, 40
messages; **08:52**, 41 messages — both *before* the 09:33 boundary), then **nothing** until the **11-18 19:18**
session, whose turn is the row that ends the freeze. Every interactive session in the window was captured, so
the earlier figures on this page (`0 writes in 38 h 46 m`, `62 h 54 m`) are real but measure **a quiet estate**,
not a broken hook: `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message
whose session source is not `cron`/`subagent`, so **silence and failure read identically**. That ambiguity — not
a broken capture path — is the durable defect on this page.
**The monitor is still failing on that same metric, after the break.** `ea79a2107e36` slots since 11-18:
`11-18 00:01:28` failed · `11-18 12:01:13` failed · **`11-19 00:01:34` `completed`** (the only clean slot since
11-17) · `11-19 12:01:17` failed · `11-20 00:01:22` failed — every failure `Script exited with code 1 / 🚨
Mnemosyne Health Monitor — exception(s): STALE MEMORY PIPELINE: capture …`, i.e. **the same signature firing on
a healthy pipeline** (`failure_streak: 2`; incidents `ea79a2_f2be2db3bfce` at 2026-11-19 12:01:58 and
`ea79a2_aae4d914f3c9` at 2026-11-20 00:03:36, both `alerted`, both `failure_type: auth` — the scheduler's
classifier, not a credential fault). The `11-19 00:01` pass and the `12:01` failure bracket the threshold: the
metric is *age of the newest captured conversational turn*, which only grows while the estate is silent.
**The other half of the original finding is unchanged, and larger.** The consolidator still consolidates zero
items: newest `consolidation_log` row **`2026-11-19T16:20:14.587267+00:00`** (= 2026-11-20 00:20 AWST, so it is
running on its ~31-minute cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle
— timestamp updated for watchdog"`. **199 of the last 200 cycles are zero-item** (was **120** at the 11-17 read
— consistent with ~31 min cadence over the three days). The zero-item-streak threshold remains **proposed, not
applied**.
**Unclosed incidents: 5 → 4** (one of the rows present at the 11-17 read has since closed). Current four, none
closed automatically: `cbfe43_fa12cca1ad4b` (*auth*, 2026-11-18 00:01:42, `alerted_at: None` — the wiki audit's
silent skip, see [[incidents/2026-11-18-wiki-audit-credit-outage]]), `96eb95_98ade422d195` (*script*,
2026-11-18 20:01:15 → last_seen 2026-11-20 00:31:16 — see
[[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]), `ea79a2_f2be2db3bfce` (2026-11-19
12:01:58) and `ea79a2_aae4d914f3c9` (2026-11-20 00:03:36).
**Standing open question, restated:** not *"why did capture stop"* but **"why does the gate treat conversational
silence as a fault?"** Until that is fixed, this page will keep re-opening on quiet days.
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
**2026-11-14 09:32:59 AWST** (the row's own `timestamp` field reads `2026-11-14T09:32:59`).
- Newest `episodic_memory.created_at`: **`2026-11-14 01:33:34`** (09:33:34 AWST). **Both** write paths
stopped inside the same minute.
- Last 8 `working_memory` writes, all on 11-14 (UTC): `01:01:20, 01:02:39, 01:02:40, 01:03:28,
01:04:19, 01:32:58, 01:32:59, 01:32:59` — then **nothing for ~15 hours**.
- Rows by day (UTC `created_at`): **11-09: 11, 11-10: 10, 11-11: 8, 11-12: 37, 11-13: 14, 11-14: 15**.
So the pipeline was *low-volume but alive* before the stall, and the stall is not a normal quiet stretch.
- Totals at audit time: **working 1130 / episodic 1249**, DB **27 MB**.
- Gap: at the 00:03:35 alert the newest capture was **14 h 30 m 36 s** old; at this audit's read
(~00:40) **≈15 h 7 m**.
## The alert (and its mislabelling)
```
🚨 Mnemosyne Health Monitor — exception(s):
STALE MEMORY PIPELINE: capture missed: user message 4.64157453 ago (desktop)
has no working_memory row (last capture 2026-11-14 01:32:59)
```
- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`,
`first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`.
Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`.
- ⚠ **The `auth` label describes the classifier, not the fault — now evidenced, not just suspected.** The
failure is a *staleness* check and the page records no auth error text. This job's **twelve earlier
`cron_incidents` rows** read `agent` ×3, `timeout` ×1, `unknown` ×8 (`ea79a2_649d8ac46e8a`,
`_f17b45759db0`, `_573d17e10fc1`, `_6fb70b032305`, `_54d6b1f0f58a`, `_747b27556fd0`, `_c3ab2d784ba3`,
`_a4661554a3c2`, `_d93c3acdc4f2`, `_9197df32320b`, `_111e67a3d83c`, `_f317fcd843ae`), and the estate-wide
table is **53 `unknown` / 47 `agent` / 15 `timeout` / 14 `script` / 5 `auth`**. `agent` is also impossible
here: the job runs `no_agent` (plain script). Treat the field as opaque routing metadata.
- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a
~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own
log line below.
## What the earlier slots actually saw — correction, re-derived 2026-11-15 01:2x AWST
**An earlier revision of this page got this backwards.** It claimed the **12:00:52 AWST** run had
"already seen the same hole and called it benign" (`capture delay -589.5 min`), giving the estate ~12 h
of unalarmed lead time, and blamed a "sign, not magnitude" threshold. **Both claims are withdrawn.** The
`-589.5 min` line is not the 12:00 run's — it sits inside the **00:03:35 run's own output block**.
Verified by line number in `~/.hermes/logs/mnemosyne-health-monitor.log` (**1103 lines**):
| line | content |
| --- | --- |
| 884 | `=== 2026-11-14 00:09:12 AWST health monitor (no_agent) ===` |
| 900 | `ℹ️ last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min` |
| 957 | `=== 2026-11-14 12:01:52 AWST health monitor (no_agent) ===` |
| 973 | `ℹ️ last user msg 2026-11-14 01:03 UTC (ios) capture delay 29.4 min` |
| 1030 | `=== 2026-11-15 00:03:35 AWST health monitor (no_agent) ===` |
| 1046 | `ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min` |
| 1047 | `❌ STALE MEMORY PIPELINE: …` |
Two further corrections to the earlier revision:
- **There was no 18:03 run.** The job's schedule is `0 */12 * * *`; the only 2026-11-14 slots are
00:09:12 and 12:01:52 (both exit 0, delivery `suppressed`). The "exited 0 at 12:00 and 18:03"
sentence is withdrawn — 18:03 belongs to some other job.
- **The detection chain was not broken.** `mnemosyne_freshness_gate.py` fires on **magnitude**
(`delay_min < -CAPTURE_LAG_MIN`, `MNEMOSYNE_CAPTURE_LAG_MIN` default **120** min) — there is no
sign-only test to blame. The 12:00:52 measurement of **+29.4 min** is the delay of the newest message
*that had been captured* (the 09:03 AWST `ios` turn, captured 09:32:59). The gate computes against the
newest **uncaptured** message and only reports a problem when one exists; with the conversation idle
from 09:03 to 19:22, every user message still had a capture row, so there was **nothing to detect**.
The first message to postdate the stall was the **19:22 AWST desktop** turn (`11:22 UTC`, the same
message the 00:03 alert names); the next scheduled slot after it was 00:03:35, which escalated
(`-589.5 min` = 09:32:59 − 19:22).
**Corrected statement of the gap:** the pipeline died at **09:32:59 AWST**; it became *detectable* at
**19:22 AWST** (first uncaptured message) and was *detected* at **00:03:35** by the next slot — a
detection latency of 4 h 41 m after detectability, driven by cadence, not by a broken threshold.
**What was genuinely unread sits in the consolidation counters:** the cycles ran every ~31 min and
returned **0 items for 29 consecutive runs** (streak opening `2026-11-14T02:04:42+00:00` = **10:04:42
AWST**; last productive cycles 4622/4621/4620 at `09:33:34` / `09:33:16` / `09:01:40`). Every run
printed `Last consolidation: … (0 items)` — including the "healthy" 12:00:52 one — but the gate tests
consolidation **age** only (`> 36 h`, default), never the item count. **An empty-but-current cycle is
the loudest signal available and the only one nothing watches.**
## Impact
> **Magnitude superseded 2026-11-16 00:18 AWST: 38 h 46 m uncaptured** (was ~15 h), with **two more
> unclosed alerts** and four 2026-11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`)
> added to the uncaptured set. The ~15 h figures below stand as the 2026-11-15 readings.
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
window. Recall over that period will answer from nothing.
- The **23:30 AWST sleep/consolidation cycle ran and found 0 items** to consolidate
(`Last consolidation: 2026-11-14T15:30:51 UTC (0 items)`), so consolidation is *not* compensating.
- Nothing else in the estate broke: **2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` /
1 `running`** in `executions.db`, deliveries **4 / 4 `delivered`**, and no other cron row went red. This is a
**memory-path incident only**. The one exception is the monitor's **own** job: it is now the fleet's
**single non-clean row** (`last_status: error`, `failure_streak` 1, previously `ok` / 0) and
**`cron_incidents` went 0 → 1 unclosed** (138 rows total). **2026-11-15 so far: 8 rows — 7
`completed` / 1 `failed`** (this job), 1 delivery `delivered`.
## Not the known config killer
The previously recorded capture-killer on this host is a **stringified list/dict** written into
`~/.hermes/config.yaml` by `hermes config set`. Checked and **clean**:
- `config.yaml` **mtime 2026-11-13 18:03** — the file was not modified during or before the stall began.
- A full YAML walk found **no** scalar that begins with `[` or `{`; `memory.mnemosyne.*` parses as
proper lists (`ignore_patterns: []`, `skip_contexts`, `sync_roles`).
## Open / unverified
- **Root cause: not established.** No capture-path error lines appear in `agent.log` after 09:33 —
the writes simply stop. Candidates to test next: a stuck capture worker or watermark, the
`pulse`/heartbeat traffic that ran 12:03–19:22, session-side `memory_enabled` /
`mnemosyne.profile_isolation` interaction, or the sleep cycle's `consolidation_claimed_at` state.
- ✅ **Resolved — the 00:03 escalation DID deliver.** `deliveries.db` holds exactly one 2026-11-15 row:
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
- ✅ **Answered 2026-11-16 00:18 — capture did NOT resume.** Two further monitor slots failed on the
same condition (`12:03:14` and `2026-11-16 00:03:15`) and the row counts are unchanged (`working`
1130 / `episodic` 1249). Closed as *no*; the underlying cause remains open.
- **The estate's only automated alarm is understating this incident.** The alert's ≈4.64 h is a cadence
artefact, not a stall duration — anyone reading the alert text alone would conclude the pipeline is
~4.6 h behind when it is 38 h 46 m. A stall-duration figure (newest capture vs now) would fix it.
## Related
- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence
- [[systems/memory-backend]] — memory backend settings
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] — the still-open secrets-at-rest page for the same database
- [[systems/scheduled-tasks]] — the fleet's execution ledger