- new: incidents/2026-11-18-wiki-audit-credit-outage (audit failed 11-18 + 11-19, deliveries suppressed, incident row alerted_at=None) - new: incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm (115 consecutive slots, 59 unresolved conflicts vs threshold 50) - corrected: incidents/2026-11-14-mnemosyne-capture-stall + systems/mnemosyne — freeze ENDED 2026-11-18 19:25 AWST (105h52m, no intervention); 'capture dead' reading was conversational silence (every interactive session captured) - systems/scheduled-tasks: new Count bullet 11-17 173(171/2) / 11-18 172(153/19) / 11-19 176(78/98); fleet 29 (26/3); non-clean last_status 1->4; unclosed incidents 5->4; deliveries 24 (23 delivered / 1 failed) - index.md feed + incidents + systems lines; log.md dated entry; current-state.md new section; frontmatter dates -> 2026-11-20
333 lines
23 KiB
Markdown
333 lines
23 KiB
Markdown
---
|
||
title: Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
|
||
type: incident
|
||
status: active
|
||
created: 2026-11-15
|
||
updated: 2026-11-20
|
||
verified_on: 2026-11-20
|
||
review_after: 2026-11-21
|
||
confidence: high
|
||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
|
||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
|
||
---
|
||
|
||
# Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
|
||
|
||
> ⚠ **2026-11-20 re-read — the freeze ended by itself at 2026-11-18 19:25 AWST (105 h 52 m total), and every
|
||
> interactive session inside the window was captured.** The "capture is dead" reading below is a *quiet estate*,
|
||
> not a broken hook. Read the newest re-read section before the earlier readings.
|
||
|
||
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
|
||
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
|
||
(exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state **`alerted`**, **unclosed**).
|
||
|
||
Found by the **2026-11-14 nightly audit**; not caused by anything in that audit's window (it started
|
||
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
|
||
the same condition was classified informational — see below).
|
||
|
||
## Re-read 2026-11-16 00:18 AWST — still open, stall 38 h 46 m, and now **three** unclosed alerts
|
||
|
||
Six hours later, at the next nightly audit, nothing has moved. The freeze is **total**, the monitor has
|
||
fired twice more, and the scheduler has minted further unclosed rows for the same fault.
|
||
|
||
- **The counts are frozen, not merely low.** `working_memory` **1130** / `episodic_memory` **1249** —
|
||
**the same two numbers as every earlier read of the stall** (2026-11-15 ~00:4x and 12:03) — so this is
|
||
not a quiet stretch or a degraded trickle. Zero writes in **38 h 46 m**.
|
||
- **The whole write path stopped, not just turn capture.** Newest rows remain the three
|
||
`[MODEL_REFRESH_PROPOSAL]` maintenance writes at **`09:32:58` / `09:32:59` AWST**; the newest
|
||
*user-turn* capture is **`2026-11-14 09:04:19 AWST`**. Maintenance outlived the last turn by ~28 min,
|
||
then both stopped inside the same minute.
|
||
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows total). All three are `ea79a2107e36`, all
|
||
`failure_type: auth`, all `state: alerted`, and **each has `first_seen_at == last_seen_at`** — a single
|
||
observation, no `last_seen_at` accumulation, no escalation:
|
||
|
||
| Incident id | `first_seen_at` | `alerted_at` |
|
||
| --- | --- | --- |
|
||
| `ea79a2_601037516abd` | `2026-11-15T00:03:35.815373+08:00` | `2026-11-15T00:03:44.559501+08:00` |
|
||
| `ea79a2_5f817708701e` | `2026-11-15T12:03:14.928500+08:00` | `2026-11-15T12:03:20.245081+08:00` |
|
||
| `ea79a2_75303493d484` | `2026-11-16T00:03:15.418001+08:00` | `2026-11-16T00:03:22.365399+08:00` |
|
||
|
||
- ⚠ **New mechanism, evidenced: the incident signature embeds the elapsed-hours float.** The three
|
||
signatures differ only by `4.64157453` / `4.634034074444445` / `4.643481075555555` — a value **never
|
||
equal twice** — so the recurrence never matches the prior row. One durable fault therefore **mints a
|
||
fresh unclosed row every slot**, and the estate's unclosed count grows ~1 per 12 h while no single row
|
||
ever escalates. This job alone now has **15** `cron_incidents` rows (12 older `unknown`, these 3
|
||
`auth`). **Proposed fix — NOT applied** (it changes the monitor's incident path, so it is Anthony's
|
||
call): normalise the value out of the signature — round it, or omit it — so the same condition
|
||
**re-opens the existing row** instead of fragmenting.
|
||
|
||
### The monitor's ≈4.64 h figure is NOT the stall — verified against the gate's own source
|
||
|
||
Every alert in this sequence prints a near-identical elapsed time while the stall grew from 14 h to
|
||
38.7 h. That number is not a measurement of the stall. `mnemosyne_freshness_gate.py`'s
|
||
`newest_user_message()` selects the newest `messages` row with `role='user'` whose session
|
||
`source NOT IN ('cron','subagent')` (the `NON_CONVERSATIONAL` set, line 52) and returns that row's
|
||
`source`; the alert prints **`(now − that message) / 3600` h**. So the figure is **the age of the newest
|
||
*conversational* message — not the age of the newest capture.**
|
||
|
||
The newest conversational messages on this box are **Pulse** turns recorded under `source: desktop`
|
||
(`Pulse delivery check pXXXX-1 …`, `Pulse morning brief …`), on a **~6 h cadence**. Each alert's
|
||
≈4.64 h lands on the then-newest Pulse turn to within about a minute:
|
||
|
||
| Alert | Elapsed printed | Now ≈ | Newest conversational message it measured |
|
||
| --- | --- | --- | --- |
|
||
| 2026-11-15 00:03:35 | `4.64157 h` | 00:00:54 | **2026-11-14 19:22:30** (desktop / Pulse) |
|
||
| 2026-11-15 12:03:14 | `4.63403 h` | 12:01:25 | **2026-11-15 07:23:25** (desktop / Pulse) |
|
||
| 2026-11-16 00:03:15 | `4.64348 h` | 00:01:33 | **2026-11-15 19:22:59** (desktop / Pulse) |
|
||
|
||
**Why they all read ≈4.64:** the `0 */12` slots happen to fire ~4.6 h after the most recent Pulse turn.
|
||
The near-constancy is a **cadence coincidence** and must **not** be read as "the stall is only ~4.6 h" —
|
||
the stall is **38 h 46 m**. Same trap class as the naive-UTC note below: a number shaped like a duration
|
||
that is not one.
|
||
|
||
**Positive proof the capture path is dead rather than merely idle:** the identical class of message *was*
|
||
captured before the stall — the 2026-11-14 00:09:12 run logged
|
||
`last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`. Four 2026-11-15 Pulse turns
|
||
(`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have **no capture row at all**.
|
||
|
||
### The consolidator is alive and every cycle is a deliberate no-op
|
||
|
||
Latest `consolidation_log` row **4696** at `2026-11-15T16:14:14.934306+00:00` (= **2026-11-16 00:14:14
|
||
AWST**), `items_consolidated: 0`, `session_id: auto-consolidate-maintenance`; cadence ~31 min
|
||
(`15:12:13` → `15:43:08` → `16:14:14` UTC). **New evidence:** the row's `summary_preview` reads
|
||
**`No-op consolidation cycle — timestamp updated for watchdog`** — the cycle **touches its own timestamp
|
||
to satisfy an age test**, so a zero-item streak can run indefinitely while every age-based gate reads
|
||
"fresh". The streak is now **~74 consecutive zero-item cycles**, opening
|
||
`2026-11-14T02:04:42+00:00` (10:04:42 AWST). **A zero-item-streak threshold — not an age threshold — is
|
||
what would have caught this at 10:04 on 11-14.**
|
||
|
||
### Adjacent fault in the same runtime — the watchdog lost a slot (2026-11-15 11:31)
|
||
|
||
`96eb95ce12f2` (**Mnemosyne Watchdog**, every 15 min) **failed** on its `11:31:20.699` slot,
|
||
`finished_at 2026-11-15T11:49:10.534610` — **17 m 49.8 s**, exit 1:
|
||
|
||
```
|
||
runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds
|
||
```
|
||
|
||
A **600 s timeout inside a 15-minute slot** overran two slots: the `11:45` slot was **skipped** (no row
|
||
at all) and the **next slot, `12:01:25.120952`, completed** (`→ 12:02:02.657231`), so it self-recovered
|
||
with one slot lost. This is a **new signature** for this job — the August and October occurrences had the
|
||
guard quoting *another* job's error text. Across 11-15 → 11-16 it is the only `failed` row of its 97.
|
||
Alert delivered (`deliveries.db` `11:49:06.052341`, `for_failure: 1`, `delivered`). **Adjacency, not
|
||
proof of common cause** — but the guard and the capture path share the same runtime, so a guard that
|
||
times out at 600 s belongs next to this stall rather than dismissed.
|
||
|
||
|
||
## Re-read 2026-11-17 00:27 AWST — stall **62 h 54 m**, **five** unclosed alerts, zero-item streak **120**
|
||
|
||
Third nightly read, sixth week of the same numbers changing only their counters. **The freeze is still
|
||
total.**
|
||
|
||
- **Counts identical for the fifth consecutive read:** `working_memory` **1130** / `episodic_memory`
|
||
**1249**; newest rows still `2026-11-14 09:32:59` AWST (working) and `09:33:34` AWST (episodic) —
|
||
**62 h 54 m** with **zero** writes.
|
||
- **`cron_incidents` 3 → 5 unclosed** (141 → **143** rows total). Both new rows are `ea79a2107e36`,
|
||
`failure_type: auth`, `state: alerted`, and again `first_seen_at == last_seen_at`:
|
||
|
||
| Incident id | `first_seen_at` | `alerted_at` |
|
||
| --- | --- | --- |
|
||
| `ea79a2_915d004a108a` | `2026-11-16T12:02:06.679532+08:00` | `2026-11-16T12:02:12.472265+08:00` |
|
||
| `ea79a2_0fab8d43db75` | `2026-11-17T00:03:37.624640+08:00` | `2026-11-17T00:03:46.151392+08:00` |
|
||
|
||
The signature-fragmentation mechanism below is holding exactly as described: **one fresh row per
|
||
firing, no escalation**, so the estate's unclosed count now grows **~2 per day** with nothing accumulating
|
||
on any single row.
|
||
- **The monitor is on a 5-slot failure streak** (`jobs.json`: `ea79a2107e36`, `last_status: error`,
|
||
`failure_streak: 5`). 11-16's ledger holds **2 failed rows of 175, both this job**
|
||
(00:01:33 → 00:03:22, 12:01:18 → 12:02:12); 11-17 so far 1 failed of 7 (00:01:39 → 00:03:46).
|
||
Every failure is `exit 1` with `STALE MEMORY PIPELINE: capture missed: us…` — the same gate.
|
||
- **The consolidator is alive and deliberately doing nothing.** `consolidation_log` latest row **4742**
|
||
at `2026-11-16T16:00:41.922076+00:00` (= 00:00:41 AWST), `items_consolidated: 0`, summary *"No-op
|
||
consolidation cycle — timestamp updated for watchdog"*. Last non-zero row: **4622** at
|
||
`2026-11-14T09:33:34` → **120 consecutive zero-item cycles** (was 74 at the 11-16 read). The cycles
|
||
still fire ~every 31 min; there is simply **nothing new to consolidate**, which is the memory-side
|
||
reason the stall cannot self-heal.
|
||
- **Still no cause, still no fix, still `status: active`.** The 11-16 read's two conclusions stand: the
|
||
failure is **total**, not a trickle, and it affects **all** write paths, not just turn capture.
|
||
`review_after` 2026-11-18 — this page is re-read at the next audit, and the monitor keeps alerting.
|
||
|
||
|
||
## Re-read 2026-11-20 00:5x AWST — the freeze ended by itself after **105 h 52 m**, and the "capture is dead" reading was wrong
|
||
|
||
**The capture freeze is over. The alarm is not.** Live reads of `mnemosyne.db` at 2026-11-20 00:0x–00:5x AWST:
|
||
|
||
| counter | 2026-11-17 00:27 | 2026-11-20 00:5x |
|
||
| --- | --- | --- |
|
||
| `working_memory` | 1130 | **1131** |
|
||
| `episodic_memory` | 1249 | **1250** |
|
||
| newest write | `2026-11-14 09:32:59` AWST | **`2026-11-18 19:25:03` AWST** (stored UTC `11:25:03`) |
|
||
|
||
The break coincides with nothing: no config edit, no restart and no repair is recorded against the capture path
|
||
in the window — the only nearby state change is the 11-18 19:18 desktop session itself. The single new
|
||
`working_memory` row carries `session_id: hermes_20261118_191808_5c9e08` and its content begins
|
||
`[USER] You are helping me migrate contex…`, which identifies the captured turn (the `desktop` session
|
||
**`20261118_191808_5c9e08`**, *"Helping migrate context AI assistant"*, 9 messages, started 19:18 AWST).
|
||
**Stall duration: 105 h 52 m** (2026-11-14 09:33 → 2026-11-18 19:25 AWST).
|
||
|
||
⚠ **The correction that matters — the window contained almost no conversation to capture.** `state.db` lists
|
||
**three** non-cron sessions since 2026-11-14 00:00 AWST: two `desktop` sessions on **11-14** (**08:35**, 40
|
||
messages; **08:52**, 41 messages — both *before* the 09:33 boundary), then **nothing** until the **11-18 19:18**
|
||
session, whose turn is the row that ends the freeze. Every interactive session in the window was captured, so
|
||
the earlier figures on this page (`0 writes in 38 h 46 m`, `62 h 54 m`) are real but measure **a quiet estate**,
|
||
not a broken hook: `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message
|
||
whose session source is not `cron`/`subagent`, so **silence and failure read identically**. That ambiguity — not
|
||
a broken capture path — is the durable defect on this page.
|
||
|
||
**The monitor is still failing on that same metric, after the break.** `ea79a2107e36` slots since 11-18:
|
||
`11-18 00:01:28` failed · `11-18 12:01:13` failed · **`11-19 00:01:34` `completed`** (the only clean slot since
|
||
11-17) · `11-19 12:01:17` failed · `11-20 00:01:22` failed — every failure `Script exited with code 1 / 🚨
|
||
Mnemosyne Health Monitor — exception(s): STALE MEMORY PIPELINE: capture …`, i.e. **the same signature firing on
|
||
a healthy pipeline** (`failure_streak: 2`; incidents `ea79a2_f2be2db3bfce` at 2026-11-19 12:01:58 and
|
||
`ea79a2_aae4d914f3c9` at 2026-11-20 00:03:36, both `alerted`, both `failure_type: auth` — the scheduler's
|
||
classifier, not a credential fault). The `11-19 00:01` pass and the `12:01` failure bracket the threshold: the
|
||
metric is *age of the newest captured conversational turn*, which only grows while the estate is silent.
|
||
|
||
**The other half of the original finding is unchanged, and larger.** The consolidator still consolidates zero
|
||
items: newest `consolidation_log` row **`2026-11-19T16:20:14.587267+00:00`** (= 2026-11-20 00:20 AWST, so it is
|
||
running on its ~31-minute cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle
|
||
— timestamp updated for watchdog"`. **199 of the last 200 cycles are zero-item** (was **120** at the 11-17 read
|
||
— consistent with ~31 min cadence over the three days). The zero-item-streak threshold remains **proposed, not
|
||
applied**.
|
||
|
||
**Unclosed incidents: 5 → 4** (one of the rows present at the 11-17 read has since closed). Current four, none
|
||
closed automatically: `cbfe43_fa12cca1ad4b` (*auth*, 2026-11-18 00:01:42, `alerted_at: None` — the wiki audit's
|
||
silent skip, see [[incidents/2026-11-18-wiki-audit-credit-outage]]), `96eb95_98ade422d195` (*script*,
|
||
2026-11-18 20:01:15 → last_seen 2026-11-20 00:31:16 — see
|
||
[[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]), `ea79a2_f2be2db3bfce` (2026-11-19
|
||
12:01:58) and `ea79a2_aae4d914f3c9` (2026-11-20 00:03:36).
|
||
|
||
**Standing open question, restated:** not *"why did capture stop"* but **"why does the gate treat conversational
|
||
silence as a fault?"** Until that is fixed, this page will keep re-opening on quiet days.
|
||
|
||
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
||
|
||
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
||
**2026-11-14 09:32:59 AWST** (the row's own `timestamp` field reads `2026-11-14T09:32:59`).
|
||
- Newest `episodic_memory.created_at`: **`2026-11-14 01:33:34`** (09:33:34 AWST). **Both** write paths
|
||
stopped inside the same minute.
|
||
- Last 8 `working_memory` writes, all on 11-14 (UTC): `01:01:20, 01:02:39, 01:02:40, 01:03:28,
|
||
01:04:19, 01:32:58, 01:32:59, 01:32:59` — then **nothing for ~15 hours**.
|
||
- Rows by day (UTC `created_at`): **11-09: 11, 11-10: 10, 11-11: 8, 11-12: 37, 11-13: 14, 11-14: 15**.
|
||
So the pipeline was *low-volume but alive* before the stall, and the stall is not a normal quiet stretch.
|
||
- Totals at audit time: **working 1130 / episodic 1249**, DB **27 MB**.
|
||
- Gap: at the 00:03:35 alert the newest capture was **14 h 30 m 36 s** old; at this audit's read
|
||
(~00:40) **≈15 h 7 m**.
|
||
|
||
## The alert (and its mislabelling)
|
||
|
||
```
|
||
🚨 Mnemosyne Health Monitor — exception(s):
|
||
|
||
STALE MEMORY PIPELINE: capture missed: user message 4.64157453 ago (desktop)
|
||
has no working_memory row (last capture 2026-11-14 01:32:59)
|
||
```
|
||
- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`,
|
||
`first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`.
|
||
Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`.
|
||
- ⚠ **The `auth` label describes the classifier, not the fault — now evidenced, not just suspected.** The
|
||
failure is a *staleness* check and the page records no auth error text. This job's **twelve earlier
|
||
`cron_incidents` rows** read `agent` ×3, `timeout` ×1, `unknown` ×8 (`ea79a2_649d8ac46e8a`,
|
||
`_f17b45759db0`, `_573d17e10fc1`, `_6fb70b032305`, `_54d6b1f0f58a`, `_747b27556fd0`, `_c3ab2d784ba3`,
|
||
`_a4661554a3c2`, `_d93c3acdc4f2`, `_9197df32320b`, `_111e67a3d83c`, `_f317fcd843ae`), and the estate-wide
|
||
table is **53 `unknown` / 47 `agent` / 15 `timeout` / 14 `script` / 5 `auth`**. `agent` is also impossible
|
||
here: the job runs `no_agent` (plain script). Treat the field as opaque routing metadata.
|
||
- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a
|
||
~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own
|
||
log line below.
|
||
|
||
## What the earlier slots actually saw — correction, re-derived 2026-11-15 01:2x AWST
|
||
|
||
**An earlier revision of this page got this backwards.** It claimed the **12:00:52 AWST** run had
|
||
"already seen the same hole and called it benign" (`capture delay -589.5 min`), giving the estate ~12 h
|
||
of unalarmed lead time, and blamed a "sign, not magnitude" threshold. **Both claims are withdrawn.** The
|
||
`-589.5 min` line is not the 12:00 run's — it sits inside the **00:03:35 run's own output block**.
|
||
|
||
Verified by line number in `~/.hermes/logs/mnemosyne-health-monitor.log` (**1103 lines**):
|
||
|
||
| line | content |
|
||
| --- | --- |
|
||
| 884 | `=== 2026-11-14 00:09:12 AWST health monitor (no_agent) ===` |
|
||
| 900 | `ℹ️ last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min` |
|
||
| 957 | `=== 2026-11-14 12:01:52 AWST health monitor (no_agent) ===` |
|
||
| 973 | `ℹ️ last user msg 2026-11-14 01:03 UTC (ios) capture delay 29.4 min` |
|
||
| 1030 | `=== 2026-11-15 00:03:35 AWST health monitor (no_agent) ===` |
|
||
| 1046 | `ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min` |
|
||
| 1047 | `❌ STALE MEMORY PIPELINE: …` |
|
||
|
||
Two further corrections to the earlier revision:
|
||
|
||
- **There was no 18:03 run.** The job's schedule is `0 */12 * * *`; the only 2026-11-14 slots are
|
||
00:09:12 and 12:01:52 (both exit 0, delivery `suppressed`). The "exited 0 at 12:00 and 18:03"
|
||
sentence is withdrawn — 18:03 belongs to some other job.
|
||
- **The detection chain was not broken.** `mnemosyne_freshness_gate.py` fires on **magnitude**
|
||
(`delay_min < -CAPTURE_LAG_MIN`, `MNEMOSYNE_CAPTURE_LAG_MIN` default **120** min) — there is no
|
||
sign-only test to blame. The 12:00:52 measurement of **+29.4 min** is the delay of the newest message
|
||
*that had been captured* (the 09:03 AWST `ios` turn, captured 09:32:59). The gate computes against the
|
||
newest **uncaptured** message and only reports a problem when one exists; with the conversation idle
|
||
from 09:03 to 19:22, every user message still had a capture row, so there was **nothing to detect**.
|
||
The first message to postdate the stall was the **19:22 AWST desktop** turn (`11:22 UTC`, the same
|
||
message the 00:03 alert names); the next scheduled slot after it was 00:03:35, which escalated
|
||
(`-589.5 min` = 09:32:59 − 19:22).
|
||
|
||
**Corrected statement of the gap:** the pipeline died at **09:32:59 AWST**; it became *detectable* at
|
||
**19:22 AWST** (first uncaptured message) and was *detected* at **00:03:35** by the next slot — a
|
||
detection latency of 4 h 41 m after detectability, driven by cadence, not by a broken threshold.
|
||
**What was genuinely unread sits in the consolidation counters:** the cycles ran every ~31 min and
|
||
returned **0 items for 29 consecutive runs** (streak opening `2026-11-14T02:04:42+00:00` = **10:04:42
|
||
AWST**; last productive cycles 4622/4621/4620 at `09:33:34` / `09:33:16` / `09:01:40`). Every run
|
||
printed `Last consolidation: … (0 items)` — including the "healthy" 12:00:52 one — but the gate tests
|
||
consolidation **age** only (`> 36 h`, default), never the item count. **An empty-but-current cycle is
|
||
the loudest signal available and the only one nothing watches.**
|
||
|
||
## Impact
|
||
|
||
> **Magnitude superseded 2026-11-16 00:18 AWST: 38 h 46 m uncaptured** (was ~15 h), with **two more
|
||
> unclosed alerts** and four 2026-11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`)
|
||
> added to the uncaptured set. The ~15 h figures below stand as the 2026-11-15 readings.
|
||
|
||
|
||
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
|
||
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
|
||
window. Recall over that period will answer from nothing.
|
||
- The **23:30 AWST sleep/consolidation cycle ran and found 0 items** to consolidate
|
||
(`Last consolidation: 2026-11-14T15:30:51 UTC (0 items)`), so consolidation is *not* compensating.
|
||
- Nothing else in the estate broke: **2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` /
|
||
1 `running`** in `executions.db`, deliveries **4 / 4 `delivered`**, and no other cron row went red. This is a
|
||
**memory-path incident only**. The one exception is the monitor's **own** job: it is now the fleet's
|
||
**single non-clean row** (`last_status: error`, `failure_streak` 1, previously `ok` / 0) and
|
||
**`cron_incidents` went 0 → 1 unclosed** (138 rows total). **2026-11-15 so far: 8 rows — 7
|
||
`completed` / 1 `failed`** (this job), 1 delivery `delivered`.
|
||
|
||
## Not the known config killer
|
||
|
||
The previously recorded capture-killer on this host is a **stringified list/dict** written into
|
||
`~/.hermes/config.yaml` by `hermes config set`. Checked and **clean**:
|
||
- `config.yaml` **mtime 2026-11-13 18:03** — the file was not modified during or before the stall began.
|
||
- A full YAML walk found **no** scalar that begins with `[` or `{`; `memory.mnemosyne.*` parses as
|
||
proper lists (`ignore_patterns: []`, `skip_contexts`, `sync_roles`).
|
||
|
||
## Open / unverified
|
||
|
||
- **Root cause: not established.** No capture-path error lines appear in `agent.log` after 09:33 —
|
||
the writes simply stop. Candidates to test next: a stuck capture worker or watermark, the
|
||
`pulse`/heartbeat traffic that ran 12:03–19:22, session-side `memory_enabled` /
|
||
`mnemosyne.profile_isolation` interaction, or the sleep cycle's `consolidation_claimed_at` state.
|
||
- ✅ **Resolved — the 00:03 escalation DID deliver.** `deliveries.db` holds exactly one 2026-11-15 row:
|
||
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
|
||
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
|
||
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
|
||
- ✅ **Answered 2026-11-16 00:18 — capture did NOT resume.** Two further monitor slots failed on the
|
||
same condition (`12:03:14` and `2026-11-16 00:03:15`) and the row counts are unchanged (`working`
|
||
1130 / `episodic` 1249). Closed as *no*; the underlying cause remains open.
|
||
- **The estate's only automated alarm is understating this incident.** The alert's ≈4.64 h is a cadence
|
||
artefact, not a stall duration — anyone reading the alert text alone would conclude the pipeline is
|
||
~4.6 h behind when it is 38 h 46 m. A stall-duration figure (newest capture vs now) would fix it.
|
||
|
||
## Related
|
||
|
||
- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence
|
||
- [[systems/memory-backend]] — memory backend settings
|
||
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] — the still-open secrets-at-rest page for the same database
|
||
- [[systems/scheduled-tasks]] — the fleet's execution ledger
|