audit 2026-11-15: capture stall 38h46m (freeze total, counts unchanged); cron_incidents 1->3 unclosed (signature embeds elapsed-hours float); monitor's 4.64h decoded as newest-conversational-message age; watchdog lost an 11:45 slot to a 600s deps-guard timeout; ledger 168 rows 165/3 (11-14 settled 167/0)
This commit is contained in:
@@ -1,17 +1,17 @@
|
||||
---
|
||||
title: Mnemosyne capture stalled for ~15 hours (2026-11-14 09:33 AWST onward) — health monitor escalated 00:03 AWST 2026-11-15
|
||||
title: Mnemosyne capture stalled for 38+ hours (2026-11-14 09:33 AWST onward) — three unclosed alerts, freeze total on re-read
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-15
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-11-17
|
||||
updated: 2026-11-16
|
||||
verified_on: 2026-11-16
|
||||
review_after: 2026-11-18
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident]
|
||||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db]
|
||||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
|
||||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
|
||||
---
|
||||
|
||||
# Mnemosyne capture stalled for ~15 hours (from 2026-11-14 09:33 AWST)
|
||||
# Mnemosyne capture stalled for 38+ hours (from 2026-11-14 09:33 AWST)
|
||||
|
||||
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
|
||||
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
|
||||
@@ -21,6 +21,95 @@ Found by the **2026-11-14 nightly audit**; not caused by anything in that audit'
|
||||
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
|
||||
the same condition was classified informational — see below).
|
||||
|
||||
## Re-read 2026-11-16 00:18 AWST — still open, stall 38 h 46 m, and now **three** unclosed alerts
|
||||
|
||||
Six hours later, at the next nightly audit, nothing has moved. The freeze is **total**, the monitor has
|
||||
fired twice more, and the scheduler has minted further unclosed rows for the same fault.
|
||||
|
||||
- **The counts are frozen, not merely low.** `working_memory` **1130** / `episodic_memory` **1249** —
|
||||
**the same two numbers as every earlier read of the stall** (2026-11-15 ~00:4x and 12:03) — so this is
|
||||
not a quiet stretch or a degraded trickle. Zero writes in **38 h 46 m**.
|
||||
- **The whole write path stopped, not just turn capture.** Newest rows remain the three
|
||||
`[MODEL_REFRESH_PROPOSAL]` maintenance writes at **`09:32:58` / `09:32:59` AWST**; the newest
|
||||
*user-turn* capture is **`2026-11-14 09:04:19 AWST`**. Maintenance outlived the last turn by ~28 min,
|
||||
then both stopped inside the same minute.
|
||||
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows total). All three are `ea79a2107e36`, all
|
||||
`failure_type: auth`, all `state: alerted`, and **each has `first_seen_at == last_seen_at`** — a single
|
||||
observation, no `last_seen_at` accumulation, no escalation:
|
||||
|
||||
| Incident id | `first_seen_at` | `alerted_at` |
|
||||
| --- | --- | --- |
|
||||
| `ea79a2_601037516abd` | `2026-11-15T00:03:35.815373+08:00` | `2026-11-15T00:03:44.559501+08:00` |
|
||||
| `ea79a2_5f817708701e` | `2026-11-15T12:03:14.928500+08:00` | `2026-11-15T12:03:20.245081+08:00` |
|
||||
| `ea79a2_75303493d484` | `2026-11-16T00:03:15.418001+08:00` | `2026-11-16T00:03:22.365399+08:00` |
|
||||
|
||||
- ⚠ **New mechanism, evidenced: the incident signature embeds the elapsed-hours float.** The three
|
||||
signatures differ only by `4.64157453` / `4.634034074444445` / `4.643481075555555` — a value **never
|
||||
equal twice** — so the recurrence never matches the prior row. One durable fault therefore **mints a
|
||||
fresh unclosed row every slot**, and the estate's unclosed count grows ~1 per 12 h while no single row
|
||||
ever escalates. This job alone now has **15** `cron_incidents` rows (12 older `unknown`, these 3
|
||||
`auth`). **Proposed fix — NOT applied** (it changes the monitor's incident path, so it is Anthony's
|
||||
call): normalise the value out of the signature — round it, or omit it — so the same condition
|
||||
**re-opens the existing row** instead of fragmenting.
|
||||
|
||||
### The monitor's ≈4.64 h figure is NOT the stall — verified against the gate's own source
|
||||
|
||||
Every alert in this sequence prints a near-identical elapsed time while the stall grew from 14 h to
|
||||
38.7 h. That number is not a measurement of the stall. `mnemosyne_freshness_gate.py`'s
|
||||
`newest_user_message()` selects the newest `messages` row with `role='user'` whose session
|
||||
`source NOT IN ('cron','subagent')` (the `NON_CONVERSATIONAL` set, line 52) and returns that row's
|
||||
`source`; the alert prints **`(now − that message) / 3600` h**. So the figure is **the age of the newest
|
||||
*conversational* message — not the age of the newest capture.**
|
||||
|
||||
The newest conversational messages on this box are **Pulse** turns recorded under `source: desktop`
|
||||
(`Pulse delivery check pXXXX-1 …`, `Pulse morning brief …`), on a **~6 h cadence**. Each alert's
|
||||
≈4.64 h lands on the then-newest Pulse turn to within about a minute:
|
||||
|
||||
| Alert | Elapsed printed | Now ≈ | Newest conversational message it measured |
|
||||
| --- | --- | --- | --- |
|
||||
| 2026-11-15 00:03:35 | `4.64157 h` | 00:00:54 | **2026-11-14 19:22:30** (desktop / Pulse) |
|
||||
| 2026-11-15 12:03:14 | `4.63403 h` | 12:01:25 | **2026-11-15 07:23:25** (desktop / Pulse) |
|
||||
| 2026-11-16 00:03:15 | `4.64348 h` | 00:01:33 | **2026-11-15 19:22:59** (desktop / Pulse) |
|
||||
|
||||
**Why they all read ≈4.64:** the `0 */12` slots happen to fire ~4.6 h after the most recent Pulse turn.
|
||||
The near-constancy is a **cadence coincidence** and must **not** be read as "the stall is only ~4.6 h" —
|
||||
the stall is **38 h 46 m**. Same trap class as the naive-UTC note below: a number shaped like a duration
|
||||
that is not one.
|
||||
|
||||
**Positive proof the capture path is dead rather than merely idle:** the identical class of message *was*
|
||||
captured before the stall — the 2026-11-14 00:09:12 run logged
|
||||
`last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`. Four 2026-11-15 Pulse turns
|
||||
(`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have **no capture row at all**.
|
||||
|
||||
### The consolidator is alive and every cycle is a deliberate no-op
|
||||
|
||||
Latest `consolidation_log` row **4696** at `2026-11-15T16:14:14.934306+00:00` (= **2026-11-16 00:14:14
|
||||
AWST**), `items_consolidated: 0`, `session_id: auto-consolidate-maintenance`; cadence ~31 min
|
||||
(`15:12:13` → `15:43:08` → `16:14:14` UTC). **New evidence:** the row's `summary_preview` reads
|
||||
**`No-op consolidation cycle — timestamp updated for watchdog`** — the cycle **touches its own timestamp
|
||||
to satisfy an age test**, so a zero-item streak can run indefinitely while every age-based gate reads
|
||||
"fresh". The streak is now **~74 consecutive zero-item cycles**, opening
|
||||
`2026-11-14T02:04:42+00:00` (10:04:42 AWST). **A zero-item-streak threshold — not an age threshold — is
|
||||
what would have caught this at 10:04 on 11-14.**
|
||||
|
||||
### Adjacent fault in the same runtime — the watchdog lost a slot (2026-11-15 11:31)
|
||||
|
||||
`96eb95ce12f2` (**Mnemosyne Watchdog**, every 15 min) **failed** on its `11:31:20.699` slot,
|
||||
`finished_at 2026-11-15T11:49:10.534610` — **17 m 49.8 s**, exit 1:
|
||||
|
||||
```
|
||||
runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds
|
||||
```
|
||||
|
||||
A **600 s timeout inside a 15-minute slot** overran two slots: the `11:45` slot was **skipped** (no row
|
||||
at all) and the **next slot, `12:01:25.120952`, completed** (`→ 12:02:02.657231`), so it self-recovered
|
||||
with one slot lost. This is a **new signature** for this job — the August and October occurrences had the
|
||||
guard quoting *another* job's error text. Across 11-15 → 11-16 it is the only `failed` row of its 97.
|
||||
Alert delivered (`deliveries.db` `11:49:06.052341`, `for_failure: 1`, `delivered`). **Adjacency, not
|
||||
proof of common cause** — but the guard and the capture path share the same runtime, so a guard that
|
||||
times out at 600 s belongs next to this stall rather than dismissed.
|
||||
|
||||
|
||||
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
||||
|
||||
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
||||
@@ -103,6 +192,11 @@ the loudest signal available and the only one nothing watches.**
|
||||
|
||||
## Impact
|
||||
|
||||
> **Magnitude superseded 2026-11-16 00:18 AWST: 38 h 46 m uncaptured** (was ~15 h), with **two more
|
||||
> unclosed alerts** and four 2026-11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`)
|
||||
> added to the uncaptured set. The ~15 h figures below stand as the 2026-11-15 readings.
|
||||
|
||||
|
||||
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
|
||||
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
|
||||
window. Recall over that period will answer from nothing.
|
||||
@@ -133,7 +227,12 @@ The previously recorded capture-killer on this host is a **stringified list/dict
|
||||
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
|
||||
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
|
||||
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
|
||||
- Whether capture resumed after this audit's read (the monitor's next slot is the natural check).
|
||||
- ✅ **Answered 2026-11-16 00:18 — capture did NOT resume.** Two further monitor slots failed on the
|
||||
same condition (`12:03:14` and `2026-11-16 00:03:15`) and the row counts are unchanged (`working`
|
||||
1130 / `episodic` 1249). Closed as *no*; the underlying cause remains open.
|
||||
- **The estate's only automated alarm is understating this incident.** The alert's ≈4.64 h is a cadence
|
||||
artefact, not a stall duration — anyone reading the alert text alone would conclude the pipeline is
|
||||
~4.6 h behind when it is 38 h 46 m. A stall-duration figure (newest capture vs now) would fix it.
|
||||
|
||||
## Related
|
||||
|
||||
|
||||
Reference in New Issue
Block a user