audit 2026-11-15: capture stall 38h46m (freeze total, counts unchanged); cron_incidents 1->3 unclosed (signature embeds elapsed-hours float); monitor's 4.64h decoded as newest-conversational-message age; watchdog lost an 11:45 slot to a 600s deps-guard timeout; ledger 168 rows 165/3 (11-14 settled 167/0)

This commit is contained in:
Tony0410
2026-11-16 00:32:12 +08:00
parent 402d8546e8
commit 9f80926dda
6 changed files with 180 additions and 22 deletions
+19
View File
@@ -1782,3 +1782,22 @@ provider, so the cold-start path is untested. Page → `status: degraded` with a
section; **fix not re-applied** (needs Anthony's call: re-port + commit locally, or upstream).
Pages corrected: [[incidents/2026-10-09-cron-provider-timeout-storm]],
[[incidents/2026-10-22-conduit-memory-start-timeout]], [[index.md]].
## 2026-11-16 00:2x — nightly audit (window 2026-11-15 00:00 – 23:59 AWST)
**Nothing new was created and nothing was archived — the window's findings all land on pages that already existed. The memory incident deepened, and a second job failed.**
**Ledger — 2026-11-15: 168 rows, 165 `completed` / 3 `failed`** (2026-11-14 settled **167 / 167 / 0**, so nothing carried over). The three: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604`; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` (**17 m 49.8 s**, exit 1 — `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds`, a **new signature** for this job; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430`. Deliveries **9 / 9 `delivered`**, 3 `for_failure`. Fleet **29 (26 / 3)** — a **tenth** window unchanged; non-clean `last_status` still 1 (`ea79a2107e36`, `error`, streak **3**).
**Capture stall: still open at 38 h 46 m, and the freeze is total.** `working_memory` **1130** / `episodic_memory` **1249** — *identical at every read across the stall* (11-15 00:4x, 11-15 12:03, 11-16 00:18), so this is a hard stop, not degradation. Newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` AWST; newest **user-turn** capture `2026-11-14 09:04:19`.
**Decoded this run — the monitor's "≈4.64 h" is not the stall.** `mnemosyne_freshness_gate.py`'s `newest_user_message()` selects the newest `messages` row with `role='user'` whose session `source NOT IN ('cron','subagent')` (`NON_CONVERSATIONAL`, line 52) and reports **that message's** age — here the **Pulse** turns on `source: desktop`, ~6 h cadence. Every `0 */12` slot lands ~4.6 h after one, which is why all three alerts read alike: 11-15 00:03:35 → **11-14 19:22:30**; 11-15 12:03:14 → **11-15 07:23:25**; 11-16 00:03:15 → **11-15 19:22:59**. Proof capture is dead rather than idle: the 11-14 00:09:12 run logged `last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`, and four 11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have no capture row. **The estate's only automatic alarm has therefore understated severity on every fire.**
**New mechanism recorded — incident-signature fragmentation.** `cron_incidents` **1 → 3 unclosed** (138 → **141** rows): `ea79a2_601037516abd`, `ea79a2_5f817708701e`, `ea79a2_75303493d484` — all this job, all `failure_type: auth`, all `state: alerted`, each `first_seen_at == last_seen_at`. The signature embeds the elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice, so one durable fault mints a fresh unclosed row every slot and no row escalates. **Fix proposed, not applied** (normalise the value out of the signature) — it needs a code change on the monitor's incident path.
**Consolidator: alive, and every cycle is a deliberate no-op.** Latest row **4696** at `2026-11-15T16:14:14.934306+00:00` (= 00:14:14 AWST), `items_consolidated: 0`, `summary_preview` `No-op consolidation cycle — timestamp updated for watchdog` — the cycle **bumps its own timestamp to satisfy an age test**, so an age-only gate reads "fresh" forever. Streak now **~74 consecutive zero-item cycles** (opened `2026-11-14T02:04:42+00:00`). A **zero-item-streak** threshold is what would have caught this at 10:04 on 11-14.
**Sessions:** 9 started on 11-15, **all `source: cron`** — no interactive session, so no human-side change the wiki missed. Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**.
Pages updated: [[incidents/2026-11-14-mnemosyne-capture-stall]] (status re-read, the `4.64 h` decode, the three-row table + fragmentation mechanism, the watchdog slot loss), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[current-state.md]], [[index.md]]. Root cause of the stall: **still not established** — the incident stays `active`, `closed_at: null`.