wiki: 2026-11-14 audit corrections — withdraw the 'sign not magnitude' / '12 h of lead time' / '18:03 run' claims; verified escalation mechanism (activity-aware magnitude gate) + 29-cycle zero-item consolidation streak

This commit is contained in:
Tony0410
2026-11-15 01:28:53 +08:00
parent 2d439145bf
commit 82876abdfb
5 changed files with 134 additions and 31 deletions
+44 -16
View File
@@ -24,11 +24,15 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
error`, **`failure_streak` 1** — its **00:00:53 slot failed at 00:03:44 on `Script exited with code 1`**
("STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59"), the first red row since the
Uber Eats job cleared on **2026-11-13**. **Unclosed incident rows 0 -> 1** (**138** total), the row being
`ea79a2_601037516abd` (`state: alerted`, `failure_type: auth` — **mislabelled**, `alerted_at`
00:03:44.5) → [[incidents/2026-11-14-mnemosyne-capture-stall]]. The job's own alert **did deliver**
(`deliveries.db` 2026-11-15 row, `for_failure: 1`, `status: delivered`, 00:03:39) — the failure was in
*detection*, not notification: at **12:00:52** the same slot had already seen the identical hole and
logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**.
`ea79a2_601037516abd` (`state: alerted`, `alerted_at` 00:03:44.5; its `failure_type: auth` is **not**
evidence of a credential fault — this job's twelve earlier rows read `agent` ×3 / `timeout` ×1 /
`unknown` ×8, and the estate-wide table is 53 `unknown` / 47 `agent` / 15 `timeout`) →
[[incidents/2026-11-14-mnemosyne-capture-stall]]. The job's own alert **did deliver**
(`deliveries.db` 2026-11-15 row, `for_failure: 1`, `status: delivered`, 00:03:39) — **notification
worked, and so did detection; the earlier slots simply had nothing to see.** *(Corrected 2026-11-15
01:2x — an earlier revision of this entry claimed the **12:00:52** slot had "already seen the identical
hole" and logged it ℹ️ at `-589.5 min`, giving "~12 h of lead time"; that was wrong on both counts. See
[[#Known Issues]] for the verified mechanism.)*
**2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (that monitor run). Next slot of the monitor:
**2026-11-15 12:00** (schedule `0 */12 * * *`) — the natural check for whether capture resumed.
- **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third
@@ -311,17 +315,41 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne
### Open 2026-11-15
- **The Mnemosyne Health Monitor (`ea79a2107e36`) is the fleet's only non-clean row** — `last_status: error`,
`failure_streak: 1`, unclosed incident `ea79a2_601037516abd` (`state: alerted`, wrongly labelled
`failure_type: auth`). Its 00:00 slot exited 1 at **00:03:44** on `STALE MEMORY PIPELINE: capture missed …
last capture 2026-11-14 01:32:59` (raw **UTC** in the alert text, so it *reads* as a 23 h gap; the true
AWST gap at the alert was **14 h 31 m**). The **12:00:52** slot saw the same hole and logged it
**ℹ️ informational** — `capture delay -589.5 min`, the negative sign hiding its growth — so the estate
had **~12 h of lead time and no alarm**. The alert itself delivered (00:03:39, `for_failure: 1`).
Root cause **not established**; the known stringified-`config.yaml` killer is ruled out. Full page:
[[incidents/2026-11-14-mnemosyne-capture-stall]]. **Lessons for this fleet:** (a) a monitor that
distinguishes "capture is late" only in *sign* will never escalate on a growing hole — the threshold must
be on magnitude; (b) alert timestamps printed from raw UTC inside a +08:00 estate are a standing
misreading trap.
`failure_streak: 1`, unclosed incident `ea79a2_601037516abd` (`state: alerted`; the row's
`failure_type: auth` is the scheduler's classifier and does **not** describe the fault — this job's
earlier rows read `agent` / `timeout` / `unknown`). Its 00:00 slot exited 1 at **00:03:44** on
`STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59` (raw **UTC** in the alert
text, so it *reads* as a ~23 h gap; the true AWST gap at the alert was **14 h 31 m**). The alert itself
delivered (00:03:39, `for_failure: 1`). Root cause **not established**; the known
stringified-`config.yaml` killer is ruled out. Full page:
[[incidents/2026-11-14-mnemosyne-capture-stall]].
- **Verified mechanism of the escalation (re-derivation pass, 2026-11-15 01:2x AWST — corrects the first
reading):** the **12:00:52** slot did **not** see the hole. `~/.hermes/logs/mnemosyne-health-monitor.log`
pins it by line number — run header `2026-11-14 12:01:52 AWST` at **line 957**, next header
(`2026-11-15 00:03:35 AWST`) at **line 1030**, and the `capture delay -589.5 min` line at **line 1046**,
i.e. *inside the 00:03:35 run's own block*, immediately above the ❌ at line 1047. The 12:00:52 run's own
measurement was `ℹ️ last user msg 2026-11-14 01:03 UTC (ios) capture delay **29.4 min**` — healthy, exit 0,
delivery `suppressed`. `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`,
`MNEMOSYNE_CAPTURE_LAG_MIN` default **120** min; consolidation age default **36 h**), but it is
**activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last
captured message before the stall was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min); the first
message to postdate it was the **19:22 AWST desktop** turn — *after* the 12:00 slot, so midnight was the
first slot that could see the miss. **The ~12 h of quiet was the `0 */12` cadence, not a threshold bug.**
- **What genuinely went unread — consolidation item counts.** The sleep cycles kept firing every ~31 min and
returned **0 items: 29 consecutive zero-item cycles**, streak opening `2026-11-14T02:04:42+00:00`
(**10:04:42 AWST**) and running to the alert; the last productive cycles were 4622/4621/4620 at
`09:33:34` / `09:33:16` / `09:01:40` (naive **AWST** — see the format trap below). Every run printed
`Last consolidation: … (0 items)`, including the healthy 12:00 one, but the gate tests consolidation
**age** (`> 36 h`) only — never the item count. A pipeline writing nothing still reads as fresh.
**Candidate threshold: an N-consecutive-zero-item-cycle exception.**
- **Format trap in `consolidation_log.created_at` (verified this pass):** rows 4640–4651 carry a `+00:00`
offset, rows 4620–4622 are **naive and local (AWST)**. Read as one axis they misplace events by 8 h and
make the streak look as if it began after the last productive cycle. Same family as the alert text below.
- **Lessons for this fleet:** (a) alert timestamps printed from raw UTC inside a +08:00 estate are a standing
misreading trap (the alert's `01:32:59` is 09:32:59 AWST); (b) freshness gates must test a pipeline's
**output volume**, not just the age of its last run — an empty-but-current cycle is the failure signature
for a dead write path; (c) when a ledger's `failure_type` disagrees with the error text, trust the text —
the classifier is `unknown`/`agent` for most rows.
### Resolved 2026-11-02