audit(2026-10-31): document CT460 5h30m host outage; correct the 'paused' health monitor (it was running, closed completed 10:02:48); reconcile fleet counts to 26/3

This commit is contained in:
hermes
2026-10-31 00:39:11 +08:00
parent db6af66878
commit 74a5b2d117
8 changed files with 241 additions and 27 deletions
+35
View File
@@ -1646,3 +1646,38 @@ not pushed.
- **Not changed:** `systems/skills-index.md` count (302 is correct), CT460's Serve route table (matched live exactly), and
the steel-removal records across the wiki (confirmed still true).
- **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a key or token value.
## [2026-10-31] audit | Nightly wiki audit — window 2026-10-30 00:00 → 2026-10-31 00:3x AWST
**Verdict: one large undocumented event found and written up, and one *documented* incident found to have been
recorded wrong.**
- **NEW PAGE — [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]].** CT460 was dark for **5 h 30 m** on
2026-10-30 and **no monitor alerted**. Evidence: **zero `executions.db` rows between 10:58:20 and 16:37:21**;
PVE host `pve` boot time **2026-10-30 16:24:00**; CT460 boot **16:28:00**; the previous gateway life
(`pid=3913841`, started 2026-10-28T17:21:58Z) logged **UNCLEANLY at 16:35:50** (*"no exit path ran —
SIGKILL / OOM / VM death"*); scheduler resumed **16:37:21** with an **8-run catch-up burst** through 16:38:22.
The day still closed **138 executions / 0 failed** against 10-29's **177 / 0** — **≈39 lost slots, no failures**.
Root cause **not established**: no OOM evidence locally or on the PVE host, so the ledger string is read as a
candidate list, not a diagnosis. **Still open:** the two jobs that could have detected it (`cbfe434a3b5e` NFS
mount monitor, `1a8e90ea2271` m93p LXC boot monitor) are disabled.
- **CORRECTED — [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] was right about the record and wrong
about the run.** `ea79a2107e36`'s 2026-10-30 00:01:08 slot was **still running** at the 03:3x reading; it closed
**`completed` at 10:02:48** after **10 h 01 m 40 s** and **delivered**, auto-closing incident
`ea79a2_d93c3acdc4f2` **`resolved` at 2026-10-30T10:02:48.662+08:00**. Live job state is now `enabled: true`,
`last_status: ok`, `failure_streak: 0`. The page's *"zero `deliveries.db` rows / the notice was never
delivered"* claim was an artefact of reading the day at 03:3x: 2026-10-30 finished with **8 delivery rows, all
`delivered`** (latest 18:46:04). Page frontmatter moved `active` → **`resolved`**, with the old snapshot kept
below as explicitly historical. ⚠ Its **12:01 slot was lost to the host outage** and is the still-`running`
16:37:21 catch-up. **Reconciliation lesson: a slot mid-flight is not a failure — check `finished_at`, not
`last_status`, before writing an incident.**
- **Reconciled — `systems/scheduled-tasks.md`.** Fleet **29 jobs (26 enabled / 3 disabled)** — the "4th disabled
job" reading is superseded (frontmatter date, fleet block, Total jobs, non-clean `last_status` **4 → 3**,
failing **1 → 0**, the health-monitor table row, the disabled table row, and a new **"Resolved 2026-10-30"**
subsection). Non-clean now **3 = 1 `error` + 2 `delivery_failed`**; unclosed incident rows **2 → 1** (survivor:
the Facebook monthly's `0e753e_e7f8194a7d06`, due 11-01).
- **Reconciled** — `current-state.md` (new leading cron entry + superseded marker), `infrastructure/hosts.md`
(cron row), `index.md` (frontmatter, banner, scheduled-tasks entry, both 10-30 incident entries),
`incidents/2026-10-30-mnemosyne-health-monitor-paused.md`.
- **Checked, unchanged:** 12 sessions active in-window (longest `local-40472e0b…`, 1,351 messages), no orphaned
`running` rows, no incidents opened or closed after 10-30, no obsolete pages to archive, no secrets written.