audit(2026-10-31): document CT460 5h30m host outage; correct the 'paused' health monitor (it was running, closed completed 10:02:48); reconcile fleet counts to 26/3
This commit is contained in:
@@ -1646,3 +1646,38 @@ not pushed.
|
||||
- **Not changed:** `systems/skills-index.md` count (302 is correct), CT460's Serve route table (matched live exactly), and
|
||||
the steel-removal records across the wiki (confirmed still true).
|
||||
- **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a key or token value.
|
||||
|
||||
## [2026-10-31] audit | Nightly wiki audit — window 2026-10-30 00:00 → 2026-10-31 00:3x AWST
|
||||
|
||||
**Verdict: one large undocumented event found and written up, and one *documented* incident found to have been
|
||||
recorded wrong.**
|
||||
|
||||
- **NEW PAGE — [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]].** CT460 was dark for **5 h 30 m** on
|
||||
2026-10-30 and **no monitor alerted**. Evidence: **zero `executions.db` rows between 10:58:20 and 16:37:21**;
|
||||
PVE host `pve` boot time **2026-10-30 16:24:00**; CT460 boot **16:28:00**; the previous gateway life
|
||||
(`pid=3913841`, started 2026-10-28T17:21:58Z) logged **UNCLEANLY at 16:35:50** (*"no exit path ran —
|
||||
SIGKILL / OOM / VM death"*); scheduler resumed **16:37:21** with an **8-run catch-up burst** through 16:38:22.
|
||||
The day still closed **138 executions / 0 failed** against 10-29's **177 / 0** — **≈39 lost slots, no failures**.
|
||||
Root cause **not established**: no OOM evidence locally or on the PVE host, so the ledger string is read as a
|
||||
candidate list, not a diagnosis. **Still open:** the two jobs that could have detected it (`cbfe434a3b5e` NFS
|
||||
mount monitor, `1a8e90ea2271` m93p LXC boot monitor) are disabled.
|
||||
- **CORRECTED — [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] was right about the record and wrong
|
||||
about the run.** `ea79a2107e36`'s 2026-10-30 00:01:08 slot was **still running** at the 03:3x reading; it closed
|
||||
**`completed` at 10:02:48** after **10 h 01 m 40 s** and **delivered**, auto-closing incident
|
||||
`ea79a2_d93c3acdc4f2` **`resolved` at 2026-10-30T10:02:48.662+08:00**. Live job state is now `enabled: true`,
|
||||
`last_status: ok`, `failure_streak: 0`. The page's *"zero `deliveries.db` rows / the notice was never
|
||||
delivered"* claim was an artefact of reading the day at 03:3x: 2026-10-30 finished with **8 delivery rows, all
|
||||
`delivered`** (latest 18:46:04). Page frontmatter moved `active` → **`resolved`**, with the old snapshot kept
|
||||
below as explicitly historical. ⚠ Its **12:01 slot was lost to the host outage** and is the still-`running`
|
||||
16:37:21 catch-up. **Reconciliation lesson: a slot mid-flight is not a failure — check `finished_at`, not
|
||||
`last_status`, before writing an incident.**
|
||||
- **Reconciled — `systems/scheduled-tasks.md`.** Fleet **29 jobs (26 enabled / 3 disabled)** — the "4th disabled
|
||||
job" reading is superseded (frontmatter date, fleet block, Total jobs, non-clean `last_status` **4 → 3**,
|
||||
failing **1 → 0**, the health-monitor table row, the disabled table row, and a new **"Resolved 2026-10-30"**
|
||||
subsection). Non-clean now **3 = 1 `error` + 2 `delivery_failed`**; unclosed incident rows **2 → 1** (survivor:
|
||||
the Facebook monthly's `0e753e_e7f8194a7d06`, due 11-01).
|
||||
- **Reconciled** — `current-state.md` (new leading cron entry + superseded marker), `infrastructure/hosts.md`
|
||||
(cron row), `index.md` (frontmatter, banner, scheduled-tasks entry, both 10-30 incident entries),
|
||||
`incidents/2026-10-30-mnemosyne-health-monitor-paused.md`.
|
||||
- **Checked, unchanged:** 12 sessions active in-window (longest `local-40472e0b…`, 1,351 messages), no orphaned
|
||||
`running` rows, no incidents opened or closed after 10-30, no obsolete pages to archive, no secrets written.
|
||||
|
||||
Reference in New Issue
Block a user