audit(2026-10-31): document CT460 5h30m host outage; correct the 'paused' health monitor (it was running, closed completed 10:02:48); reconcile fleet counts to 26/3

This commit is contained in:
hermes
2026-10-31 00:39:11 +08:00
parent db6af66878
commit 74a5b2d117
8 changed files with 241 additions and 27 deletions
+17 -3
View File
@@ -3,8 +3,8 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-10-30
verified_on: 2026-10-30
updated: 2026-10-31
verified_on: 2026-10-31
review_after: 2026-11-07
confidence: high
tags: [system]
@@ -43,7 +43,21 @@ Updated: 2026-10-30
**10 deliveries, all 10 `delivered`, 0 failures**. Non-clean `last_status` **5 → 3**; unclosed incident rows
**4 → 1** (the two radar rows closed `resolved` at 10:16:12). Fleet unchanged at **29 jobs (26 enabled / 3
disabled)** at that audit.
- **Cron fleet changed shape overnight on 2026-10-30 — the Mnemosyne Health Monitor is paused and its failure
- **Cron: 2026-10-30 closed 138 / 0 — after a 5 h 30 m HOST OUTAGE — and the "auto-paused" health monitor was never
dead, it was running.** Fleet **29 jobs (26 enabled / 3 disabled)**
(`jobs.json.updated_at 2026-10-31T00:29:59+08:00`), re-verified 2026-10-31 00:3x. **Zero execution rows exist
between 10:58:20 and 16:37:21** — PVE `pve` rebooted (boot **2026-10-30 16:24:00**), CT460 came back **16:28:00**,
the old gateway life (`pid=3913841`) was logged **UNCLEANLY** at **16:35:50** (*"no exit path ran — SIGKILL / OOM /
VM death"* — a candidate list, not a diagnosis; root cause **not established**, no OOM evidence), and 8 catch-up
runs fired at **16:37:21–16:38:22** → [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]. Against 10-29's
**177 / 0**, that is **≈39 lost slots and no failures**. **`ea79a2107e36` (Mnemosyne Health Monitor) is ENABLED
and clean again** — `last_status: ok`, streak **0**, incident `ea79a2_d93c3acdc4f2` closed **`resolved`
2026-10-30T10:02:48.662** when its 00:01:08 slot finished after **10 h 01 m 40 s**; its **12:01 slot was lost to
the outage** and is the still-`running` 16:37:21 catch-up. **Non-clean `last_status` 4 → 3**, **failing 1 → 0**,
**unclosed incident rows 2 → 1** (survivor: `0e753e_e7f8194a7d06`, due 11-01). **`deliveries.db` finished 10-30
with 8 rows, all `delivered`** (latest 18:46:04) — the 03:3x "zero rows / notice never delivered" reading is
superseded.
- **⚠ Superseded by the 2026-10-31 entry above — read that first.** Cron fleet changed shape overnight on 2026-10-30 — the Mnemosyne Health Monitor is paused and its failure
notice was never delivered.** `ea79a2107e36`'s 00:01:08 slot closed **`unknown`** at 01:04:22 on a
gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`:
Gateway connection failed; no local fallback"*), opening incident `ea79a2_d93c3acdc4f2` (`detected`), and the