audit(2026-10-31): document CT460 5h30m host outage; correct the 'paused' health monitor (it was running, closed completed 10:02:48); reconcile fleet counts to 26/3

This commit is contained in:
hermes
2026-10-31 00:39:11 +08:00
parent db6af66878
commit 74a5b2d117
8 changed files with 241 additions and 27 deletions
@@ -0,0 +1,103 @@
---
title: CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30)
type: incident
status: resolved
created: 2026-10-31
updated: 2026-10-31
verified_on: 2026-10-31
confidence: high
tags: [cron, gateway, host, outage, pve, ct460, incident]
sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-agent-log, live-deliveries-db, ssh-pve-uptime]
---
# CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30)
**Status: RESOLVED** — the container and scheduler came back on 2026-10-30 ~16:28–16:37 and the day
closed clean (138 executions, 136 `completed`, **0 `failed`**). **Root cause of the host reboot is not
established.** No monitor alerted; the gap was found by this audit.
## Summary
The Hermes runtime went dark at **~10:58 AWST on 2026-10-30** and did not return until
**16:28** (container) / **16:37:21** (first execution). The PVE host `pve` itself rebooted — its boot
time is **2026-10-30 16:24:00** — so this was a **host-level outage, not a cron or gateway fault**.
The gateway's previous life died with no exit path, which the lifecycle ledger logs as
`SIGKILL / OOM / VM death`. Nothing noticed for **5 h 37 m**.
## Timeline (all times AWST, 2026-10-30)
| Time | Event | Evidence |
|---|---|---|
| 10:49:54 | Last execution before the gap starts | `executions.db` last row `< 16:37` |
| ~10:58:20 | Gateway life `pid=3913841` dies; no exit path ran | `gateway-exit-diag.log` `gateway.previous_unclean_exit` (prior life started 2026-10-28T17:21:58Z) |
| 10:58:20 → 16:37:21 | **Zero execution rows** (5 h 39 m) | `executions.db` count in window = **0** |
| 16:24:00 | PVE host `pve` (`100.96.100.82`) boots | `ssh root@100.96.100.82 'uptime -s; who -b'` → both read `2026-10-30 16:24` |
| 16:28:00 | CT460 container boots | local `uptime -s` → `2026-10-30 16:28:00` |
| 16:35:50 | Unclean exit noticed and logged | `logs/agent.log`, `logs/errors.log` — `Previous gateway life (pid=3913841, started_at=2026-10-28T17:21:58Z) exited UNCLEANLY (no exit path ran — SIGKILL / OOM / VM death)` |
| **16:37:21** | Scheduler resumes — **8-run catch-up burst** (16:37:21 → 16:38:22) | `executions.db` |
| 23:59:58 | Daily wiki audit (this run) starts; the 16:37 health-monitor catch-up is still `running` | `executions.db` |
## The resume burst (16:37:21 → 16:38:22)
Eight jobs fired within 61 seconds of the scheduler coming back. Seven closed `completed`;
the eighth is still running.
| Job | Started | Closed |
|---|---|---|
| FreshRSS Midday News Briefing (`ecc1fc13655a`) | 16:37:21 | 17:45:31 — the 12:00 slot, delivered late |
| Mnemosyne Auto-Consolidate | 16:37:21 | 16:37:54 |
| Mnemosyne Embedding Coverage Monitor | 16:37:21 | 16:41:31 |
| Mnemosyne Watchdog | 16:37:21 | 16:38:33 |
| **Mnemosyne Health Monitor (`ea79a2107e36`)** | **16:37:21** | **RUNNING at audit** — the late 12:01 slot |
| Job Tracker Dashboard Refresh | 16:37:23 | 16:37:35 |
| 🦝 Hermes Config Backup to Gitea (Daily) | 16:38:06 | 16:42:02 |
| Daily Marketing Job Radar - Write and Letters | 16:38:22 | 16:41:39 |
## Impact
- **≈39 scheduled runs lost.** 2026-10-29 fired **177** executions (177 `completed`, 0 `failed`); 2026-10-30
fired **138** (136 `completed`, 0 `failed`, 2 `running` — the 16:37 catch-up plus this audit). No row exists
inside the dark window, so the deficit is unfired slots, not failures.
- **No failures were caused.** Nothing was recorded `failed` on 2026-10-30 — the outage simply removed the
opportunities to run.
- **The two long Mnemosyne runs absorbed it.** The FreshRSS midday briefing regenerated on the late fire
(16:37:21 → 17:45:31), and the health monitor's 12:01 slot is the still-running 16:37:21 fire — normal for
this job, whose 00:01 slot took **10 h 01 m 40 s** the same day.
- **Deliveries were unaffected once back up:** `deliveries.db` holds **8 rows for 2026-10-30, all `delivered`**
(latest 18:46:04) — the opposite of the 03:3x snapshot, which read zero rows because the day had barely started.
- **2026-10-31 opened clean:** 5 executions (00:01:05 ×3, 00:15:59, 00:23:57), all `completed`.
## Root cause — not established
- The host rebooted. `pve` boot time is 16:24:00; **no OOM evidence was found** on the CT460 side (local
`dmesg` grep empty, `/var/log/wtmp` absent, no `suspected_oom` marker in the webui session store).
- The ledger's own wording (`SIGKILL / OOM / VM death`) is a **list of candidates for "no exit path ran"**, not a
finding. Treating it as a diagnosis would be wrong.
- Same-day, different host: [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] records **m93p**
reboots (~15:55) that took down NFS-backed containers. The two events are **separate hosts and separate
causes as far as the evidence goes** — they are not merged here, but the timing is close enough to warrant
checking the two hypervisors' last-boot records together before either is called a one-off.
- **To close it:** PVE syslog / `journalctl -b -1` on `100.96.100.82` around 2026-10-30 10:58, and the host's
own last-boot records (`last -x reboot`). Neither was read for this page — say so rather than guess.
## Why nothing alerted
The jobs positioned to notice a host-down event are **disabled**:
| Job | ID | State |
|---|---|---|
| NFS mount monitor | `cbfe434a3b5e` | disabled |
| m93p LXC boot monitor | `1a8e90ea2271` | disabled |
Everything else is *inside* the scheduler that went dark, so it could not have noticed. **A host-level outage of
the Hermes VM currently has no in-estate detector** — the fleet learned about it from the 23:59 audit, 13 h
after the fact. This is the incident's one open item.
## Related
- [[systems/scheduled-tasks]] — the fleet entry for this window
- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — the same day's health-monitor story: its 00:01 slot
closed `unknown` at 01:04:22, incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved` at 10:02:48** when that
run finally completed, and the job is live again (`enabled: true`, `last_status: ok`, streak 0)
- [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] — sister-host reboot, same day
- [[infrastructure/hosts]] — `pve` / CT460 addressing
@@ -1,10 +1,10 @@
---
title: Mnemosyne Health Monitor auto-paused after gateway-connection failure
type: incident
status: active
status: resolved
created: 2026-10-30
updated: 2026-10-30
verified_on: 2026-10-30
updated: 2026-10-31
verified_on: 2026-10-31
confidence: high
tags: [cron, mnemosyne, gateway, incident, delivery]
sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents]
@@ -12,7 +12,17 @@ sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-inci
# Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30)
**Status: ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
**Status: RESOLVED — 2026-10-30T10:02:48.662+08:00** (updated by the 2026-10-31 audit; everything below is the
2026-10-30 03:3x snapshot and its "job is disabled / notice never delivered" reading did not survive the day).
**What actually happened:** the 00:01:08 slot was **still running**, not dead — it closed **`completed` at
10:02:48** after **10 h 01 m 40 s** and **delivered**; incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved`
at 2026-10-30T10:02:48.662+08:00**, and the job is live again (`enabled: true`, `last_status: ok`,
`failure_streak: 0`). `deliveries.db` finished 2026-10-30 with **8 rows, all `delivered`** (latest 18:46:04) — the
zero-row reading was an artefact of the day having barely begun. ⚠ Its **12:01 slot was then lost to the CT460
host outage** and fired late in the 16:37:21 catch-up burst, still `running` at the 2026-10-31 audit — see
[[incidents/2026-10-30-ct460-host-outage-scheduler-gap]].
**Historical snapshot (2026-10-30 03:3x): ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
## Summary