--- title: CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30) type: incident status: resolved created: 2026-10-31 updated: 2026-10-31 verified_on: 2026-10-31 confidence: high tags: [cron, gateway, host, outage, pve, ct460, incident] sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-agent-log, live-deliveries-db, ssh-pve-uptime] --- # CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30) **Status: RESOLVED** — the container and scheduler came back on 2026-10-30 ~16:28–16:37 and the day closed clean (138 executions, 136 `completed`, **0 `failed`**). **Root cause of the host reboot is not established.** No monitor alerted; the gap was found by this audit. ## Summary The Hermes runtime went dark at **~10:58 AWST on 2026-10-30** and did not return until **16:28** (container) / **16:37:21** (first execution). The PVE host `pve` itself rebooted — its boot time is **2026-10-30 16:24:00** — so this was a **host-level outage, not a cron or gateway fault**. The gateway's previous life died with no exit path, which the lifecycle ledger logs as `SIGKILL / OOM / VM death`. Nothing noticed for **5 h 37 m**. ## Timeline (all times AWST, 2026-10-30) | Time | Event | Evidence | |---|---|---| | 10:49:54 | Last execution before the gap starts | `executions.db` last row `< 16:37` | | ~10:58:20 | Gateway life `pid=3913841` dies; no exit path ran | `gateway-exit-diag.log` `gateway.previous_unclean_exit` (prior life started 2026-10-28T17:21:58Z) | | 10:58:20 → 16:37:21 | **Zero execution rows** (5 h 39 m) | `executions.db` count in window = **0** | | 16:24:00 | PVE host `pve` (`100.96.100.82`) boots | `ssh root@100.96.100.82 'uptime -s; who -b'` → both read `2026-10-30 16:24` | | 16:28:00 | CT460 container boots | local `uptime -s` → `2026-10-30 16:28:00` | | 16:35:50 | Unclean exit noticed and logged | `logs/agent.log`, `logs/errors.log` — `Previous gateway life (pid=3913841, started_at=2026-10-28T17:21:58Z) exited UNCLEANLY (no exit path ran — SIGKILL / OOM / VM death)` | | **16:37:21** | Scheduler resumes — **8-run catch-up burst** (16:37:21 → 16:38:22) | `executions.db` | | 23:59:58 | Daily wiki audit (this run) starts; the 16:37 health-monitor catch-up is still `running` | `executions.db` | ## The resume burst (16:37:21 → 16:38:22) Eight jobs fired within 61 seconds of the scheduler coming back. Seven closed `completed`; the eighth is still running. | Job | Started | Closed | |---|---|---| | FreshRSS Midday News Briefing (`ecc1fc13655a`) | 16:37:21 | 17:45:31 — the 12:00 slot, delivered late | | Mnemosyne Auto-Consolidate | 16:37:21 | 16:37:54 | | Mnemosyne Embedding Coverage Monitor | 16:37:21 | 16:41:31 | | Mnemosyne Watchdog | 16:37:21 | 16:38:33 | | **Mnemosyne Health Monitor (`ea79a2107e36`)** | **16:37:21** | **RUNNING at audit** — the late 12:01 slot | | Job Tracker Dashboard Refresh | 16:37:23 | 16:37:35 | | 🦝 Hermes Config Backup to Gitea (Daily) | 16:38:06 | 16:42:02 | | Daily Marketing Job Radar - Write and Letters | 16:38:22 | 16:41:39 | ## Impact - **≈39 scheduled runs lost.** 2026-10-29 fired **177** executions (177 `completed`, 0 `failed`); 2026-10-30 fired **138** (136 `completed`, 0 `failed`, 2 `running` — the 16:37 catch-up plus this audit). No row exists inside the dark window, so the deficit is unfired slots, not failures. - **No failures were caused.** Nothing was recorded `failed` on 2026-10-30 — the outage simply removed the opportunities to run. - **The two long Mnemosyne runs absorbed it.** The FreshRSS midday briefing regenerated on the late fire (16:37:21 → 17:45:31), and the health monitor's 12:01 slot is the still-running 16:37:21 fire — normal for this job, whose 00:01 slot took **10 h 01 m 40 s** the same day. - **Deliveries were unaffected once back up:** `deliveries.db` holds **8 rows for 2026-10-30, all `delivered`** (latest 18:46:04) — the opposite of the 03:3x snapshot, which read zero rows because the day had barely started. - **2026-10-31 opened clean:** 5 executions (00:01:05 ×3, 00:15:59, 00:23:57), all `completed`. ## Root cause — not established - The host rebooted. `pve` boot time is 16:24:00; **no OOM evidence was found** on the CT460 side (local `dmesg` grep empty, `/var/log/wtmp` absent, no `suspected_oom` marker in the webui session store). - The ledger's own wording (`SIGKILL / OOM / VM death`) is a **list of candidates for "no exit path ran"**, not a finding. Treating it as a diagnosis would be wrong. - Same-day, different host: [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] records **m93p** reboots (~15:55) that took down NFS-backed containers. The two events are **separate hosts and separate causes as far as the evidence goes** — they are not merged here, but the timing is close enough to warrant checking the two hypervisors' last-boot records together before either is called a one-off. - **To close it:** PVE syslog / `journalctl -b -1` on `100.96.100.82` around 2026-10-30 10:58, and the host's own last-boot records (`last -x reboot`). Neither was read for this page — say so rather than guess. ## Why nothing alerted The jobs positioned to notice a host-down event are **disabled**: | Job | ID | State | |---|---|---| | NFS mount monitor | `cbfe434a3b5e` | disabled | | m93p LXC boot monitor | `1a8e90ea2271` | disabled | Everything else is *inside* the scheduler that went dark, so it could not have noticed. **A host-level outage of the Hermes VM currently has no in-estate detector** — the fleet learned about it from the 23:59 audit, 13 h after the fact. This is the incident's one open item. ## Related - [[systems/scheduled-tasks]] — the fleet entry for this window - [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — the same day's health-monitor story: its 00:01 slot closed `unknown` at 01:04:22, incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved` at 10:02:48** when that run finally completed, and the job is live again (`enabled: true`, `last_status: ok`, streak 0) - [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] — sister-host reboot, same day - [[infrastructure/hosts]] — `pve` / CT460 addressing