6.5 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30) | incident | resolved | 2026-10-31 | 2026-11-21 | 2026-11-21 | high |
|
|
CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30)
Status: RESOLVED — the container and scheduler came back on 2026-10-30 ~16:28–16:37 and the day
closed clean (138 executions, 136 completed, 0 failed). Root cause of the host reboot is not
established. No monitor alerted; the gap was found by this audit.
Re-verification — 2026-11-21 (stocktake)
CT460's host has rebooted once more since this incident: current uptime -s reads
2026-11-10 22:03:51, matching the 2026-11-10 unclean gateway exit in gateway.log
(previous life pid=3109529, started 2026-11-07, SIGKILL/OOM-class exit, suspected_oom=false).
This page's 2026-10-30 window stands as recorded; later scheduler gaps were recorded as separate
incidents.
Summary
The Hermes runtime went dark at ~10:58 AWST on 2026-10-30 and did not return until
16:28 (container) / 16:37:21 (first execution). The PVE host pve itself rebooted — its boot
time is 2026-10-30 16:24:00 — so this was a host-level outage, not a cron or gateway fault.
The gateway's previous life died with no exit path, which the lifecycle ledger logs as
SIGKILL / OOM / VM death. Nothing noticed for 5 h 37 m.
Timeline (all times AWST, 2026-10-30)
| Time | Event | Evidence |
|---|---|---|
| 10:49:54 | Last execution before the gap starts | executions.db last row < 16:37 |
| ~10:58:20 | Gateway life pid=3913841 dies; no exit path ran |
gateway-exit-diag.log gateway.previous_unclean_exit (prior life started 2026-10-28T17:21:58Z) |
| 10:58:20 → 16:37:21 | Zero execution rows (5 h 39 m) | executions.db count in window = 0 |
| 16:24:00 | PVE host pve (100.96.100.82) boots |
ssh root@100.96.100.82 'uptime -s; who -b' → both read 2026-10-30 16:24 |
| 16:28:00 | CT460 container boots | local uptime -s → 2026-10-30 16:28:00 |
| 16:35:50 | Unclean exit noticed and logged | logs/agent.log, logs/errors.log — Previous gateway life (pid=3913841, started_at=2026-10-28T17:21:58Z) exited UNCLEANLY (no exit path ran — SIGKILL / OOM / VM death) |
| 16:37:21 | Scheduler resumes — 8-run catch-up burst (16:37:21 → 16:38:22) | executions.db |
| 23:59:58 | Daily wiki audit (this run) starts; the 16:37 health-monitor catch-up is still running |
executions.db |
The resume burst (16:37:21 → 16:38:22)
Eight jobs fired within 61 seconds of the scheduler coming back. Seven closed completed;
the eighth is still running.
| Job | Started | Closed |
|---|---|---|
FreshRSS Midday News Briefing (ecc1fc13655a) |
16:37:21 | 17:45:31 — the 12:00 slot, delivered late |
| Mnemosyne Auto-Consolidate | 16:37:21 | 16:37:54 |
| Mnemosyne Embedding Coverage Monitor | 16:37:21 | 16:41:31 |
| Mnemosyne Watchdog | 16:37:21 | 16:38:33 |
Mnemosyne Health Monitor (ea79a2107e36) |
16:37:21 | RUNNING at audit — the late 12:01 slot |
| Job Tracker Dashboard Refresh | 16:37:23 | 16:37:35 |
| 🦝 Hermes Config Backup to Gitea (Daily) | 16:38:06 | 16:42:02 |
| Daily Marketing Job Radar - Write and Letters | 16:38:22 | 16:41:39 |
Impact
- ≈39 scheduled runs lost. 2026-10-29 fired 177 executions (177
completed, 0failed); 2026-10-30 fired 138 (136completed, 0failed, 2running— the 16:37 catch-up plus this audit). No row exists inside the dark window, so the deficit is unfired slots, not failures. - No failures were caused. Nothing was recorded
failedon 2026-10-30 — the outage simply removed the opportunities to run. - The two long Mnemosyne runs absorbed it. The FreshRSS midday briefing regenerated on the late fire (16:37:21 → 17:45:31), and the health monitor's 12:01 slot is the still-running 16:37:21 fire — normal for this job, whose 00:01 slot took 10 h 01 m 40 s the same day.
- Deliveries were unaffected once back up:
deliveries.dbholds 8 rows for 2026-10-30, alldelivered(latest 18:46:04) — the opposite of the 03:3x snapshot, which read zero rows because the day had barely started. - 2026-10-31 opened clean: 5 executions (00:01:05 ×3, 00:15:59, 00:23:57), all
completed.
Root cause — not established
- The host rebooted.
pveboot time is 16:24:00; no OOM evidence was found on the CT460 side (localdmesggrep empty,/var/log/wtmpabsent, nosuspected_oommarker in the webui session store). - The ledger's own wording (
SIGKILL / OOM / VM death) is a list of candidates for "no exit path ran", not a finding. Treating it as a diagnosis would be wrong. - Same-day, different host: incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure records m93p reboots (~15:55) that took down NFS-backed containers. The two events are separate hosts and separate causes as far as the evidence goes — they are not merged here, but the timing is close enough to warrant checking the two hypervisors' last-boot records together before either is called a one-off.
- To close it: PVE syslog /
journalctl -b -1on100.96.100.82around 2026-10-30 10:58, and the host's own last-boot records (last -x reboot). Neither was read for this page — say so rather than guess.
Why nothing alerted
The jobs positioned to notice a host-down event are disabled:
| Job | ID | State |
|---|---|---|
| NFS mount monitor | cbfe434a3b5e |
disabled |
| m93p LXC boot monitor | 1a8e90ea2271 |
disabled |
Everything else is inside the scheduler that went dark, so it could not have noticed. A host-level outage of the Hermes VM currently has no in-estate detector — the fleet learned about it from the 23:59 audit, 13 h after the fact. This is the incident's one open item.
Related
- systems/scheduled-tasks — the fleet entry for this window
- incidents/2026-10-30-mnemosyne-health-monitor-paused — the same day's health-monitor story: its 00:01 slot
closed
unknownat 01:04:22, incidentea79a2_d93c3acdc4f2auto-closedresolvedat 10:02:48 when that run finally completed, and the job is live again (enabled: true,last_status: ok, streak 0) - incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure — sister-host reboot, same day
- infrastructure/hosts —
pve/ CT460 addressing