CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30)
incident
resolved
2026-10-31
2026-10-31
2026-10-31
high
cron
gateway
host
outage
pve
ct460
incident
live-executions-db
live-jobs-json
live-gateway-exit-diag-log
live-agent-log
live-deliveries-db
ssh-pve-uptime
CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30)
Status: RESOLVED — the container and scheduler came back on 2026-10-30 ~16:28–16:37 and the day
closed clean (138 executions, 136 completed, 0 failed). Root cause of the host reboot is not
established. No monitor alerted; the gap was found by this audit.
Summary
The Hermes runtime went dark at ~10:58 AWST on 2026-10-30 and did not return until
16:28 (container) / 16:37:21 (first execution). The PVE host pve itself rebooted — its boot
time is 2026-10-30 16:24:00 — so this was a host-level outage, not a cron or gateway fault.
The gateway's previous life died with no exit path, which the lifecycle ledger logs as
SIGKILL / OOM / VM death. Nothing noticed for 5 h 37 m.
Timeline (all times AWST, 2026-10-30)
Time
Event
Evidence
10:49:54
Last execution before the gap starts
executions.db last row < 16:37
~10:58:20
Gateway life pid=3913841 dies; no exit path ran
gateway-exit-diag.loggateway.previous_unclean_exit (prior life started 2026-10-28T17:21:58Z)
10:58:20 → 16:37:21
Zero execution rows (5 h 39 m)
executions.db count in window = 0
16:24:00
PVE host pve (100.96.100.82) boots
ssh root@100.96.100.82 'uptime -s; who -b' → both read 2026-10-30 16:24
16:28:00
CT460 container boots
local uptime -s → 2026-10-30 16:28:00
16:35:50
Unclean exit noticed and logged
logs/agent.log, logs/errors.log — Previous gateway life (pid=3913841, started_at=2026-10-28T17:21:58Z) exited UNCLEANLY (no exit path ran — SIGKILL / OOM / VM death)
Daily wiki audit (this run) starts; the 16:37 health-monitor catch-up is still running
executions.db
The resume burst (16:37:21 → 16:38:22)
Eight jobs fired within 61 seconds of the scheduler coming back. Seven closed completed;
the eighth is still running.
Job
Started
Closed
FreshRSS Midday News Briefing (ecc1fc13655a)
16:37:21
17:45:31 — the 12:00 slot, delivered late
Mnemosyne Auto-Consolidate
16:37:21
16:37:54
Mnemosyne Embedding Coverage Monitor
16:37:21
16:41:31
Mnemosyne Watchdog
16:37:21
16:38:33
Mnemosyne Health Monitor (ea79a2107e36)
16:37:21
RUNNING at audit — the late 12:01 slot
Job Tracker Dashboard Refresh
16:37:23
16:37:35
🦝 Hermes Config Backup to Gitea (Daily)
16:38:06
16:42:02
Daily Marketing Job Radar - Write and Letters
16:38:22
16:41:39
Impact
≈39 scheduled runs lost. 2026-10-29 fired 177 executions (177 completed, 0 failed); 2026-10-30
fired 138 (136 completed, 0 failed, 2 running — the 16:37 catch-up plus this audit). No row exists
inside the dark window, so the deficit is unfired slots, not failures.
No failures were caused. Nothing was recorded failed on 2026-10-30 — the outage simply removed the
opportunities to run.
The two long Mnemosyne runs absorbed it. The FreshRSS midday briefing regenerated on the late fire
(16:37:21 → 17:45:31), and the health monitor's 12:01 slot is the still-running 16:37:21 fire — normal for
this job, whose 00:01 slot took 10 h 01 m 40 s the same day.
Deliveries were unaffected once back up:deliveries.db holds 8 rows for 2026-10-30, all delivered
(latest 18:46:04) — the opposite of the 03:3x snapshot, which read zero rows because the day had barely started.
2026-10-31 opened clean: 5 executions (00:01:05 ×3, 00:15:59, 00:23:57), all completed.
Root cause — not established
The host rebooted. pve boot time is 16:24:00; no OOM evidence was found on the CT460 side (local
dmesg grep empty, /var/log/wtmp absent, no suspected_oom marker in the webui session store).
The ledger's own wording (SIGKILL / OOM / VM death) is a list of candidates for "no exit path ran", not a
finding. Treating it as a diagnosis would be wrong.
Same-day, different host: incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure records m93p
reboots (~15:55) that took down NFS-backed containers. The two events are separate hosts and separate
causes as far as the evidence goes — they are not merged here, but the timing is close enough to warrant
checking the two hypervisors' last-boot records together before either is called a one-off.
To close it: PVE syslog / journalctl -b -1 on 100.96.100.82 around 2026-10-30 10:58, and the host's
own last-boot records (last -x reboot). Neither was read for this page — say so rather than guess.
Why nothing alerted
The jobs positioned to notice a host-down event are disabled:
Job
ID
State
NFS mount monitor
cbfe434a3b5e
disabled
m93p LXC boot monitor
1a8e90ea2271
disabled
Everything else is inside the scheduler that went dark, so it could not have noticed. A host-level outage of
the Hermes VM currently has no in-estate detector — the fleet learned about it from the 23:59 audit, 13 h
after the fact. This is the incident's one open item.
incidents/2026-10-30-mnemosyne-health-monitor-paused — the same day's health-monitor story: its 00:01 slot
closed unknown at 01:04:22, incident ea79a2_d93c3acdc4f2auto-closed resolved at 10:02:48 when that
run finally completed, and the job is live again (enabled: true, last_status: ok, streak 0)