Files
agent-estate-wiki/incidents/2026-10-30-ct460-host-outage-scheduler-gap.md

6.1 KiB
Raw Permalink Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30) incident resolved 2026-10-31 2026-10-31 2026-10-31 high
cron
gateway
host
outage
pve
ct460
incident
live-executions-db
live-jobs-json
live-gateway-exit-diag-log
live-agent-log
live-deliveries-db
ssh-pve-uptime

CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30)

Status: RESOLVED — the container and scheduler came back on 2026-10-30 ~16:28–16:37 and the day closed clean (138 executions, 136 completed, 0 failed). Root cause of the host reboot is not established. No monitor alerted; the gap was found by this audit.

Summary

The Hermes runtime went dark at ~10:58 AWST on 2026-10-30 and did not return until 16:28 (container) / 16:37:21 (first execution). The PVE host pve itself rebooted — its boot time is 2026-10-30 16:24:00 — so this was a host-level outage, not a cron or gateway fault. The gateway's previous life died with no exit path, which the lifecycle ledger logs as SIGKILL / OOM / VM death. Nothing noticed for 5 h 37 m.

Timeline (all times AWST, 2026-10-30)

Time Event Evidence
10:49:54 Last execution before the gap starts executions.db last row < 16:37
~10:58:20 Gateway life pid=3913841 dies; no exit path ran gateway-exit-diag.log gateway.previous_unclean_exit (prior life started 2026-10-28T17:21:58Z)
10:58:20 → 16:37:21 Zero execution rows (5 h 39 m) executions.db count in window = 0
16:24:00 PVE host pve (100.96.100.82) boots ssh root@100.96.100.82 'uptime -s; who -b' → both read 2026-10-30 16:24
16:28:00 CT460 container boots local uptime -s → 2026-10-30 16:28:00
16:35:50 Unclean exit noticed and logged logs/agent.log, logs/errors.log — Previous gateway life (pid=3913841, started_at=2026-10-28T17:21:58Z) exited UNCLEANLY (no exit path ran — SIGKILL / OOM / VM death)
16:37:21 Scheduler resumes — 8-run catch-up burst (16:37:21 → 16:38:22) executions.db
23:59:58 Daily wiki audit (this run) starts; the 16:37 health-monitor catch-up is still running executions.db

The resume burst (16:37:21 → 16:38:22)

Eight jobs fired within 61 seconds of the scheduler coming back. Seven closed completed; the eighth is still running.

Job Started Closed
FreshRSS Midday News Briefing (ecc1fc13655a) 16:37:21 17:45:31 — the 12:00 slot, delivered late
Mnemosyne Auto-Consolidate 16:37:21 16:37:54
Mnemosyne Embedding Coverage Monitor 16:37:21 16:41:31
Mnemosyne Watchdog 16:37:21 16:38:33
Mnemosyne Health Monitor (ea79a2107e36) 16:37:21 RUNNING at audit — the late 12:01 slot
Job Tracker Dashboard Refresh 16:37:23 16:37:35
🦝 Hermes Config Backup to Gitea (Daily) 16:38:06 16:42:02
Daily Marketing Job Radar - Write and Letters 16:38:22 16:41:39

Impact

  • ≈39 scheduled runs lost. 2026-10-29 fired 177 executions (177 completed, 0 failed); 2026-10-30 fired 138 (136 completed, 0 failed, 2 running — the 16:37 catch-up plus this audit). No row exists inside the dark window, so the deficit is unfired slots, not failures.
  • No failures were caused. Nothing was recorded failed on 2026-10-30 — the outage simply removed the opportunities to run.
  • The two long Mnemosyne runs absorbed it. The FreshRSS midday briefing regenerated on the late fire (16:37:21 → 17:45:31), and the health monitor's 12:01 slot is the still-running 16:37:21 fire — normal for this job, whose 00:01 slot took 10 h 01 m 40 s the same day.
  • Deliveries were unaffected once back up: deliveries.db holds 8 rows for 2026-10-30, all delivered (latest 18:46:04) — the opposite of the 03:3x snapshot, which read zero rows because the day had barely started.
  • 2026-10-31 opened clean: 5 executions (00:01:05 ×3, 00:15:59, 00:23:57), all completed.

Root cause — not established

  • The host rebooted. pve boot time is 16:24:00; no OOM evidence was found on the CT460 side (local dmesg grep empty, /var/log/wtmp absent, no suspected_oom marker in the webui session store).
  • The ledger's own wording (SIGKILL / OOM / VM death) is a list of candidates for "no exit path ran", not a finding. Treating it as a diagnosis would be wrong.
  • Same-day, different host: incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure records m93p reboots (~15:55) that took down NFS-backed containers. The two events are separate hosts and separate causes as far as the evidence goes — they are not merged here, but the timing is close enough to warrant checking the two hypervisors' last-boot records together before either is called a one-off.
  • To close it: PVE syslog / journalctl -b -1 on 100.96.100.82 around 2026-10-30 10:58, and the host's own last-boot records (last -x reboot). Neither was read for this page — say so rather than guess.

Why nothing alerted

The jobs positioned to notice a host-down event are disabled:

Job ID State
NFS mount monitor cbfe434a3b5e disabled
m93p LXC boot monitor 1a8e90ea2271 disabled

Everything else is inside the scheduler that went dark, so it could not have noticed. A host-level outage of the Hermes VM currently has no in-estate detector — the fleet learned about it from the 23:59 audit, 13 h after the fact. This is the incident's one open item.