audit(2026-10-31): document CT460 5h30m host outage; correct the 'paused' health monitor (it was running, closed completed 10:02:48); reconcile fleet counts to 26/3
This commit is contained in:
@@ -0,0 +1,103 @@
|
||||
---
|
||||
title: CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30)
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-31
|
||||
updated: 2026-10-31
|
||||
verified_on: 2026-10-31
|
||||
confidence: high
|
||||
tags: [cron, gateway, host, outage, pve, ct460, incident]
|
||||
sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-agent-log, live-deliveries-db, ssh-pve-uptime]
|
||||
---
|
||||
|
||||
# CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30)
|
||||
|
||||
**Status: RESOLVED** — the container and scheduler came back on 2026-10-30 ~16:28–16:37 and the day
|
||||
closed clean (138 executions, 136 `completed`, **0 `failed`**). **Root cause of the host reboot is not
|
||||
established.** No monitor alerted; the gap was found by this audit.
|
||||
|
||||
## Summary
|
||||
|
||||
The Hermes runtime went dark at **~10:58 AWST on 2026-10-30** and did not return until
|
||||
**16:28** (container) / **16:37:21** (first execution). The PVE host `pve` itself rebooted — its boot
|
||||
time is **2026-10-30 16:24:00** — so this was a **host-level outage, not a cron or gateway fault**.
|
||||
The gateway's previous life died with no exit path, which the lifecycle ledger logs as
|
||||
`SIGKILL / OOM / VM death`. Nothing noticed for **5 h 37 m**.
|
||||
|
||||
## Timeline (all times AWST, 2026-10-30)
|
||||
|
||||
| Time | Event | Evidence |
|
||||
|---|---|---|
|
||||
| 10:49:54 | Last execution before the gap starts | `executions.db` last row `< 16:37` |
|
||||
| ~10:58:20 | Gateway life `pid=3913841` dies; no exit path ran | `gateway-exit-diag.log` `gateway.previous_unclean_exit` (prior life started 2026-10-28T17:21:58Z) |
|
||||
| 10:58:20 → 16:37:21 | **Zero execution rows** (5 h 39 m) | `executions.db` count in window = **0** |
|
||||
| 16:24:00 | PVE host `pve` (`100.96.100.82`) boots | `ssh root@100.96.100.82 'uptime -s; who -b'` → both read `2026-10-30 16:24` |
|
||||
| 16:28:00 | CT460 container boots | local `uptime -s` → `2026-10-30 16:28:00` |
|
||||
| 16:35:50 | Unclean exit noticed and logged | `logs/agent.log`, `logs/errors.log` — `Previous gateway life (pid=3913841, started_at=2026-10-28T17:21:58Z) exited UNCLEANLY (no exit path ran — SIGKILL / OOM / VM death)` |
|
||||
| **16:37:21** | Scheduler resumes — **8-run catch-up burst** (16:37:21 → 16:38:22) | `executions.db` |
|
||||
| 23:59:58 | Daily wiki audit (this run) starts; the 16:37 health-monitor catch-up is still `running` | `executions.db` |
|
||||
|
||||
## The resume burst (16:37:21 → 16:38:22)
|
||||
|
||||
Eight jobs fired within 61 seconds of the scheduler coming back. Seven closed `completed`;
|
||||
the eighth is still running.
|
||||
|
||||
| Job | Started | Closed |
|
||||
|---|---|---|
|
||||
| FreshRSS Midday News Briefing (`ecc1fc13655a`) | 16:37:21 | 17:45:31 — the 12:00 slot, delivered late |
|
||||
| Mnemosyne Auto-Consolidate | 16:37:21 | 16:37:54 |
|
||||
| Mnemosyne Embedding Coverage Monitor | 16:37:21 | 16:41:31 |
|
||||
| Mnemosyne Watchdog | 16:37:21 | 16:38:33 |
|
||||
| **Mnemosyne Health Monitor (`ea79a2107e36`)** | **16:37:21** | **RUNNING at audit** — the late 12:01 slot |
|
||||
| Job Tracker Dashboard Refresh | 16:37:23 | 16:37:35 |
|
||||
| 🦝 Hermes Config Backup to Gitea (Daily) | 16:38:06 | 16:42:02 |
|
||||
| Daily Marketing Job Radar - Write and Letters | 16:38:22 | 16:41:39 |
|
||||
|
||||
## Impact
|
||||
|
||||
- **≈39 scheduled runs lost.** 2026-10-29 fired **177** executions (177 `completed`, 0 `failed`); 2026-10-30
|
||||
fired **138** (136 `completed`, 0 `failed`, 2 `running` — the 16:37 catch-up plus this audit). No row exists
|
||||
inside the dark window, so the deficit is unfired slots, not failures.
|
||||
- **No failures were caused.** Nothing was recorded `failed` on 2026-10-30 — the outage simply removed the
|
||||
opportunities to run.
|
||||
- **The two long Mnemosyne runs absorbed it.** The FreshRSS midday briefing regenerated on the late fire
|
||||
(16:37:21 → 17:45:31), and the health monitor's 12:01 slot is the still-running 16:37:21 fire — normal for
|
||||
this job, whose 00:01 slot took **10 h 01 m 40 s** the same day.
|
||||
- **Deliveries were unaffected once back up:** `deliveries.db` holds **8 rows for 2026-10-30, all `delivered`**
|
||||
(latest 18:46:04) — the opposite of the 03:3x snapshot, which read zero rows because the day had barely started.
|
||||
- **2026-10-31 opened clean:** 5 executions (00:01:05 ×3, 00:15:59, 00:23:57), all `completed`.
|
||||
|
||||
## Root cause — not established
|
||||
|
||||
- The host rebooted. `pve` boot time is 16:24:00; **no OOM evidence was found** on the CT460 side (local
|
||||
`dmesg` grep empty, `/var/log/wtmp` absent, no `suspected_oom` marker in the webui session store).
|
||||
- The ledger's own wording (`SIGKILL / OOM / VM death`) is a **list of candidates for "no exit path ran"**, not a
|
||||
finding. Treating it as a diagnosis would be wrong.
|
||||
- Same-day, different host: [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] records **m93p**
|
||||
reboots (~15:55) that took down NFS-backed containers. The two events are **separate hosts and separate
|
||||
causes as far as the evidence goes** — they are not merged here, but the timing is close enough to warrant
|
||||
checking the two hypervisors' last-boot records together before either is called a one-off.
|
||||
- **To close it:** PVE syslog / `journalctl -b -1` on `100.96.100.82` around 2026-10-30 10:58, and the host's
|
||||
own last-boot records (`last -x reboot`). Neither was read for this page — say so rather than guess.
|
||||
|
||||
## Why nothing alerted
|
||||
|
||||
The jobs positioned to notice a host-down event are **disabled**:
|
||||
|
||||
| Job | ID | State |
|
||||
|---|---|---|
|
||||
| NFS mount monitor | `cbfe434a3b5e` | disabled |
|
||||
| m93p LXC boot monitor | `1a8e90ea2271` | disabled |
|
||||
|
||||
Everything else is *inside* the scheduler that went dark, so it could not have noticed. **A host-level outage of
|
||||
the Hermes VM currently has no in-estate detector** — the fleet learned about it from the 23:59 audit, 13 h
|
||||
after the fact. This is the incident's one open item.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]] — the fleet entry for this window
|
||||
- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — the same day's health-monitor story: its 00:01 slot
|
||||
closed `unknown` at 01:04:22, incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved` at 10:02:48** when that
|
||||
run finally completed, and the job is live again (`enabled: true`, `last_status: ok`, streak 0)
|
||||
- [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] — sister-host reboot, same day
|
||||
- [[infrastructure/hosts]] — `pve` / CT460 addressing
|
||||
@@ -1,10 +1,10 @@
|
||||
---
|
||||
title: Mnemosyne Health Monitor auto-paused after gateway-connection failure
|
||||
type: incident
|
||||
status: active
|
||||
status: resolved
|
||||
created: 2026-10-30
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
updated: 2026-10-31
|
||||
verified_on: 2026-10-31
|
||||
confidence: high
|
||||
tags: [cron, mnemosyne, gateway, incident, delivery]
|
||||
sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents]
|
||||
@@ -12,7 +12,17 @@ sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-inci
|
||||
|
||||
# Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30)
|
||||
|
||||
**Status: ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
|
||||
**Status: RESOLVED — 2026-10-30T10:02:48.662+08:00** (updated by the 2026-10-31 audit; everything below is the
|
||||
2026-10-30 03:3x snapshot and its "job is disabled / notice never delivered" reading did not survive the day).
|
||||
**What actually happened:** the 00:01:08 slot was **still running**, not dead — it closed **`completed` at
|
||||
10:02:48** after **10 h 01 m 40 s** and **delivered**; incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved`
|
||||
at 2026-10-30T10:02:48.662+08:00**, and the job is live again (`enabled: true`, `last_status: ok`,
|
||||
`failure_streak: 0`). `deliveries.db` finished 2026-10-30 with **8 rows, all `delivered`** (latest 18:46:04) — the
|
||||
zero-row reading was an artefact of the day having barely begun. ⚠ Its **12:01 slot was then lost to the CT460
|
||||
host outage** and fired late in the 16:37:21 catch-up burst, still `running` at the 2026-10-31 audit — see
|
||||
[[incidents/2026-10-30-ct460-host-outage-scheduler-gap]].
|
||||
|
||||
**Historical snapshot (2026-10-30 03:3x): ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
|
||||
|
||||
## Summary
|
||||
|
||||
|
||||
Reference in New Issue
Block a user