audit: 2026-10-26 nightly — planned restart orphaned a live cron run (new incident page); 503 burst fully closed; radar cover-letter PDF pipeline moved to Chromium-over-HTML

This commit is contained in:
Tony0410
2026-10-27 00:11:50 +08:00
parent 3f796dc3b9
commit a2dd0bd650
8 changed files with 236 additions and 20 deletions
@@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu
type: incident
status: monitoring
created: 2026-10-18
updated: 2026-10-22
verified_on: 2026-10-22
review_after: 2026-10-29
updated: 2026-10-27
verified_on: 2026-10-27
review_after: 2026-11-03
confidence: high
tags: [cron, router, provider, infrastructure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
@@ -28,6 +28,13 @@ days** (10-16 → 10-17 and 10-19) with an identical signature on an identical n
> most of the fleet. Two bursts four days apart, both self-clearing, is the pattern to watch; this is a standing
> risk, not a resolved incident.
> **2026-10-26 update — every incident row from this failure mode is now closed.** The last open row,
> `c0a9fa_764558b5ae83` (`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.
> Retry shortly.`, job `c0a9fa314542` GitHub Trending Radar, `alerted` since 2026-10-19 09:02), **auto-closed
> 2026-10-26 09:13:00** when that job's 09:00 slot ran clean. **No execution row carries the resource-pressure
> string on 2026-10-25 or 2026-10-26.** Status stays **`monitoring`** for the same reason as 10-21: the rows are
> shut, the single-node exposure is not.
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
@@ -3,9 +3,9 @@ title: Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status
type: incident
status: active
created: 2026-10-21
updated: 2026-10-21
verified_on: 2026-10-21
review_after: 2026-10-28
updated: 2026-10-27
verified_on: 2026-10-27
review_after: 2026-11-03
confidence: high
tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure]
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
@@ -87,6 +87,7 @@ sqlite3 ~/.hermes/.hermes/cron/deliveries.db \
## Related
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **recurrence on 2026-10-26, and the loud-orphan gap here is now partly closed.** This one was a **planned `SIGTERM`** (not an unclean exit) and the drain still **timed out after 1.35 s**, interrupting `ea79a2107e36` (Mnemosyne Health Monitor) 2 h 34 m in. Unlike this page's case it closed **`failed`** on a **new** string (`lost its durable fire claim ownership`), **raised two `cron_incidents` rows**, and **delivered** the failure — but the work still did not happen. Any triage keyed on this page's `unknown` / "owner exited before a durable terminal state" wording will miss it.
- [[systems/scheduled-tasks]]
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — the other 2026-10-20 non-delivery (a canned substitute served to a user).
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the other "the row is green, the work did not happen" case found in this audit.
@@ -0,0 +1,103 @@
---
title: Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in)
type: incident
status: active
created: 2026-10-27
updated: 2026-10-27
verified_on: 2026-10-27
review_after: 2026-11-03
confidence: high
tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure]
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-probe]
---
# Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in)
## Headline
On **2026-10-26 at 14:33:35 AWST** the gateway received a **planned `SIGTERM`** (`under_systemd=yes`,
`parent_name=systemd`) — and the shutdown drain **still interrupted a running cron execution**, because
the drain reported `timed_out=True` after only **1.35 s**. The orphaned run was
`ea79a2107e36` **Mnemosyne Health Monitor** (12-hourly `0 */12 * * *`), which had been claimed at
**12:01:13** and was **2 h 34 m** into its work.
This is the **second instance** of the orphaned-cron-run class documented for 2026-10-20
([[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]), with three material differences:
| | 2026-10-20 | 2026-10-26 |
|---|---|---|
| Stop type | **unclean exit** (no exit path, `suspected_oom=False`) | **planned `SIGTERM`** under systemd |
| Execution closed as | `unknown` ("owner exited before a durable terminal state") | **`failed`** with a **new** error string |
| Incident row raised | **no** | **yes — two** |
| Failure delivered | no | **yes**, 14:35:59 |
So the loud-orphan recommendation from the 10-20 page is **now partly implemented** — the run no longer
disappears silently. What it did not do is *finish the work* or *avoid the interruption*.
## Timeline (all times AWST, 2026-10-26)
| Time | Event | Source |
|---|---|---|
| 12:01:13 | `ea79a2107e36` Mnemosyne Health Monitor execution claimed, `running` | `executions.db` |
| 00:01:46 → 00:10:45 | the same job's previous (00:01) slot — **completed in 9 m** | `executions.db` |
| 14:33:35 | `Received SIGTERM as a planned gateway stop`, `under_systemd=yes`, `parent_name=systemd`, `loadavg_1m=2.15` | `gateway.log` |
| 14:33:37 | `Thread 34795 not found` while sending the shutdown notice (same dead thread as the birthday/Uber jobs) | `gateway.log` |
| 14:33:37 | incident row `ea79a2_c3ab2d784ba3` opens — *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/724a69108b…json`"* (`alerted`) | `cron_incidents` |
| 14:33:38 | `Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True` → `interrupting remaining work` | `gateway.log` |
| 14:34:25 | `1 authority task(s) did not settle within 40s; left for recovery`; `Profile /home/hermes/.hermes still has live writers at shutdown; its admissions stay for recovery` | `gateway.log` |
| 14:34:27 | `ERROR gateway.session_authority: Admitted turn 28ad127ffb2a44c0bcd17283a93aa496 failed`; cron session `local-253a96…` **paused: runtime_draining** | `gateway.log` |
| 14:34:27 | that session finalises with `end_reason: cron_complete` (66 msgs, 39 tool calls, 127 API calls) | `state.db` |
| 14:34:29 | `Gateway stopped (total teardown 54.06s)`; **`Skipping .clean_shutdown marker — drain timed out with interrupted agents`** | `gateway.log` |
| 14:35:18 | next gateway life starts (`Starting Hermes Gateway...`) | `gateway.log` |
| 14:35:59 | execution closes **`failed`**: `RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership` | `executions.db` |
| 14:35:59 | incident row `ea79a2_a4661554a3c2` opens on that RuntimeError (`alerted`); execution `c5ac27e3…` **delivered** the failure | `cron_incidents`, `deliveries.db` |
| 14:46:12 / 15:01:15 | Mnemosyne Watchdog `96eb95ce12f2` fails twice (`Script exited with code 1`) reacting to the log `ERROR` — rows `96eb95_0c89bd373a0a`, `96eb95_e32174134546` | `cron_incidents` |
| 15:16:12 | both watchdog rows **`resolved`** (the error aged out of its 15-min log window); watchdog back to `ok` | `cron_incidents` |
## What is new or unexplained
1. **A planned stop is enough to orphan a run.** No crash was required. The drain had a **30 s** budget and
reported `timed_out=True` after **1.35 s** with `cron_now=0` — i.e. the drain did **not** wait out its
budget before taking the interrupt path. Logged verbatim; **not diagnosed**.
2. **New terminal error string:** `lost its durable fire claim ownership`. The 10-20 orphan closed as
`unknown` with a scheduler-side note; this one closed as `failed` from the worker's own perspective.
Any alerting or triage that pattern-matches on the 10-20 wording will miss this one.
3. **Two rows for one fault.** `ea79a2_a4661554a3c2` (fire claim) and `ea79a2_c3ab2d784ba3` (unverified /
reconcile admission) both remain **`alerted`, unclosed** as of 2026-10-27 00:1x, and the job carries
**`failure_streak: 2`** from a **single** interrupted fire.
4. **The admission was reconciled.** `~/.hermes/cron/admissions/` no longer holds `724a69108b…json`
(only the 00:00 / 00:01 files from the following night), so the "reconcile" step in the row's text has
been satisfied — the rows were simply never closed.
5. **The watchdog works.** This fault **did** alert, twice, and self-resolved — which is a genuine
improvement over 10-20 (one wakes up via the incident ledger, not via a missing digest).
## Impact
- **One 12-hourly memory consolidation cycle did not run.** `ea79a2107e36` is the Mnemosyne
health/consolidation job → [[systems/mnemosyne]]. The 00:01 slot completed normally; the 12:01 slot
produced no completed work.
- **Day tally:** 2026-10-26 settled **169 `completed` / 3 `failed`** — and **all three failures are this one
fault** (1 Health Monitor + 2 derivative Watchdog reds). 2026-10-25 was **172 / 2**.
- **Session cost sunk:** the interrupted run burned 66 messages / 39 tool calls / 127 API calls
(~597 k input, 254 k output tokens) before the interrupt.
- **Unclosed incident rows 2 → 3** (`0e753e_e7f8194a7d06` carried over from 2026-10-01, plus the two new
`ea79a2_*` rows).
## Recommended follow-ups (not yet done)
1. Decide the drain policy for **long-running cron jobs**: this job legitimately occupies 9 m → 2 h+.
Either exempt a cron execution from the drain deadline, or interrupt it with a durable
*"interrupted → re-fire this slot"* marker instead of a terminal `RuntimeError`.
2. **Close or ack the two `ea79a2_*` rows** — the admitted turn failed and the admission is gone, so they
are historical.
3. **Verify the drain-timeout arithmetic** (`timed_out=True` at 1.35 s against a 30 s budget) — if the
deadline is being bypassed whenever a writer is live, every future restart can interrupt work.
4. Confirm `failure_streak` should count the *unverified* row at all — a single interrupted fire reading
**streak 2** inflates the job's apparent health.
## Related
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — first instance of this class
- [[systems/scheduled-tasks]] — fleet ledger and per-window tallies
- [[systems/mnemosyne]] — the interrupted job's purpose and settings
- [[systems/hermes-agent]] — gateway lifecycle and restart history