audit: 2026-10-26 nightly — planned restart orphaned a live cron run (new incident page); 503 burst fully closed; radar cover-letter PDF pipeline moved to Chromium-over-HTML

This commit is contained in:
Tony0410
2026-10-27 00:11:50 +08:00
parent 3f796dc3b9
commit a2dd0bd650
8 changed files with 236 additions and 20 deletions
@@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu
type: incident
status: monitoring
created: 2026-10-18
updated: 2026-10-22
verified_on: 2026-10-22
review_after: 2026-10-29
updated: 2026-10-27
verified_on: 2026-10-27
review_after: 2026-11-03
confidence: high
tags: [cron, router, provider, infrastructure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
@@ -28,6 +28,13 @@ days** (10-16 → 10-17 and 10-19) with an identical signature on an identical n
> most of the fleet. Two bursts four days apart, both self-clearing, is the pattern to watch; this is a standing
> risk, not a resolved incident.
> **2026-10-26 update — every incident row from this failure mode is now closed.** The last open row,
> `c0a9fa_764558b5ae83` (`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.
> Retry shortly.`, job `c0a9fa314542` GitHub Trending Radar, `alerted` since 2026-10-19 09:02), **auto-closed
> 2026-10-26 09:13:00** when that job's 09:00 slot ran clean. **No execution row carries the resource-pressure
> string on 2026-10-25 or 2026-10-26.** Status stays **`monitoring`** for the same reason as 10-21: the rows are
> shut, the single-node exposure is not.
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven