audit: 2026-10-26 nightly — planned restart orphaned a live cron run (new incident page); 503 burst fully closed; radar cover-letter PDF pipeline moved to Chromium-over-HTML
This commit is contained in:
@@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu
|
||||
type: incident
|
||||
status: monitoring
|
||||
created: 2026-10-18
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
review_after: 2026-10-29
|
||||
updated: 2026-10-27
|
||||
verified_on: 2026-10-27
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
tags: [cron, router, provider, infrastructure]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
|
||||
@@ -28,6 +28,13 @@ days** (10-16 → 10-17 and 10-19) with an identical signature on an identical n
|
||||
> most of the fleet. Two bursts four days apart, both self-clearing, is the pattern to watch; this is a standing
|
||||
> risk, not a resolved incident.
|
||||
|
||||
> **2026-10-26 update — every incident row from this failure mode is now closed.** The last open row,
|
||||
> `c0a9fa_764558b5ae83` (`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.
|
||||
> Retry shortly.`, job `c0a9fa314542` GitHub Trending Radar, `alerted` since 2026-10-19 09:02), **auto-closed
|
||||
> 2026-10-26 09:13:00** when that job's 09:00 slot ran clean. **No execution row carries the resource-pressure
|
||||
> string on 2026-10-25 or 2026-10-26.** Status stays **`monitoring`** for the same reason as 10-21: the rows are
|
||||
> shut, the single-node exposure is not.
|
||||
|
||||
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
|
||||
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
|
||||
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
|
||||
|
||||
Reference in New Issue
Block a user