audit(2026-10-13): 10-12 clean (174/0, 3rd day) but GitHub Trending Radar token DEAD (getMe 401) - supersedes the wrong 2026-10-06 'refuted' entry
Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline. NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450). Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
This commit is contained in:
@@ -3,8 +3,8 @@ title: Fleet-Wide Provider Timeout Storm — 2026-10-09
|
||||
type: incident
|
||||
status: unresolved
|
||||
created: 2026-10-09
|
||||
updated: 2026-10-09
|
||||
verified_on: 2026-10-09
|
||||
updated: 2026-10-13
|
||||
verified_on: 2026-10-13
|
||||
review_after: 2026-10-16
|
||||
confidence: high
|
||||
tags: [cron, omniroute, router, timeout, incident, unresolved]
|
||||
@@ -14,7 +14,9 @@ sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-2026100
|
||||
# Fleet-Wide Provider Timeout Storm — 2026-10-09
|
||||
|
||||
**Window:** 2026-10-09 00:00 → 15:00 AWST (last failure 15:00; every run from 18:00 onward completed)
|
||||
**Status:** Unresolved — the storm stopped and four incidents auto-closed, but **eight incidents opened on 2026-10-08/09 are still open** and `1a8e90ea2271` is the only job left at `failure_streak: 2`
|
||||
**Status:** Unresolved — the storm ended and **11 of its 12 incidents are now closed**; only
|
||||
`1b23c0_c35703f637b8` remains open, and it cannot clear before that job's next scheduled slot
|
||||
(**Fri 2026-10-16 08:00**). Nothing here is still symptomatic.
|
||||
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
|
||||
|
||||
## What happened
|
||||
@@ -79,18 +81,26 @@ therefore serving normally again well before the window closed.
|
||||
|
||||
| Incident | Job | State | Failure type | First seen | Closed |
|
||||
|----------|-----|-------|--------------|------------|--------|
|
||||
| `1a8e90_789224ad589c` | `1a8e90ea2271` (wiki-stocktake-maintenance) | detected | unknown | 2026-10-08 03:01 | — |
|
||||
| `1a8e90_789224ad589c` | `1a8e90ea2271` (wiki-stocktake-maintenance) | resolved | unknown | 2026-10-08 03:01 | 2026-10-10 03:06 |
|
||||
| `ea79a2_6fb70b032305` | `ea79a2107e36` (Mnemosyne Health Monitor) | resolved | timeout | 2026-10-09 00:18 | 2026-10-10 00:03 |
|
||||
| `b63ed5_f2c93dd24181` | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | resolved | timeout | 2026-10-09 00:18 | 2026-10-09 18:01 |
|
||||
| `cbfe43_fa81052baaf7` | `cbfe434a3b5e` (Wiki Audit — Nightly) | detected | timeout | 2026-10-09 00:32 | — |
|
||||
| `1a8e90_dc6b2d40ec44` | `1a8e90ea2271` (wiki-stocktake-maintenance) | detected | timeout | 2026-10-09 03:17 | — |
|
||||
| `cbfe43_fa81052baaf7` | `cbfe434a3b5e` (Wiki Audit — Nightly) | resolved | timeout | 2026-10-09 00:32 | 2026-10-10 00:27 |
|
||||
| `1a8e90_dc6b2d40ec44` | `1a8e90ea2271` (wiki-stocktake-maintenance) | resolved | timeout | 2026-10-09 03:17 | 2026-10-10 03:06 |
|
||||
| `96eb95_acb3ae5ec223` | `96eb95ce12f2` (Mnemosyne Watchdog) | resolved | timeout | 2026-10-09 06:30 | 2026-10-09 07:01 |
|
||||
| `dca848_23fcd2b18aaa` | `dca8482e4f76` (Daily Marketing Job Radar) | alerted | timeout | 2026-10-09 07:18 | — |
|
||||
| `dca848_23fcd2b18aaa` | `dca8482e4f76` (Daily Marketing Job Radar) | resolved | timeout | 2026-10-09 07:18 | **2026-10-12 07:33** |
|
||||
| `1b3118_f49c8d2c6e7f` | `1b31187397a5` (Mosquito Tracker) | resolved | timeout | 2026-10-09 07:18 | 2026-10-09 22:05 |
|
||||
| `85edf6_18bedbeec5cd` | `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | alerted | timeout | 2026-10-09 07:18 | — |
|
||||
| `1b23c0_c35703f637b8` | `1b23c042e17d` (Enhanced Weekend Adventure Planner) | alerted | timeout | 2026-10-09 08:17 | — |
|
||||
| `08eb86_634f1a296f78` | `08eb86f02d37` (Gmail Action Digest) | alerted | timeout | 2026-10-09 08:48 | — |
|
||||
| `ecc1fc_58b6c38296f3` | `ecc1fc13655a` (FreshRSS Midday News Briefing) | alerted | timeout | 2026-10-09 12:21 | — |
|
||||
| `85edf6_18bedbeec5cd` | `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | resolved | timeout | 2026-10-09 07:18 | 2026-10-10 07:04 |
|
||||
| `1b23c0_c35703f637b8` | `1b23c042e17d` (Enhanced Weekend Adventure Planner) | **alerted — still open** | timeout | 2026-10-09 08:17 | — (next slot Fri 2026-10-16 08:00) |
|
||||
| `08eb86_634f1a296f78` | `08eb86f02d37` (Gmail Action Digest) | resolved | timeout | 2026-10-09 08:48 | 2026-10-10 08:32 |
|
||||
| `ecc1fc_58b6c38296f3` | `ecc1fc13655a` (FreshRSS Midday News Briefing) | resolved | timeout | 2026-10-09 12:21 | 2026-10-10 12:05 |
|
||||
|
||||
**Closure timeline (re-read 2026-10-13):** four closed inside the storm window itself (2026-10-09
|
||||
07:01 → 22:05); five more closed on 2026-10-10 (00:03, 00:27, 03:06 ×2, 07:04, 08:32, 12:05); the
|
||||
in-window `dca848_23fcd2b18aaa` — the **last one to hold out** — only cleared on **2026-10-12 07:33:41**,
|
||||
nine seconds before its own job's 07:00 run finished successfully. That is the pattern worth remembering:
|
||||
each incident clears on the *next successful scheduled run of its own job*, so a weekly job carries its
|
||||
incident for up to a week after the underlying fault is gone. `1b23c042e17d` is the remaining example —
|
||||
its fault is long gone, but the row cannot close until Friday.
|
||||
|
||||
## Why it reached Anthony
|
||||
|
||||
|
||||
Reference in New Issue
Block a user