Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline. NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450). Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
8.7 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Fleet-Wide Provider Timeout Storm — 2026-10-09 | incident | unresolved | 2026-10-09 | 2026-10-13 | 2026-10-13 | 2026-10-16 | high |
|
|
Fleet-Wide Provider Timeout Storm — 2026-10-09
Window: 2026-10-09 00:00 → 15:00 AWST (last failure 15:00; every run from 18:00 onward completed)
Status: Unresolved — the storm ended and 11 of its 12 incidents are now closed; only
1b23c0_c35703f637b8 remains open, and it cannot clear before that job's next scheduled slot
(Fri 2026-10-16 08:00). Nothing here is still symptomatic.
Impact: 15 failed executions across 10 distinct jobs, against baselines of 1 failed on 2026-10-07 and 2 on 2026-10-08
What happened
Every failure in the window is the same shape — RuntimeError: Request timed out. — i.e. the Hermes
runtime gave up waiting on the inference call. The ten affected jobs are pinned to routing aliases
(smart-route, free-stack, deepseek-paid-smart) on provider custom:omni-101, whose base URL is
the local OmniRoute gateway. A job pinned to its own provider+model does not fall back to the
global fallback_providers chain, which is why ten unrelated jobs all failed at once instead of
quietly rerouting (see incidents/2026-10-05-dead-model-pin-repin).
Timeline (2026-10-09 AWST)
| Time | Job | Error |
|---|---|---|
| 00:00 | ea79a2107e36 (Mnemosyne Health Monitor) |
RuntimeError: Request timed out. |
| 00:00 | b63ed5817212 (Mnemosyne Embedding Coverage Monitor) |
RuntimeError: Request timed out. |
| 03:00 | 1a8e90ea2271 (wiki-stocktake-maintenance) |
RuntimeError: Request timed out. |
| 06:00 | b63ed5817212 (Mnemosyne Embedding Coverage Monitor) |
RuntimeError: Request timed out. |
| 06:30 | 96eb95ce12f2 (Mnemosyne Watchdog) |
Script exited with code 1 |
| 06:45 | 96eb95ce12f2 (Mnemosyne Watchdog) |
Script exited with code 1 |
| 07:00 | 85edf6772bca (Tech & AI Newsletter Digest (Analytical)) |
RuntimeError: Request timed out. |
| 07:00 | 1b31187397a5 (Mosquito Tracker) |
RuntimeError: Request timed out. |
| 07:00 | dca8482e4f76 (Daily Marketing Job Radar) |
RuntimeError: Request timed out. |
| 08:00 | 1b23c042e17d (Enhanced Weekend Adventure Planner) |
RuntimeError: Request timed out. |
| 08:30 | 08eb86f02d37 (Gmail Action Digest) |
RuntimeError: Request timed out. |
| 12:00 | ecc1fc13655a (FreshRSS Midday News Briefing) |
RuntimeError: Request timed out. |
| 12:00 | ea79a2107e36 (Mnemosyne Health Monitor) |
RuntimeError: Request timed out. |
| 12:00 | b63ed5817212 (Mnemosyne Embedding Coverage Monitor) |
RuntimeError: Request timed out. |
| 15:00 | 1b31187397a5 (Mosquito Tracker) |
RuntimeError: Request timed out. |
| 23:59 | cbfe434a3b5e (Wiki Audit — Nightly) |
(no error text recorded) |
The 16 failed executions cover 11 distinct jobs:
| Job | Failures | Note |
|---|---|---|
b63ed5817212 (Mnemosyne Embedding Coverage Monitor) |
3 | provider timeout |
ea79a2107e36 (Mnemosyne Health Monitor) |
2 | provider timeout |
96eb95ce12f2 (Mnemosyne Watchdog) |
2 | cascade — watchdog, not victim |
1b31187397a5 (Mosquito Tracker) |
2 | provider timeout |
1a8e90ea2271 (wiki-stocktake-maintenance) |
1 | provider timeout |
85edf6772bca (Tech & AI Newsletter Digest (Analytical)) |
1 | provider timeout |
dca8482e4f76 (Daily Marketing Job Radar) |
1 | provider timeout |
1b23c042e17d (Enhanced Weekend Adventure Planner) |
1 | provider timeout |
08eb86f02d37 (Gmail Action Digest) |
1 | provider timeout |
ecc1fc13655a (FreshRSS Midday News Briefing) |
1 | provider timeout |
cbfe434a3b5e (Wiki Audit — Nightly) |
1 | provider timeout |
Not established: whether the trigger was OmniRoute saturation, an upstream provider outage, or
the runtime's own timeout budget. A late-evening probe of the gateway (from a different host alias,
not from inside the storm window) failed to connect outright (HTTP 000) — corroborating
intermittent reachability, but recorded here as an observation, not a root cause.
Recovery
Every run from 18:00 onward completed, and four incidents auto-closed on the next clean run:
96eb95_acb3ae5ec223 07:01, b63ed5_f2c93dd24181 18:01, 1b3118_f49c8d2c6e7f 22:05, and
ea79a2_6fb70b032305 at 2026-10-10 00:03:44 (the first post-window run). The gateway was
therefore serving normally again well before the window closed.
Incidents
| Incident | Job | State | Failure type | First seen | Closed |
|---|---|---|---|---|---|
1a8e90_789224ad589c |
1a8e90ea2271 (wiki-stocktake-maintenance) |
resolved | unknown | 2026-10-08 03:01 | 2026-10-10 03:06 |
ea79a2_6fb70b032305 |
ea79a2107e36 (Mnemosyne Health Monitor) |
resolved | timeout | 2026-10-09 00:18 | 2026-10-10 00:03 |
b63ed5_f2c93dd24181 |
b63ed5817212 (Mnemosyne Embedding Coverage Monitor) |
resolved | timeout | 2026-10-09 00:18 | 2026-10-09 18:01 |
cbfe43_fa81052baaf7 |
cbfe434a3b5e (Wiki Audit — Nightly) |
resolved | timeout | 2026-10-09 00:32 | 2026-10-10 00:27 |
1a8e90_dc6b2d40ec44 |
1a8e90ea2271 (wiki-stocktake-maintenance) |
resolved | timeout | 2026-10-09 03:17 | 2026-10-10 03:06 |
96eb95_acb3ae5ec223 |
96eb95ce12f2 (Mnemosyne Watchdog) |
resolved | timeout | 2026-10-09 06:30 | 2026-10-09 07:01 |
dca848_23fcd2b18aaa |
dca8482e4f76 (Daily Marketing Job Radar) |
resolved | timeout | 2026-10-09 07:18 | 2026-10-12 07:33 |
1b3118_f49c8d2c6e7f |
1b31187397a5 (Mosquito Tracker) |
resolved | timeout | 2026-10-09 07:18 | 2026-10-09 22:05 |
85edf6_18bedbeec5cd |
85edf6772bca (Tech & AI Newsletter Digest (Analytical)) |
resolved | timeout | 2026-10-09 07:18 | 2026-10-10 07:04 |
1b23c0_c35703f637b8 |
1b23c042e17d (Enhanced Weekend Adventure Planner) |
alerted — still open | timeout | 2026-10-09 08:17 | — (next slot Fri 2026-10-16 08:00) |
08eb86_634f1a296f78 |
08eb86f02d37 (Gmail Action Digest) |
resolved | timeout | 2026-10-09 08:48 | 2026-10-10 08:32 |
ecc1fc_58b6c38296f3 |
ecc1fc13655a (FreshRSS Midday News Briefing) |
resolved | timeout | 2026-10-09 12:21 | 2026-10-10 12:05 |
Closure timeline (re-read 2026-10-13): four closed inside the storm window itself (2026-10-09
07:01 → 22:05); five more closed on 2026-10-10 (00:03, 00:27, 03:06 ×2, 07:04, 08:32, 12:05); the
in-window dca848_23fcd2b18aaa — the last one to hold out — only cleared on 2026-10-12 07:33:41,
nine seconds before its own job's 07:00 run finished successfully. That is the pattern worth remembering:
each incident clears on the next successful scheduled run of its own job, so a weekly job carries its
incident for up to a week after the underlying fault is gone. 1b23c042e17d is the remaining example —
its fault is long gone, but the row cannot close until Friday.
Why it reached Anthony
b63ed5817212 (Embedding Coverage, every 6h) failed three consecutive runs (00:00, 06:00, 12:00)
while pinned to its own provider/model, so the framework's "failed N runs in a row — it does not
fall back" notice was delivered; Anthony replied to it at 20:26 on 2026-10-09. That same exchange is
where the DHL-monitor retirement was requested (see systems/scheduled-tasks, open item #1).
The watchdog cascade
96eb95ce12f2 (Mnemosyne Watchdog) failed twice at 06:30 and 06:45 with
Script exited with code 1 — 🚨 Log ERROR in agent.log, which is the watchdog reporting the storm
rather than suffering it. Read its failures as a symptom, not as two more victims.
Evidence (how to re-check)
sqlite3 ~/.hermes/cron/executions.db \
"SELECT status,count(*) FROM executions
WHERE substr(datetime(started_at,'+8 hours'),1,10)='2026-10-09' GROUP BY 1"
# today's read: completed 162 | failed 15 | running 1
sqlite3 ~/.hermes/cron/executions.db \
"SELECT job_id,started_at,substr(error,1,60) FROM executions
WHERE status!='completed' AND substr(datetime(started_at,'+8 hours'),1,10)='2026-10-09'"
sqlite3 ~/.hermes/cron/executions.db \
"SELECT id,job_id,state,first_seen_at,closed_at FROM cron_incidents WHERE closed_at IS NULL"
Baselines read the same way: 2026-10-07 → {'completed': 177, 'failed': 1}, 2026-10-08 → {'completed': 178, 'failed': 2}.
Related
- systems/scheduled-tasks — the job register and this window's re-verification block
- incidents/2026-10-05-dead-model-pin-repin — why the fleet is aliased on
custom:omni-101 - concepts/model-routing — routing aliases and fallback semantics
- systems/mnemosyne — the three memory jobs caught in the storm