--- title: Fleet-Wide Provider Timeout Storm — 2026-10-09 type: incident status: unresolved created: 2026-10-09 updated: 2026-10-09 verified_on: 2026-10-09 review_after: 2026-10-16 confidence: high tags: [cron, omniroute, router, timeout, incident, unresolved] sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1] --- # Fleet-Wide Provider Timeout Storm — 2026-10-09 **Window:** 2026-10-09 00:00 → 15:00 AWST (last failure 15:00; every run from 18:00 onward completed) **Status:** Unresolved — the storm stopped and four incidents auto-closed, but **eight incidents opened on 2026-10-08/09 are still open** and `1a8e90ea2271` is the only job left at `failure_streak: 2` **Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08** ## What happened Every failure in the window is the same shape — `RuntimeError: Request timed out.` — i.e. the Hermes runtime gave up waiting on the inference call. The ten affected jobs are pinned to routing aliases (`smart-route`, `free-stack`, `deepseek-paid-smart`) on provider `custom:omni-101`, whose base URL is the local OmniRoute gateway. A job pinned to its own provider+model does **not** fall back to the global `fallback_providers` chain, which is why ten unrelated jobs all failed at once instead of quietly rerouting (see [[incidents/2026-10-05-dead-model-pin-repin]]). ## Timeline (2026-10-09 AWST) | Time | Job | Error | |------|-----|-------| | 00:00 | `ea79a2107e36` (Mnemosyne Health Monitor) | RuntimeError: Request timed out. | | 00:00 | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | RuntimeError: Request timed out. | | 03:00 | `1a8e90ea2271` (wiki-stocktake-maintenance) | RuntimeError: Request timed out. | | 06:00 | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | RuntimeError: Request timed out. | | 06:30 | `96eb95ce12f2` (Mnemosyne Watchdog) | Script exited with code 1 | | 06:45 | `96eb95ce12f2` (Mnemosyne Watchdog) | Script exited with code 1 | | 07:00 | `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | RuntimeError: Request timed out. | | 07:00 | `1b31187397a5` (Mosquito Tracker) | RuntimeError: Request timed out. | | 07:00 | `dca8482e4f76` (Daily Marketing Job Radar) | RuntimeError: Request timed out. | | 08:00 | `1b23c042e17d` (Enhanced Weekend Adventure Planner) | RuntimeError: Request timed out. | | 08:30 | `08eb86f02d37` (Gmail Action Digest) | RuntimeError: Request timed out. | | 12:00 | `ecc1fc13655a` (FreshRSS Midday News Briefing) | RuntimeError: Request timed out. | | 12:00 | `ea79a2107e36` (Mnemosyne Health Monitor) | RuntimeError: Request timed out. | | 12:00 | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | RuntimeError: Request timed out. | | 15:00 | `1b31187397a5` (Mosquito Tracker) | RuntimeError: Request timed out. | | 23:59 | `cbfe434a3b5e` (Wiki Audit — Nightly) | (no error text recorded) | The 16 failed executions cover 11 distinct jobs: | Job | Failures | Note | |-----|----------|------| | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | 3 | provider timeout | | `ea79a2107e36` (Mnemosyne Health Monitor) | 2 | provider timeout | | `96eb95ce12f2` (Mnemosyne Watchdog) | 2 | cascade — watchdog, not victim | | `1b31187397a5` (Mosquito Tracker) | 2 | provider timeout | | `1a8e90ea2271` (wiki-stocktake-maintenance) | 1 | provider timeout | | `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | 1 | provider timeout | | `dca8482e4f76` (Daily Marketing Job Radar) | 1 | provider timeout | | `1b23c042e17d` (Enhanced Weekend Adventure Planner) | 1 | provider timeout | | `08eb86f02d37` (Gmail Action Digest) | 1 | provider timeout | | `ecc1fc13655a` (FreshRSS Midday News Briefing) | 1 | provider timeout | | `cbfe434a3b5e` (Wiki Audit — Nightly) | 1 | provider timeout | **Not established:** whether the trigger was OmniRoute saturation, an upstream provider outage, or the runtime's own timeout budget. A late-evening probe of the gateway (from a different host alias, not from inside the storm window) failed to connect outright (`HTTP 000`) — corroborating intermittent reachability, but recorded here as an observation, not a root cause. ## Recovery Every run from 18:00 onward completed, and four incidents auto-closed on the next clean run: `96eb95_acb3ae5ec223` 07:01, `b63ed5_f2c93dd24181` 18:01, `1b3118_f49c8d2c6e7f` 22:05, and `ea79a2_6fb70b032305` at **2026-10-10 00:03:44** (the first post-window run). The gateway was therefore serving normally again well before the window closed. ## Incidents | Incident | Job | State | Failure type | First seen | Closed | |----------|-----|-------|--------------|------------|--------| | `1a8e90_789224ad589c` | `1a8e90ea2271` (wiki-stocktake-maintenance) | detected | unknown | 2026-10-08 03:01 | — | | `ea79a2_6fb70b032305` | `ea79a2107e36` (Mnemosyne Health Monitor) | resolved | timeout | 2026-10-09 00:18 | 2026-10-10 00:03 | | `b63ed5_f2c93dd24181` | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | resolved | timeout | 2026-10-09 00:18 | 2026-10-09 18:01 | | `cbfe43_fa81052baaf7` | `cbfe434a3b5e` (Wiki Audit — Nightly) | detected | timeout | 2026-10-09 00:32 | — | | `1a8e90_dc6b2d40ec44` | `1a8e90ea2271` (wiki-stocktake-maintenance) | detected | timeout | 2026-10-09 03:17 | — | | `96eb95_acb3ae5ec223` | `96eb95ce12f2` (Mnemosyne Watchdog) | resolved | timeout | 2026-10-09 06:30 | 2026-10-09 07:01 | | `dca848_23fcd2b18aaa` | `dca8482e4f76` (Daily Marketing Job Radar) | alerted | timeout | 2026-10-09 07:18 | — | | `1b3118_f49c8d2c6e7f` | `1b31187397a5` (Mosquito Tracker) | resolved | timeout | 2026-10-09 07:18 | 2026-10-09 22:05 | | `85edf6_18bedbeec5cd` | `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | alerted | timeout | 2026-10-09 07:18 | — | | `1b23c0_c35703f637b8` | `1b23c042e17d` (Enhanced Weekend Adventure Planner) | alerted | timeout | 2026-10-09 08:17 | — | | `08eb86_634f1a296f78` | `08eb86f02d37` (Gmail Action Digest) | alerted | timeout | 2026-10-09 08:48 | — | | `ecc1fc_58b6c38296f3` | `ecc1fc13655a` (FreshRSS Midday News Briefing) | alerted | timeout | 2026-10-09 12:21 | — | ## Why it reached Anthony `b63ed5817212` (Embedding Coverage, every 6h) failed three consecutive runs (00:00, 06:00, 12:00) while pinned to its own provider/model, so the framework's "failed N runs in a row — it does not fall back" notice was delivered; Anthony replied to it at 20:26 on 2026-10-09. That same exchange is where the DHL-monitor retirement was requested (see [[systems/scheduled-tasks]], open item #1). ## The watchdog cascade `96eb95ce12f2` (Mnemosyne Watchdog) failed twice at 06:30 and 06:45 with `Script exited with code 1 — 🚨 Log ERROR in agent.log`, which is the watchdog **reporting** the storm rather than suffering it. Read its failures as a symptom, not as two more victims. ## Evidence (how to re-check) ```bash sqlite3 ~/.hermes/cron/executions.db \ "SELECT status,count(*) FROM executions WHERE substr(datetime(started_at,'+8 hours'),1,10)='2026-10-09' GROUP BY 1" # today's read: completed 162 | failed 15 | running 1 sqlite3 ~/.hermes/cron/executions.db \ "SELECT job_id,started_at,substr(error,1,60) FROM executions WHERE status!='completed' AND substr(datetime(started_at,'+8 hours'),1,10)='2026-10-09'" sqlite3 ~/.hermes/cron/executions.db \ "SELECT id,job_id,state,first_seen_at,closed_at FROM cron_incidents WHERE closed_at IS NULL" ``` Baselines read the same way: 2026-10-07 → {'completed': 177, 'failed': 1}, 2026-10-08 → {'completed': 178, 'failed': 2}. ## Related - [[systems/scheduled-tasks]] — the job register and this window's re-verification block - [[incidents/2026-10-05-dead-model-pin-repin]] — why the fleet is aliased on `custom:omni-101` - [[concepts/model-routing]] — routing aliases and fallback semantics - [[systems/mnemosyne]] — the three memory jobs caught in the storm