Files
agent-estate-wiki/incidents/2026-10-09-cron-provider-timeout-storm.md
T

152 lines
9.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Fleet-Wide Provider Timeout Storm — 2026-10-09
type: incident
status: resolved
created: 2026-10-09
updated: 2026-11-15
verified_on: 2026-11-15
review_after: 2026-12-15
confidence: high
tags: [cron, omniroute, router, timeout, incident, resolved]
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
---
# Fleet-Wide Provider Timeout Storm — 2026-10-09
**Window:** 2026-10-09 00:00 → 15:00 AWST (last failure 15:00; every run from 18:00 onward completed)
**Status:** Resolved — **all 12 of its incident rows are closed.** The last hold-outs
(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job)
closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again.
The storm contributes **no** unclosed rows. Nothing here is still symptomatic.
> **Re-verified 2026-11-15** (3 a.m. stocktake): all 12 of the storm's `cron_incidents` rows still read
> `resolved` — none reopened. `1b23c042e17d` (the weekly Friday job, last hold-out) is healthy —
> `last_status: ok`, last run 2026-11-13 08:11 AWST. One incident is currently open fleet-wide
> (`ea79a2_601037516abd`, Mnemosyne Health Monitor, first seen 2026-11-15 00:03) but it is the
> capture-stall detection firing → [[incidents/2026-11-14-mnemosyne-capture-stall]], unrelated to
> this storm. ⚠ Evidence window: `executions.db` retention now reaches back only to 2026-11-09, so
> the 2026-10-09 execution counts below are **no longer re-derivable from the DB** — they stand on
> the 2026-10-24 read (the incident rows themselves persist, which is what this check re-read).
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
## What happened
Every failure in the window is the same shape — `RuntimeError: Request timed out.` — i.e. the Hermes
runtime gave up waiting on the inference call. The ten affected jobs are pinned to routing aliases
(`smart-route`, `free-stack`, `deepseek-paid-smart`) on provider `custom:omni-101`, whose base URL is
the local OmniRoute gateway. A job pinned to its own provider+model does **not** fall back to the
global `fallback_providers` chain, which is why ten unrelated jobs all failed at once instead of
quietly rerouting (see [[incidents/2026-10-05-dead-model-pin-repin]]).
## Timeline (2026-10-09 AWST)
| Time | Job | Error |
|------|-----|-------|
| 00:00 | `ea79a2107e36` (Mnemosyne Health Monitor) | RuntimeError: Request timed out. |
| 00:00 | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | RuntimeError: Request timed out. |
| 03:00 | `1a8e90ea2271` (wiki-stocktake-maintenance) | RuntimeError: Request timed out. |
| 06:00 | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | RuntimeError: Request timed out. |
| 06:30 | `96eb95ce12f2` (Mnemosyne Watchdog) | Script exited with code 1 |
| 06:45 | `96eb95ce12f2` (Mnemosyne Watchdog) | Script exited with code 1 |
| 07:00 | `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | RuntimeError: Request timed out. |
| 07:00 | `1b31187397a5` (Mosquito Tracker) | RuntimeError: Request timed out. |
| 07:00 | `dca8482e4f76` (Daily Marketing Job Radar) | RuntimeError: Request timed out. |
| 08:00 | `1b23c042e17d` (Enhanced Weekend Adventure Planner) | RuntimeError: Request timed out. |
| 08:30 | `08eb86f02d37` (Gmail Action Digest) | RuntimeError: Request timed out. |
| 12:00 | `ecc1fc13655a` (FreshRSS Midday News Briefing) | RuntimeError: Request timed out. |
| 12:00 | `ea79a2107e36` (Mnemosyne Health Monitor) | RuntimeError: Request timed out. |
| 12:00 | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | RuntimeError: Request timed out. |
| 15:00 | `1b31187397a5` (Mosquito Tracker) | RuntimeError: Request timed out. |
| 23:59 | `cbfe434a3b5e` (Wiki Audit — Nightly) | (no error text recorded) |
The 16 failed executions cover 11 distinct jobs:
| Job | Failures | Note |
|-----|----------|------|
| `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | 3 | provider timeout |
| `ea79a2107e36` (Mnemosyne Health Monitor) | 2 | provider timeout |
| `96eb95ce12f2` (Mnemosyne Watchdog) | 2 | cascade — watchdog, not victim |
| `1b31187397a5` (Mosquito Tracker) | 2 | provider timeout |
| `1a8e90ea2271` (wiki-stocktake-maintenance) | 1 | provider timeout |
| `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | 1 | provider timeout |
| `dca8482e4f76` (Daily Marketing Job Radar) | 1 | provider timeout |
| `1b23c042e17d` (Enhanced Weekend Adventure Planner) | 1 | provider timeout |
| `08eb86f02d37` (Gmail Action Digest) | 1 | provider timeout |
| `ecc1fc13655a` (FreshRSS Midday News Briefing) | 1 | provider timeout |
| `cbfe434a3b5e` (Wiki Audit — Nightly) | 1 | provider timeout |
**Not established:** whether the trigger was OmniRoute saturation, an upstream provider outage, or
the runtime's own timeout budget. A late-evening probe of the gateway (from a different host alias,
not from inside the storm window) failed to connect outright (`HTTP 000`) — corroborating
intermittent reachability, but recorded here as an observation, not a root cause.
## Recovery
Every run from 18:00 onward completed, and four incidents auto-closed on the next clean run:
`96eb95_acb3ae5ec223` 07:01, `b63ed5_f2c93dd24181` 18:01, `1b3118_f49c8d2c6e7f` 22:05, and
`ea79a2_6fb70b032305` at **2026-10-10 00:03:44** (the first post-window run). The gateway was
therefore serving normally again well before the window closed.
## Incidents
| Incident | Job | State | Failure type | First seen | Closed |
|----------|-----|-------|--------------|------------|--------|
| `1a8e90_789224ad589c` | `1a8e90ea2271` (wiki-stocktake-maintenance) | resolved | unknown | 2026-10-08 03:01 | 2026-10-10 03:06 |
| `ea79a2_6fb70b032305` | `ea79a2107e36` (Mnemosyne Health Monitor) | resolved | timeout | 2026-10-09 00:18 | 2026-10-10 00:03 |
| `b63ed5_f2c93dd24181` | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | resolved | timeout | 2026-10-09 00:18 | 2026-10-09 18:01 |
| `cbfe43_fa81052baaf7` | `cbfe434a3b5e` (Wiki Audit — Nightly) | resolved | timeout | 2026-10-09 00:32 | 2026-10-10 00:27 |
| `1a8e90_dc6b2d40ec44` | `1a8e90ea2271` (wiki-stocktake-maintenance) | resolved | timeout | 2026-10-09 03:17 | 2026-10-10 03:06 |
| `96eb95_acb3ae5ec223` | `96eb95ce12f2` (Mnemosyne Watchdog) | resolved | timeout | 2026-10-09 06:30 | 2026-10-09 07:01 |
| `dca848_23fcd2b18aaa` | `dca8482e4f76` (Daily Marketing Job Radar) | resolved | timeout | 2026-10-09 07:18 | **2026-10-12 07:33** |
| `1b3118_f49c8d2c6e7f` | `1b31187397a5` (Mosquito Tracker) | resolved | timeout | 2026-10-09 07:18 | 2026-10-09 22:05 |
| `85edf6_18bedbeec5cd` | `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | resolved | timeout | 2026-10-09 07:18 | 2026-10-10 07:04 |
| `1b23c0_c35703f637b8` | `1b23c042e17d` (Enhanced Weekend Adventure Planner) | **alerted — still open** | timeout | 2026-10-09 08:17 | — (next slot Fri 2026-10-16 08:00) |
| `08eb86_634f1a296f78` | `08eb86f02d37` (Gmail Action Digest) | resolved | timeout | 2026-10-09 08:48 | 2026-10-10 08:32 |
| `ecc1fc_58b6c38296f3` | `ecc1fc13655a` (FreshRSS Midday News Briefing) | resolved | timeout | 2026-10-09 12:21 | 2026-10-10 12:05 |
**Closure timeline (re-read 2026-10-13):** four closed inside the storm window itself (2026-10-09
07:01 → 22:05); five more closed on 2026-10-10 (00:03, 00:27, 03:06 ×2, 07:04, 08:32, 12:05); the
in-window `dca848_23fcd2b18aaa` — the **last one to hold out** — only cleared on **2026-10-12 07:33:41**,
nine seconds before its own job's 07:00 run finished successfully. That is the pattern worth remembering:
each incident clears on the *next successful scheduled run of its own job*, so a weekly job carries its
incident for up to a week after the underlying fault is gone. `1b23c042e17d` is the remaining example —
its fault is long gone, but the row cannot close until Friday.
## Why it reached Anthony
`b63ed5817212` (Embedding Coverage, every 6h) failed three consecutive runs (00:00, 06:00, 12:00)
while pinned to its own provider/model, so the framework's "failed N runs in a row — it does not
fall back" notice was delivered; Anthony replied to it at 20:26 on 2026-10-09. That same exchange is
where the DHL-monitor retirement was requested (see [[systems/scheduled-tasks]], open item #1).
## The watchdog cascade
`96eb95ce12f2` (Mnemosyne Watchdog) failed twice at 06:30 and 06:45 with
`Script exited with code 1 — 🚨 Log ERROR in agent.log`, which is the watchdog **reporting** the storm
rather than suffering it. Read its failures as a symptom, not as two more victims.
## Evidence (how to re-check)
```bash
sqlite3 ~/.hermes/cron/executions.db \
"SELECT status,count(*) FROM executions
WHERE substr(datetime(started_at,'+8 hours'),1,10)='2026-10-09' GROUP BY 1"
# today's read: completed 162 | failed 15 | running 1
sqlite3 ~/.hermes/cron/executions.db \
"SELECT job_id,started_at,substr(error,1,60) FROM executions
WHERE status!='completed' AND substr(datetime(started_at,'+8 hours'),1,10)='2026-10-09'"
sqlite3 ~/.hermes/cron/executions.db \
"SELECT id,job_id,state,first_seen_at,closed_at FROM cron_incidents WHERE closed_at IS NULL"
```
Baselines read the same way: 2026-10-07 → {'completed': 177, 'failed': 1}, 2026-10-08 → {'completed': 178, 'failed': 2}.
## Related
- [[systems/scheduled-tasks]] — the job register and this window's re-verification block
- [[incidents/2026-10-05-dead-model-pin-repin]] — why the fleet is aliased on `custom:omni-101`
- [[concepts/model-routing]] — routing aliases and fallback semantics
- [[systems/mnemosyne]] — the three memory jobs caught in the storm