Files
agent-estate-wiki/incidents/2026-10-09-cron-provider-timeout-storm.md
T

7.8 KiB

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Fleet-Wide Provider Timeout Storm — 2026-10-09 incident unresolved 2026-10-09 2026-10-09 2026-10-09 2026-10-16 high
cron
omniroute
router
timeout
incident
unresolved
cron-executions-db
cron-incidents-db
live-jobs-json
session-20261009_225935_7b1cb1

Fleet-Wide Provider Timeout Storm — 2026-10-09

Window: 2026-10-09 00:00 → 15:00 AWST (last failure 15:00; every run from 18:00 onward completed) Status: Unresolved — the storm stopped and four incidents auto-closed, but eight incidents opened on 2026-10-08/09 are still open and 1a8e90ea2271 is the only job left at failure_streak: 2 Impact: 15 failed executions across 10 distinct jobs, against baselines of 1 failed on 2026-10-07 and 2 on 2026-10-08

What happened

Every failure in the window is the same shape — RuntimeError: Request timed out. — i.e. the Hermes runtime gave up waiting on the inference call. The ten affected jobs are pinned to routing aliases (smart-route, free-stack, deepseek-paid-smart) on provider custom:omni-101, whose base URL is the local OmniRoute gateway. A job pinned to its own provider+model does not fall back to the global fallback_providers chain, which is why ten unrelated jobs all failed at once instead of quietly rerouting (see incidents/2026-10-05-dead-model-pin-repin).

Timeline (2026-10-09 AWST)

Time Job Error
00:00 ea79a2107e36 (Mnemosyne Health Monitor) RuntimeError: Request timed out.
00:00 b63ed5817212 (Mnemosyne Embedding Coverage Monitor) RuntimeError: Request timed out.
03:00 1a8e90ea2271 (wiki-stocktake-maintenance) RuntimeError: Request timed out.
06:00 b63ed5817212 (Mnemosyne Embedding Coverage Monitor) RuntimeError: Request timed out.
06:30 96eb95ce12f2 (Mnemosyne Watchdog) Script exited with code 1
06:45 96eb95ce12f2 (Mnemosyne Watchdog) Script exited with code 1
07:00 85edf6772bca (Tech & AI Newsletter Digest (Analytical)) RuntimeError: Request timed out.
07:00 1b31187397a5 (Mosquito Tracker) RuntimeError: Request timed out.
07:00 dca8482e4f76 (Daily Marketing Job Radar) RuntimeError: Request timed out.
08:00 1b23c042e17d (Enhanced Weekend Adventure Planner) RuntimeError: Request timed out.
08:30 08eb86f02d37 (Gmail Action Digest) RuntimeError: Request timed out.
12:00 ecc1fc13655a (FreshRSS Midday News Briefing) RuntimeError: Request timed out.
12:00 ea79a2107e36 (Mnemosyne Health Monitor) RuntimeError: Request timed out.
12:00 b63ed5817212 (Mnemosyne Embedding Coverage Monitor) RuntimeError: Request timed out.
15:00 1b31187397a5 (Mosquito Tracker) RuntimeError: Request timed out.
23:59 cbfe434a3b5e (Wiki Audit — Nightly) (no error text recorded)

The 16 failed executions cover 11 distinct jobs:

Job Failures Note
b63ed5817212 (Mnemosyne Embedding Coverage Monitor) 3 provider timeout
ea79a2107e36 (Mnemosyne Health Monitor) 2 provider timeout
96eb95ce12f2 (Mnemosyne Watchdog) 2 cascade — watchdog, not victim
1b31187397a5 (Mosquito Tracker) 2 provider timeout
1a8e90ea2271 (wiki-stocktake-maintenance) 1 provider timeout
85edf6772bca (Tech & AI Newsletter Digest (Analytical)) 1 provider timeout
dca8482e4f76 (Daily Marketing Job Radar) 1 provider timeout
1b23c042e17d (Enhanced Weekend Adventure Planner) 1 provider timeout
08eb86f02d37 (Gmail Action Digest) 1 provider timeout
ecc1fc13655a (FreshRSS Midday News Briefing) 1 provider timeout
cbfe434a3b5e (Wiki Audit — Nightly) 1 provider timeout

Not established: whether the trigger was OmniRoute saturation, an upstream provider outage, or the runtime's own timeout budget. A late-evening probe of the gateway (from a different host alias, not from inside the storm window) failed to connect outright (HTTP 000) — corroborating intermittent reachability, but recorded here as an observation, not a root cause.

Recovery

Every run from 18:00 onward completed, and four incidents auto-closed on the next clean run: 96eb95_acb3ae5ec223 07:01, b63ed5_f2c93dd24181 18:01, 1b3118_f49c8d2c6e7f 22:05, and ea79a2_6fb70b032305 at 2026-10-10 00:03:44 (the first post-window run). The gateway was therefore serving normally again well before the window closed.

Incidents

Incident Job State Failure type First seen Closed
1a8e90_789224ad589c 1a8e90ea2271 (wiki-stocktake-maintenance) detected unknown 2026-10-08 03:01 —
ea79a2_6fb70b032305 ea79a2107e36 (Mnemosyne Health Monitor) resolved timeout 2026-10-09 00:18 2026-10-10 00:03
b63ed5_f2c93dd24181 b63ed5817212 (Mnemosyne Embedding Coverage Monitor) resolved timeout 2026-10-09 00:18 2026-10-09 18:01
cbfe43_fa81052baaf7 cbfe434a3b5e (Wiki Audit — Nightly) detected timeout 2026-10-09 00:32 —
1a8e90_dc6b2d40ec44 1a8e90ea2271 (wiki-stocktake-maintenance) detected timeout 2026-10-09 03:17 —
96eb95_acb3ae5ec223 96eb95ce12f2 (Mnemosyne Watchdog) resolved timeout 2026-10-09 06:30 2026-10-09 07:01
dca848_23fcd2b18aaa dca8482e4f76 (Daily Marketing Job Radar) alerted timeout 2026-10-09 07:18 —
1b3118_f49c8d2c6e7f 1b31187397a5 (Mosquito Tracker) resolved timeout 2026-10-09 07:18 2026-10-09 22:05
85edf6_18bedbeec5cd 85edf6772bca (Tech & AI Newsletter Digest (Analytical)) alerted timeout 2026-10-09 07:18 —
1b23c0_c35703f637b8 1b23c042e17d (Enhanced Weekend Adventure Planner) alerted timeout 2026-10-09 08:17 —
08eb86_634f1a296f78 08eb86f02d37 (Gmail Action Digest) alerted timeout 2026-10-09 08:48 —
ecc1fc_58b6c38296f3 ecc1fc13655a (FreshRSS Midday News Briefing) alerted timeout 2026-10-09 12:21 —

Why it reached Anthony

b63ed5817212 (Embedding Coverage, every 6h) failed three consecutive runs (00:00, 06:00, 12:00) while pinned to its own provider/model, so the framework's "failed N runs in a row — it does not fall back" notice was delivered; Anthony replied to it at 20:26 on 2026-10-09. That same exchange is where the DHL-monitor retirement was requested (see systems/scheduled-tasks, open item #1).

The watchdog cascade

96eb95ce12f2 (Mnemosyne Watchdog) failed twice at 06:30 and 06:45 with Script exited with code 1 — 🚨 Log ERROR in agent.log, which is the watchdog reporting the storm rather than suffering it. Read its failures as a symptom, not as two more victims.

Evidence (how to re-check)

sqlite3 ~/.hermes/cron/executions.db \
  "SELECT status,count(*) FROM executions
   WHERE substr(datetime(started_at,'+8 hours'),1,10)='2026-10-09' GROUP BY 1"
# today's read: completed 162 | failed 15 | running 1

sqlite3 ~/.hermes/cron/executions.db \
  "SELECT job_id,started_at,substr(error,1,60) FROM executions
   WHERE status!='completed' AND substr(datetime(started_at,'+8 hours'),1,10)='2026-10-09'"

sqlite3 ~/.hermes/cron/executions.db \
  "SELECT id,job_id,state,first_seen_at,closed_at FROM cron_incidents WHERE closed_at IS NULL"

Baselines read the same way: 2026-10-07 → {'completed': 177, 'failed': 1}, 2026-10-08 → {'completed': 178, 'failed': 2}.