incidents/2026-10-14-omniroute-node-connection-refused-storm.md was at exactly 21d and
would have tripped the threshold on the next run. Its residual claim ("omniroute-1
answers again") was re-probed live and still holds: TCP connect to 100.78.63.20:20128
succeeds (no Errno 111), the Tailscale peer reads Online:true, and a keyed
POST /v1/chat/completions returns HTTP 200. Page re-verified (updated/verified_on ->
2026-11-06, review_after -> 2026-12-06) with the probe recorded on the page.
New trap recorded on the page and in systems/scheduled-tasks.md: on omniroute-1 the
RAW model id deepseek/deepseek-v4-flash returns HTTP 401 "No active credentials for
provider: deepseek", while the fleet's deepseek-paid-smart alias returns HTTP 200 and
resolves to that same model. The 401 is an alias-routing artefact, not an outage - probe
the alias the fleet pins, never the raw model id. (A first probe using the wrong
key_env also produced a 401; omni-101's key env is HERMES_CUSTOM_100_78_63_20_20128_API_KEY.)
Freshness re-run after the edits: 0 stale / 124 fresh / 3 no-date of 127 pages.
Quartz rebuilt and redeployed. Vault mirror synced to both conventions (sha-verified).
10 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OmniRoute `omni-101` Node Outage — Connection-Refused Storm | concept | resolved | 2026-10-16 | 2026-11-06 | 2026-11-06 | 2026-12-06 | high |
|
|
OmniRoute omni-101 Node Outage — Connection-Refused Storm (2026-10-14 → 2026-10-15)
Window: 2026-10-14 08:30 → 2026-10-15 18:07 AWST (~34 hours)
Status: Resolved. omniroute-1 answers again; four scheduled jobs have since completed on it.
Re-verified 2026-11-06 (stocktake) — the residual claim still holds, and the node is genuinely healthy.
Live probe of the incident's own endpoint: TCP connect to 100.78.63.20:20128 succeeds (no
Errno 111 Connection refused — the incident's root cause has not recurred), the Tailscale peer
omniroute-1 reads Online: true, and a keyed POST /v1/chat/completions returns HTTP 200.
auto/best-chat → codestral-2508 (200), auto/best-reasoning → deepseek-ai/deepseek-v4.1-flash (200),
and the fleet's deepseek-paid-smart pin → deepseek/deepseek-v4-flash (200) — confirming the
systems/scheduled-tasks note that this pin dispatches on the omniroute-1 node. ⚠ One trap worth
recording: requesting the raw model id deepseek/deepseek-v4-flash (rather than the
deepseek-paid-smart alias) returns HTTP 401 No active credentials for provider: deepseek — that is
an alias-routing artefact on this node, not an outage and not the incident recurring. Probe the
alias the fleet actually uses.
Impact: 36 failed execution rows across 13 jobs — 24 genuine model-call failures on 12 jobs, plus
12 watchdog symptom reds. Every model call attempted in the window failed (see below). For scale:
10-12 recorded 0 failed and 10-13 recorded 1.
Severity: the largest failure count since the 2026-10-09 storm (15 in one day); this one ran two days
and silently suppressed most of the day's digests.
Root cause
TCP connect refused on http://100.78.63.20:20128 — the omniroute-1 node (Tailscale peer
omniroute-1, 100.78.63.20), registered in Hermes as provider custom:omni-101
(config.yaml: custom provider name: omni-101, base_url: http://100.78.63.20:20128).
httpcore.ConnectError: [Errno 111] Connection refused
→ openai.APIConnectionError: Connection error.
→ RuntimeError: Connection error. (as recorded in executions.db)
hermes-errors.log carried 328 Errno 111 occurrences in the live file alone (504 across the
rotations during the window). Every failing run recorded the identical leaf string, so all 24 model-call
failures are one fault, not 24 different ones.
Timeline (AWST)
| Time | Event |
|---|---|
| 10-14 08:30:27 | First failed run — Gmail Action Digest (08eb86f02d37). Isolated: 06:00 and 09:00–11:45 runs were clean. |
| 10-14 12:00 | Cascade — all three noon jobs fail (b63ed5817212, ea79a2107e36, ecc1fc13655a) and the watchdog reddens at 12:15/12:30. |
| 10-14 15:00 / 18:00 / 22:00 | Mosquito Tracker ×2, Embedding Coverage, Health Monitor all fail. |
| 10-14 23:59 | The nightly Wiki Audit itself failed — so the 10-14 window was never audited. |
| 10-15 03:00 | wiki-stocktake-maintenance (1a8e90ea2271) failed — no freshness-20261015.txt written. |
| 10-15 07:00–10:00 | Prime-time digests fail: Tech & AI, Daily Marketing Job Radar, Political News, AI Newsletter Curation for Patrick. |
| 10-15 12:00 | Noon jobs fail again. |
| 10-15 18:00:41 | Last genuine failure — Embedding Coverage Monitor. |
| 10-15 18:30:26 | Last watchdog red. |
| 10-15 22:01:56 | Recovery marker — Mosquito Tracker (1b31187397a5) completes on omni-101; streak 4 → 0. |
| 10-16 00:00–00:10 | Embedding Coverage (00:02:29), Health Monitor (00:09:48) and the Watchdog (00:01:36) all complete — three streaks cleared. |
| 10-16 00:1x | Live re-probe: GET /v1/models on 100.78.63.20:20128 → HTTP 401 in 10 ms (alive, auth required). Same on omniroute:20128 / 100.88.81.19. |
Blast radius
| Job | Name | Failures | First → last |
|---|---|---|---|
b63ed5817212 |
Mnemosyne Embedding Coverage (6-hourly) | 6 | 10-14 12:00 → 10-15 18:00 |
1b31187397a5 |
Mosquito Tracker (4×/day) | 4 | 10-14 15:00 → 10-15 15:00 |
ea79a2107e36 |
Mnemosyne Health Monitor (12-hourly) | 3 | 10-14 12:00 → 10-15 12:00 |
08eb86f02d37 |
Gmail Action Digest (daily 08:30) | 2 | 10-14 08:30 → 10-15 08:30 |
ecc1fc13655a |
FreshRSS Midday Briefing (daily 12:00) | 2 | 10-14 12:00 → 10-15 12:00 |
85edf6772bca |
Tech & AI Newsletter Digest (daily 07:00) | 1 | 10-15 07:00 |
dca8482e4f76 |
Daily Marketing Job Radar (weekdays 07:00) | 1 | 10-15 07:00 |
0e519874d280 |
Political News Digest (Mon/Thu 08:00) | 1 | 10-15 08:00 |
06a0a1c8704f |
AI Newsletter Curation for Patrick (Thu 10:00) | 1 | 10-15 10:00 |
c8d52904e1d6 |
Perth New Restaurant (Thu 18:00) | 1 | 10-15 18:00 |
1a8e90ea2271 |
wiki-stocktake-maintenance (daily 03:00) | 1 | 10-15 03:00 |
cbfe434a3b5e |
Wiki Audit (daily 23:59) | 1 | 10-14 23:59 |
96eb95ce12f2 |
Mnemosyne Watchdog (every 15 min) | 12 | symptom only — see below |
Day tallies: 10-14 = 160 completed / 12 failed (172), 10-15 = 151 completed / 24 failed / 1 running (176).
User-visible cost: several digests simply never went out — Tech & AI, Daily Marketing Job Radar, Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick, and the weekly Perth New Restaurant (missed outright; next chance 10-22).
What stayed green — and what that does not prove
Script-only jobs kept running normally. The fleet holds 10 no_agent jobs (no LLM turn: Mnemosyne
Auto-Consolidate, DB Backup, Watchdog, Daily Birthday Checker, Job Tracker Dashboard Refresh, Hermes Config
Backup to Gitea, weekly-save-restricted-bot-resources, Evening Stand-Down Briefing, plus two disabled).
Completing runs inside the storm window by provider: 210 no_agent / no provider + 1 no_agent on
custom:omni — that is the entire green of the fleet, and zero model calls completed in the 34 hours.
Every model-calling job in the fleet is on the one node that failed. Verified from jobs.json: all 18
enabled agent-mode jobs are pinned to custom:omni-101 (smart-route ×10, free-stack ×6,
deepseek-paid-smart ×2).
⚠ Correction to a first pass at this window: Evening Stand-Down Briefing (e530c89f2be0) carries
provider: custom:omni — but it is also no_agent: true, a script-only job whose provider/model fields
are vestigial. Its clean 18:45 runs on both 10-14 and 10-15 are therefore not evidence that CT600
stayed healthy; it never called a model. No cron job exercises custom:omni with an LLM turn
(d40085da631c, the only other custom:omni pin, is disabled), so the cron fleet cannot localise
this outage between the two OmniRoute nodes. What is reachable is the leaf error itself — Connection refused to 100.78.63.20:20128 — which points at omniroute-1.
The watchdog's reds are derivative
96eb95ce12f2 (Mnemosyne Watchdog, no_agent, script-only) accounts for 12 of the 36 failure rows — it
reddens when its log scan finds errors, so during a provider storm it is reporting the weather, not the
fault. All six of its incidents auto-resolved. Do not chase watchdog reds as a root cause; read
executions.error on the jobs that actually make model calls. It also reddens in bursts (12:15/12:30,
then 18:15/18:30), not continuously — its trigger is a log sentinel, so it under-reports the window rather
than bracketing it.
Incident bookkeeping
Open incidents after the storm: 8 alerted + 2 detected + 2 pre-existing = 12 open
(9 alerted / 3 detected). Each clears only on the next successful run of its own job, which for the
weekly and twice-weekly jobs means days of visible noise:
08eb86f02d37 10-16 08:30 · dca8482e4f76 10-16 07:00 · 85edf6772bca 10-16 07:00 ·
ecc1fc13655a 10-16 12:00 · 1a8e90ea2271 10-16 03:00 · 0e519874d280 10-19 (Mon) ·
06a0a1c8704f 10-22 (Thu) · c8d52904e1d6 10-22 (Thu).
Pre-existing and not storm-caused: 1b23c042e17d (alerted since 10-09, next slot 10-16 08:00) and
0e753e_e7f8194a7d06 (FB Birthday Monthly, detected since 10-01, next slot 11-01).
Lessons
- The entire scheduled estate rides one OmniRoute node. All 18 enabled model-calling jobs are pinned to
custom:omni-101; the only other model-capable cron pin,custom:omni, sits on ano_agentjob. Two live OmniRoute nodes exist (omnirouteCT600 at 100.88.81.19,omniroute-1at 100.78.63.20) with no failover between them, so one node's availability gates the whole estate's output. This is the second multi-day storm from the same topology → incidents/2026-10-09-cron-provider-timeout-storm. provideron ano_agentjob is decorative. Never read a script-only job's green row as evidence of provider health — checkno_agentbefore concluding anything from a clean run.- Both Mnemosyne monitors sit on the same node as the memory system they watch — both failed, so coverage figures went unrefreshed for the whole window (last explicit figures remain 2026-10-12). Monitoring that shares a blast radius with its subject is degraded monitoring.
- A log-sentinel watchdog multiplies failure counts. 36 failure rows were really 24, across 12 workers. Tally per-job, not per-row, or the scale reads wrong.
- A failed nightly audit leaves a hole, not a gap in coverage. The 10-14 audit died on this same storm, so this page covers two windows (10-14 → 10-16).
Related
- incidents/2026-10-09-cron-provider-timeout-storm — the earlier storm; same topology, same failure class.
- systems/omniroute — node layout, providers and pins.
- systems/scheduled-tasks — per-window fleet tally and incident ledger.
- systems/mnemosyne — the two coverage monitors that went dark.