Files
agent-estate-wiki/incidents/2026-10-14-omniroute-node-connection-refused-storm.md
Tony0410 2d5219b068 stocktake 2026-11-06 (pass 2): re-verify the one page sitting on the 21-day threshold
incidents/2026-10-14-omniroute-node-connection-refused-storm.md was at exactly 21d and
would have tripped the threshold on the next run. Its residual claim ("omniroute-1
answers again") was re-probed live and still holds: TCP connect to 100.78.63.20:20128
succeeds (no Errno 111), the Tailscale peer reads Online:true, and a keyed
POST /v1/chat/completions returns HTTP 200. Page re-verified (updated/verified_on ->
2026-11-06, review_after -> 2026-12-06) with the probe recorded on the page.

New trap recorded on the page and in systems/scheduled-tasks.md: on omniroute-1 the
RAW model id deepseek/deepseek-v4-flash returns HTTP 401 "No active credentials for
provider: deepseek", while the fleet's deepseek-paid-smart alias returns HTTP 200 and
resolves to that same model. The 401 is an alias-routing artefact, not an outage - probe
the alias the fleet pins, never the raw model id. (A first probe using the wrong
key_env also produced a 401; omni-101's key env is HERMES_CUSTOM_100_78_63_20_20128_API_KEY.)

Freshness re-run after the edits: 0 stale / 124 fresh / 3 no-date of 127 pages.
Quartz rebuilt and redeployed. Vault mirror synced to both conventions (sha-verified).
2026-11-06 04:07:12 +08:00

10 KiB
Raw Permalink Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
OmniRoute `omni-101` Node Outage — Connection-Refused Storm concept resolved 2026-10-16 2026-11-06 2026-11-06 2026-12-06 high
cron
router
provider
infrastructure
live-sqlite-cron-executions
live-cron-incidents
live-jobs-json
hermes-errors-log
live-probe

OmniRoute omni-101 Node Outage — Connection-Refused Storm (2026-10-14 → 2026-10-15)

Window: 2026-10-14 08:30 → 2026-10-15 18:07 AWST (~34 hours) Status: Resolved. omniroute-1 answers again; four scheduled jobs have since completed on it.

Re-verified 2026-11-06 (stocktake) — the residual claim still holds, and the node is genuinely healthy. Live probe of the incident's own endpoint: TCP connect to 100.78.63.20:20128 succeeds (no Errno 111 Connection refused — the incident's root cause has not recurred), the Tailscale peer omniroute-1 reads Online: true, and a keyed POST /v1/chat/completions returns HTTP 200. auto/best-chat → codestral-2508 (200), auto/best-reasoning → deepseek-ai/deepseek-v4.1-flash (200), and the fleet's deepseek-paid-smart pin → deepseek/deepseek-v4-flash (200) — confirming the systems/scheduled-tasks note that this pin dispatches on the omniroute-1 node. ⚠ One trap worth recording: requesting the raw model id deepseek/deepseek-v4-flash (rather than the deepseek-paid-smart alias) returns HTTP 401 No active credentials for provider: deepseek — that is an alias-routing artefact on this node, not an outage and not the incident recurring. Probe the alias the fleet actually uses. Impact: 36 failed execution rows across 13 jobs — 24 genuine model-call failures on 12 jobs, plus 12 watchdog symptom reds. Every model call attempted in the window failed (see below). For scale: 10-12 recorded 0 failed and 10-13 recorded 1. Severity: the largest failure count since the 2026-10-09 storm (15 in one day); this one ran two days and silently suppressed most of the day's digests.

Root cause

TCP connect refused on http://100.78.63.20:20128 — the omniroute-1 node (Tailscale peer omniroute-1, 100.78.63.20), registered in Hermes as provider custom:omni-101 (config.yaml: custom provider name: omni-101, base_url: http://100.78.63.20:20128).

httpcore.ConnectError: [Errno 111] Connection refused
   → openai.APIConnectionError: Connection error.
      → RuntimeError: Connection error.        (as recorded in executions.db)

hermes-errors.log carried 328 Errno 111 occurrences in the live file alone (504 across the rotations during the window). Every failing run recorded the identical leaf string, so all 24 model-call failures are one fault, not 24 different ones.

Timeline (AWST)

Time Event
10-14 08:30:27 First failed run — Gmail Action Digest (08eb86f02d37). Isolated: 06:00 and 09:00–11:45 runs were clean.
10-14 12:00 Cascade — all three noon jobs fail (b63ed5817212, ea79a2107e36, ecc1fc13655a) and the watchdog reddens at 12:15/12:30.
10-14 15:00 / 18:00 / 22:00 Mosquito Tracker ×2, Embedding Coverage, Health Monitor all fail.
10-14 23:59 The nightly Wiki Audit itself failed — so the 10-14 window was never audited.
10-15 03:00 wiki-stocktake-maintenance (1a8e90ea2271) failed — no freshness-20261015.txt written.
10-15 07:00–10:00 Prime-time digests fail: Tech & AI, Daily Marketing Job Radar, Political News, AI Newsletter Curation for Patrick.
10-15 12:00 Noon jobs fail again.
10-15 18:00:41 Last genuine failure — Embedding Coverage Monitor.
10-15 18:30:26 Last watchdog red.
10-15 22:01:56 Recovery marker — Mosquito Tracker (1b31187397a5) completes on omni-101; streak 4 → 0.
10-16 00:00–00:10 Embedding Coverage (00:02:29), Health Monitor (00:09:48) and the Watchdog (00:01:36) all complete — three streaks cleared.
10-16 00:1x Live re-probe: GET /v1/models on 100.78.63.20:20128 → HTTP 401 in 10 ms (alive, auth required). Same on omniroute:20128 / 100.88.81.19.

Blast radius

Job Name Failures First → last
b63ed5817212 Mnemosyne Embedding Coverage (6-hourly) 6 10-14 12:00 → 10-15 18:00
1b31187397a5 Mosquito Tracker (4×/day) 4 10-14 15:00 → 10-15 15:00
ea79a2107e36 Mnemosyne Health Monitor (12-hourly) 3 10-14 12:00 → 10-15 12:00
08eb86f02d37 Gmail Action Digest (daily 08:30) 2 10-14 08:30 → 10-15 08:30
ecc1fc13655a FreshRSS Midday Briefing (daily 12:00) 2 10-14 12:00 → 10-15 12:00
85edf6772bca Tech & AI Newsletter Digest (daily 07:00) 1 10-15 07:00
dca8482e4f76 Daily Marketing Job Radar (weekdays 07:00) 1 10-15 07:00
0e519874d280 Political News Digest (Mon/Thu 08:00) 1 10-15 08:00
06a0a1c8704f AI Newsletter Curation for Patrick (Thu 10:00) 1 10-15 10:00
c8d52904e1d6 Perth New Restaurant (Thu 18:00) 1 10-15 18:00
1a8e90ea2271 wiki-stocktake-maintenance (daily 03:00) 1 10-15 03:00
cbfe434a3b5e Wiki Audit (daily 23:59) 1 10-14 23:59
96eb95ce12f2 Mnemosyne Watchdog (every 15 min) 12 symptom only — see below

Day tallies: 10-14 = 160 completed / 12 failed (172), 10-15 = 151 completed / 24 failed / 1 running (176).

User-visible cost: several digests simply never went out — Tech & AI, Daily Marketing Job Radar, Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick, and the weekly Perth New Restaurant (missed outright; next chance 10-22).

What stayed green — and what that does not prove

Script-only jobs kept running normally. The fleet holds 10 no_agent jobs (no LLM turn: Mnemosyne Auto-Consolidate, DB Backup, Watchdog, Daily Birthday Checker, Job Tracker Dashboard Refresh, Hermes Config Backup to Gitea, weekly-save-restricted-bot-resources, Evening Stand-Down Briefing, plus two disabled). Completing runs inside the storm window by provider: 210 no_agent / no provider + 1 no_agent on custom:omni — that is the entire green of the fleet, and zero model calls completed in the 34 hours.

Every model-calling job in the fleet is on the one node that failed. Verified from jobs.json: all 18 enabled agent-mode jobs are pinned to custom:omni-101 (smart-route ×10, free-stack ×6, deepseek-paid-smart ×2).

⚠ Correction to a first pass at this window: Evening Stand-Down Briefing (e530c89f2be0) carries provider: custom:omni — but it is also no_agent: true, a script-only job whose provider/model fields are vestigial. Its clean 18:45 runs on both 10-14 and 10-15 are therefore not evidence that CT600 stayed healthy; it never called a model. No cron job exercises custom:omni with an LLM turn (d40085da631c, the only other custom:omni pin, is disabled), so the cron fleet cannot localise this outage between the two OmniRoute nodes. What is reachable is the leaf error itself — Connection refused to 100.78.63.20:20128 — which points at omniroute-1.

The watchdog's reds are derivative

96eb95ce12f2 (Mnemosyne Watchdog, no_agent, script-only) accounts for 12 of the 36 failure rows — it reddens when its log scan finds errors, so during a provider storm it is reporting the weather, not the fault. All six of its incidents auto-resolved. Do not chase watchdog reds as a root cause; read executions.error on the jobs that actually make model calls. It also reddens in bursts (12:15/12:30, then 18:15/18:30), not continuously — its trigger is a log sentinel, so it under-reports the window rather than bracketing it.

Incident bookkeeping

Open incidents after the storm: 8 alerted + 2 detected + 2 pre-existing = 12 open (9 alerted / 3 detected). Each clears only on the next successful run of its own job, which for the weekly and twice-weekly jobs means days of visible noise: 08eb86f02d37 10-16 08:30 · dca8482e4f76 10-16 07:00 · 85edf6772bca 10-16 07:00 · ecc1fc13655a 10-16 12:00 · 1a8e90ea2271 10-16 03:00 · 0e519874d280 10-19 (Mon) · 06a0a1c8704f 10-22 (Thu) · c8d52904e1d6 10-22 (Thu). Pre-existing and not storm-caused: 1b23c042e17d (alerted since 10-09, next slot 10-16 08:00) and 0e753e_e7f8194a7d06 (FB Birthday Monthly, detected since 10-01, next slot 11-01).

Lessons

  1. The entire scheduled estate rides one OmniRoute node. All 18 enabled model-calling jobs are pinned to custom:omni-101; the only other model-capable cron pin, custom:omni, sits on a no_agent job. Two live OmniRoute nodes exist (omniroute CT600 at 100.88.81.19, omniroute-1 at 100.78.63.20) with no failover between them, so one node's availability gates the whole estate's output. This is the second multi-day storm from the same topology → incidents/2026-10-09-cron-provider-timeout-storm.
  2. provider on a no_agent job is decorative. Never read a script-only job's green row as evidence of provider health — check no_agent before concluding anything from a clean run.
  3. Both Mnemosyne monitors sit on the same node as the memory system they watch — both failed, so coverage figures went unrefreshed for the whole window (last explicit figures remain 2026-10-12). Monitoring that shares a blast radius with its subject is degraded monitoring.
  4. A log-sentinel watchdog multiplies failure counts. 36 failure rows were really 24, across 12 workers. Tally per-job, not per-row, or the scale reads wrong.
  5. A failed nightly audit leaves a hole, not a gap in coverage. The 10-14 audit died on this same storm, so this page covers two windows (10-14 → 10-16).