--- title: OmniRoute `omni-101` Node Outage — Connection-Refused Storm type: concept status: resolved created: 2026-10-16 updated: 2026-10-16 verified_on: 2026-10-16 review_after: 2026-10-23 confidence: high tags: [cron, router, provider, infrastructure] sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, hermes-errors-log, live-probe] --- # OmniRoute `omni-101` Node Outage — Connection-Refused Storm (2026-10-14 → 2026-10-15) **Window:** 2026-10-14 08:30 → 2026-10-15 18:07 AWST (**~34 hours**) **Status:** **Resolved.** `omniroute-1` answers again; four scheduled jobs have since completed on it. **Impact:** **36 failed execution rows across 13 jobs** — 24 genuine model-call failures on 12 jobs, plus 12 watchdog *symptom* reds. **Every model call attempted in the window failed** (see below). For scale: 10-12 recorded `0 failed` and 10-13 recorded `1`. **Severity:** the largest failure count since the 2026-10-09 storm (15 in one day); this one ran **two days** and silently suppressed most of the day's digests. ## Root cause `TCP connect refused` on **`http://100.78.63.20:20128`** — the **`omniroute-1`** node (Tailscale peer `omniroute-1`, 100.78.63.20), registered in Hermes as provider **`custom:omni-101`** (`config.yaml`: custom provider `name: omni-101`, `base_url: http://100.78.63.20:20128`). ``` httpcore.ConnectError: [Errno 111] Connection refused → openai.APIConnectionError: Connection error. → RuntimeError: Connection error. (as recorded in executions.db) ``` `hermes-errors.log` carried **328 `Errno 111`** occurrences in the live file alone (504 across the rotations during the window). Every failing run recorded the identical leaf string, so all 24 model-call failures are **one fault**, not 24 different ones. ## Timeline (AWST) | Time | Event | |---|---| | **10-14 08:30:27** | First failed run — Gmail Action Digest (`08eb86f02d37`). Isolated: 06:00 and 09:00–11:45 runs were clean. | | **10-14 12:00** | Cascade — all three noon jobs fail (`b63ed5817212`, `ea79a2107e36`, `ecc1fc13655a`) and the watchdog reddens at 12:15/12:30. | | **10-14 15:00 / 18:00 / 22:00** | Mosquito Tracker ×2, Embedding Coverage, Health Monitor all fail. | | **10-14 23:59** | **The nightly Wiki Audit itself failed** — so the 10-14 window was never audited. | | **10-15 03:00** | `wiki-stocktake-maintenance` (`1a8e90ea2271`) failed — no `freshness-20261015.txt` written. | | **10-15 07:00–10:00** | Prime-time digests fail: Tech & AI, Daily Marketing Job Radar, Political News, AI Newsletter Curation for Patrick. | | **10-15 12:00** | Noon jobs fail again. | | **10-15 18:00:41** | **Last genuine failure** — Embedding Coverage Monitor. | | **10-15 18:30:26** | Last watchdog red. | | **10-15 22:01:56** | **Recovery marker** — Mosquito Tracker (`1b31187397a5`) completes on `omni-101`; streak 4 → 0. | | **10-16 00:00–00:10** | Embedding Coverage (00:02:29), Health Monitor (00:09:48) and the Watchdog (00:01:36) all complete — three streaks cleared. | | **10-16 00:1x** | **Live re-probe:** `GET /v1/models` on 100.78.63.20:20128 → **HTTP 401** in 10 ms (alive, auth required). Same on `omniroute:20128` / 100.88.81.19. | ## Blast radius | Job | Name | Failures | First → last | |---|---|---|---| | `b63ed5817212` | Mnemosyne Embedding Coverage (6-hourly) | 6 | 10-14 12:00 → 10-15 18:00 | | `1b31187397a5` | Mosquito Tracker (4×/day) | 4 | 10-14 15:00 → 10-15 15:00 | | `ea79a2107e36` | Mnemosyne Health Monitor (12-hourly) | 3 | 10-14 12:00 → 10-15 12:00 | | `08eb86f02d37` | Gmail Action Digest (daily 08:30) | 2 | 10-14 08:30 → 10-15 08:30 | | `ecc1fc13655a` | FreshRSS Midday Briefing (daily 12:00) | 2 | 10-14 12:00 → 10-15 12:00 | | `85edf6772bca` | Tech & AI Newsletter Digest (daily 07:00) | 1 | 10-15 07:00 | | `dca8482e4f76` | Daily Marketing Job Radar (weekdays 07:00) | 1 | 10-15 07:00 | | `0e519874d280` | Political News Digest (Mon/Thu 08:00) | 1 | 10-15 08:00 | | `06a0a1c8704f` | AI Newsletter Curation for Patrick (Thu 10:00) | 1 | 10-15 10:00 | | `c8d52904e1d6` | Perth New Restaurant (Thu 18:00) | 1 | 10-15 18:00 | | `1a8e90ea2271` | wiki-stocktake-maintenance (daily 03:00) | 1 | 10-15 03:00 | | `cbfe434a3b5e` | Wiki Audit (daily 23:59) | 1 | 10-14 23:59 | | `96eb95ce12f2` | Mnemosyne Watchdog (every 15 min) | 12 | **symptom only** — see below | Day tallies: **10-14 = 160 completed / 12 failed** (172), **10-15 = 151 completed / 24 failed / 1 running** (176). **User-visible cost:** several digests simply never went out — Tech & AI, Daily Marketing Job Radar, Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick, and the weekly Perth New Restaurant (missed outright; next chance 10-22). ## What stayed green — and what that does *not* prove **Script-only jobs kept running normally.** The fleet holds **10 `no_agent` jobs** (no LLM turn: Mnemosyne Auto-Consolidate, DB Backup, Watchdog, Daily Birthday Checker, Job Tracker Dashboard Refresh, Hermes Config Backup to Gitea, weekly-save-restricted-bot-resources, Evening Stand-Down Briefing, plus two disabled). Completing runs inside the storm window by provider: **210 `no_agent` / no provider + 1 `no_agent` on `custom:omni`** — that is the *entire* green of the fleet, and **zero model calls completed in the 34 hours.** **Every model-calling job in the fleet is on the one node that failed.** Verified from `jobs.json`: **all 18 enabled agent-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6, `deepseek-paid-smart` ×2). ⚠ **Correction to a first pass at this window:** Evening Stand-Down Briefing (`e530c89f2be0`) carries `provider: custom:omni` — but it is also **`no_agent: true`**, a script-only job whose provider/model fields are vestigial. Its clean 18:45 runs on *both* 10-14 and 10-15 are therefore **not** evidence that CT600 stayed healthy; it never called a model. No cron job exercises `custom:omni` with an LLM turn (`d40085da631c`, the only other `custom:omni` pin, is **disabled**), so the cron fleet **cannot** localise this outage between the two OmniRoute nodes. What *is* reachable is the leaf error itself — `Connection refused` to `100.78.63.20:20128` — which points at **`omniroute-1`**. ## The watchdog's reds are derivative `96eb95ce12f2` (Mnemosyne Watchdog, `no_agent`, script-only) accounts for **12 of the 36 failure rows** — it reddens when its log scan finds errors, so during a provider storm it is *reporting the weather, not the fault*. All six of its incidents auto-resolved. Do not chase watchdog reds as a root cause; read `executions.error` on the jobs that actually make model calls. It also reddens in **bursts** (12:15/12:30, then 18:15/18:30), not continuously — its trigger is a log sentinel, so it under-reports the window rather than bracketing it. ## Incident bookkeeping Open incidents after the storm: **8 `alerted`** + **2 `detected`** + 2 pre-existing = **12 open** (9 alerted / 3 detected). Each clears **only on the next successful run of its own job**, which for the weekly and twice-weekly jobs means days of visible noise: `08eb86f02d37` 10-16 08:30 · `dca8482e4f76` 10-16 07:00 · `85edf6772bca` 10-16 07:00 · `ecc1fc13655a` 10-16 12:00 · `1a8e90ea2271` 10-16 03:00 · `0e519874d280` 10-19 (Mon) · `06a0a1c8704f` 10-22 (Thu) · `c8d52904e1d6` 10-22 (Thu). Pre-existing and **not** storm-caused: `1b23c042e17d` (alerted since 10-09, next slot 10-16 08:00) and `0e753e_e7f8194a7d06` (FB Birthday Monthly, detected since 10-01, next slot 11-01). ## Lessons 1. **The entire scheduled estate rides one OmniRoute node.** All 18 enabled model-calling jobs are pinned to `custom:omni-101`; the only other model-capable cron pin, `custom:omni`, sits on a `no_agent` job. Two live OmniRoute nodes exist (`omniroute` CT600 at 100.88.81.19, `omniroute-1` at 100.78.63.20) with **no failover between them**, so one node's availability gates the whole estate's output. This is the second multi-day storm from the same topology → [[incidents/2026-10-09-cron-provider-timeout-storm]]. 2. **`provider` on a `no_agent` job is decorative.** Never read a script-only job's green row as evidence of provider health — check `no_agent` before concluding anything from a clean run. 3. **Both Mnemosyne monitors sit on the same node as the memory system they watch** — both failed, so coverage figures went unrefreshed for the whole window (last explicit figures remain 2026-10-12). Monitoring that shares a blast radius with its subject is degraded monitoring. 4. **A log-sentinel watchdog multiplies failure counts.** 36 failure rows were really 24, across 12 workers. Tally per-job, not per-row, or the scale reads wrong. 5. **A failed nightly audit leaves a hole, not a gap in coverage.** The 10-14 audit died on this same storm, so this page covers **two** windows (10-14 → 10-16). ## Related - [[incidents/2026-10-09-cron-provider-timeout-storm]] — the earlier storm; same topology, same failure class. - [[systems/omniroute]] — node layout, providers and pins. - [[systems/scheduled-tasks]] — per-window fleet tally and incident ledger. - [[systems/mnemosyne]] — the two coverage monitors that went dark.