incidents/2026-10-14-omniroute-node-connection-refused-storm.md was at exactly 21d and
would have tripped the threshold on the next run. Its residual claim ("omniroute-1
answers again") was re-probed live and still holds: TCP connect to 100.78.63.20:20128
succeeds (no Errno 111), the Tailscale peer reads Online:true, and a keyed
POST /v1/chat/completions returns HTTP 200. Page re-verified (updated/verified_on ->
2026-11-06, review_after -> 2026-12-06) with the probe recorded on the page.
New trap recorded on the page and in systems/scheduled-tasks.md: on omniroute-1 the
RAW model id deepseek/deepseek-v4-flash returns HTTP 401 "No active credentials for
provider: deepseek", while the fleet's deepseek-paid-smart alias returns HTTP 200 and
resolves to that same model. The 401 is an alias-routing artefact, not an outage - probe
the alias the fleet pins, never the raw model id. (A first probe using the wrong
key_env also produced a 401; omni-101's key env is HERMES_CUSTOM_100_78_63_20_20128_API_KEY.)
Freshness re-run after the edits: 0 stale / 124 fresh / 3 no-date of 127 pages.
Quartz rebuilt and redeployed. Vault mirror synced to both conventions (sha-verified).
154 lines
10 KiB
Markdown
154 lines
10 KiB
Markdown
---
|
||
title: OmniRoute `omni-101` Node Outage — Connection-Refused Storm
|
||
type: concept
|
||
status: resolved
|
||
created: 2026-10-16
|
||
updated: 2026-11-06
|
||
verified_on: 2026-11-06
|
||
review_after: 2026-12-06
|
||
confidence: high
|
||
tags: [cron, router, provider, infrastructure]
|
||
sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, hermes-errors-log, live-probe]
|
||
---
|
||
|
||
# OmniRoute `omni-101` Node Outage — Connection-Refused Storm (2026-10-14 → 2026-10-15)
|
||
|
||
**Window:** 2026-10-14 08:30 → 2026-10-15 18:07 AWST (**~34 hours**)
|
||
**Status:** **Resolved.** `omniroute-1` answers again; four scheduled jobs have since completed on it.
|
||
|
||
**Re-verified 2026-11-06 (stocktake) — the residual claim still holds, and the node is genuinely healthy.**
|
||
Live probe of the incident's own endpoint: TCP **connect to `100.78.63.20:20128` succeeds** (no
|
||
`Errno 111 Connection refused` — the incident's root cause has not recurred), the Tailscale peer
|
||
`omniroute-1` reads `Online: true`, and a keyed `POST /v1/chat/completions` returns **HTTP 200**.
|
||
`auto/best-chat` → `codestral-2508` (200), `auto/best-reasoning` → `deepseek-ai/deepseek-v4.1-flash` (200),
|
||
and the fleet's `deepseek-paid-smart` pin → `deepseek/deepseek-v4-flash` (**200**) — confirming the
|
||
[[systems/scheduled-tasks]] note that this pin dispatches on the `omniroute-1` node. ⚠ **One trap worth
|
||
recording:** requesting the **raw** model id `deepseek/deepseek-v4-flash` (rather than the
|
||
`deepseek-paid-smart` alias) returns **HTTP 401 `No active credentials for provider: deepseek`** — that is
|
||
an alias-routing artefact on this node, **not** an outage and **not** the incident recurring. Probe the
|
||
**alias** the fleet actually uses.
|
||
**Impact:** **36 failed execution rows across 13 jobs** — 24 genuine model-call failures on 12 jobs, plus
|
||
12 watchdog *symptom* reds. **Every model call attempted in the window failed** (see below). For scale:
|
||
10-12 recorded `0 failed` and 10-13 recorded `1`.
|
||
**Severity:** the largest failure count since the 2026-10-09 storm (15 in one day); this one ran **two days**
|
||
and silently suppressed most of the day's digests.
|
||
|
||
## Root cause
|
||
|
||
`TCP connect refused` on **`http://100.78.63.20:20128`** — the **`omniroute-1`** node (Tailscale peer
|
||
`omniroute-1`, 100.78.63.20), registered in Hermes as provider **`custom:omni-101`**
|
||
(`config.yaml`: custom provider `name: omni-101`, `base_url: http://100.78.63.20:20128`).
|
||
|
||
```
|
||
httpcore.ConnectError: [Errno 111] Connection refused
|
||
→ openai.APIConnectionError: Connection error.
|
||
→ RuntimeError: Connection error. (as recorded in executions.db)
|
||
```
|
||
`hermes-errors.log` carried **328 `Errno 111`** occurrences in the live file alone (504 across the
|
||
rotations during the window). Every failing run recorded the identical leaf string, so all 24 model-call
|
||
failures are **one fault**, not 24 different ones.
|
||
|
||
## Timeline (AWST)
|
||
|
||
| Time | Event |
|
||
|---|---|
|
||
| **10-14 08:30:27** | First failed run — Gmail Action Digest (`08eb86f02d37`). Isolated: 06:00 and 09:00–11:45 runs were clean. |
|
||
| **10-14 12:00** | Cascade — all three noon jobs fail (`b63ed5817212`, `ea79a2107e36`, `ecc1fc13655a`) and the watchdog reddens at 12:15/12:30. |
|
||
| **10-14 15:00 / 18:00 / 22:00** | Mosquito Tracker ×2, Embedding Coverage, Health Monitor all fail. |
|
||
| **10-14 23:59** | **The nightly Wiki Audit itself failed** — so the 10-14 window was never audited. |
|
||
| **10-15 03:00** | `wiki-stocktake-maintenance` (`1a8e90ea2271`) failed — no `freshness-20261015.txt` written. |
|
||
| **10-15 07:00–10:00** | Prime-time digests fail: Tech & AI, Daily Marketing Job Radar, Political News, AI Newsletter Curation for Patrick. |
|
||
| **10-15 12:00** | Noon jobs fail again. |
|
||
| **10-15 18:00:41** | **Last genuine failure** — Embedding Coverage Monitor. |
|
||
| **10-15 18:30:26** | Last watchdog red. |
|
||
| **10-15 22:01:56** | **Recovery marker** — Mosquito Tracker (`1b31187397a5`) completes on `omni-101`; streak 4 → 0. |
|
||
| **10-16 00:00–00:10** | Embedding Coverage (00:02:29), Health Monitor (00:09:48) and the Watchdog (00:01:36) all complete — three streaks cleared. |
|
||
| **10-16 00:1x** | **Live re-probe:** `GET /v1/models` on 100.78.63.20:20128 → **HTTP 401** in 10 ms (alive, auth required). Same on `omniroute:20128` / 100.88.81.19. |
|
||
|
||
## Blast radius
|
||
|
||
| Job | Name | Failures | First → last |
|
||
|---|---|---|---|
|
||
| `b63ed5817212` | Mnemosyne Embedding Coverage (6-hourly) | 6 | 10-14 12:00 → 10-15 18:00 |
|
||
| `1b31187397a5` | Mosquito Tracker (4×/day) | 4 | 10-14 15:00 → 10-15 15:00 |
|
||
| `ea79a2107e36` | Mnemosyne Health Monitor (12-hourly) | 3 | 10-14 12:00 → 10-15 12:00 |
|
||
| `08eb86f02d37` | Gmail Action Digest (daily 08:30) | 2 | 10-14 08:30 → 10-15 08:30 |
|
||
| `ecc1fc13655a` | FreshRSS Midday Briefing (daily 12:00) | 2 | 10-14 12:00 → 10-15 12:00 |
|
||
| `85edf6772bca` | Tech & AI Newsletter Digest (daily 07:00) | 1 | 10-15 07:00 |
|
||
| `dca8482e4f76` | Daily Marketing Job Radar (weekdays 07:00) | 1 | 10-15 07:00 |
|
||
| `0e519874d280` | Political News Digest (Mon/Thu 08:00) | 1 | 10-15 08:00 |
|
||
| `06a0a1c8704f` | AI Newsletter Curation for Patrick (Thu 10:00) | 1 | 10-15 10:00 |
|
||
| `c8d52904e1d6` | Perth New Restaurant (Thu 18:00) | 1 | 10-15 18:00 |
|
||
| `1a8e90ea2271` | wiki-stocktake-maintenance (daily 03:00) | 1 | 10-15 03:00 |
|
||
| `cbfe434a3b5e` | Wiki Audit (daily 23:59) | 1 | 10-14 23:59 |
|
||
| `96eb95ce12f2` | Mnemosyne Watchdog (every 15 min) | 12 | **symptom only** — see below |
|
||
|
||
Day tallies: **10-14 = 160 completed / 12 failed** (172), **10-15 = 151 completed / 24 failed / 1 running** (176).
|
||
|
||
**User-visible cost:** several digests simply never went out — Tech & AI, Daily Marketing Job Radar,
|
||
Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick, and the
|
||
weekly Perth New Restaurant (missed outright; next chance 10-22).
|
||
|
||
## What stayed green — and what that does *not* prove
|
||
|
||
**Script-only jobs kept running normally.** The fleet holds **10 `no_agent` jobs** (no LLM turn: Mnemosyne
|
||
Auto-Consolidate, DB Backup, Watchdog, Daily Birthday Checker, Job Tracker Dashboard Refresh, Hermes Config
|
||
Backup to Gitea, weekly-save-restricted-bot-resources, Evening Stand-Down Briefing, plus two disabled).
|
||
Completing runs inside the storm window by provider: **210 `no_agent` / no provider + 1 `no_agent` on
|
||
`custom:omni`** — that is the *entire* green of the fleet, and **zero model calls completed in the 34 hours.**
|
||
|
||
**Every model-calling job in the fleet is on the one node that failed.** Verified from `jobs.json`: **all 18
|
||
enabled agent-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6,
|
||
`deepseek-paid-smart` ×2).
|
||
|
||
⚠ **Correction to a first pass at this window:** Evening Stand-Down Briefing (`e530c89f2be0`) carries
|
||
`provider: custom:omni` — but it is also **`no_agent: true`**, a script-only job whose provider/model fields
|
||
are vestigial. Its clean 18:45 runs on *both* 10-14 and 10-15 are therefore **not** evidence that CT600
|
||
stayed healthy; it never called a model. No cron job exercises `custom:omni` with an LLM turn
|
||
(`d40085da631c`, the only other `custom:omni` pin, is **disabled**), so the cron fleet **cannot** localise
|
||
this outage between the two OmniRoute nodes. What *is* reachable is the leaf error itself — `Connection
|
||
refused` to `100.78.63.20:20128` — which points at **`omniroute-1`**.
|
||
|
||
## The watchdog's reds are derivative
|
||
|
||
`96eb95ce12f2` (Mnemosyne Watchdog, `no_agent`, script-only) accounts for **12 of the 36 failure rows** — it
|
||
reddens when its log scan finds errors, so during a provider storm it is *reporting the weather, not the
|
||
fault*. All six of its incidents auto-resolved. Do not chase watchdog reds as a root cause; read
|
||
`executions.error` on the jobs that actually make model calls. It also reddens in **bursts** (12:15/12:30,
|
||
then 18:15/18:30), not continuously — its trigger is a log sentinel, so it under-reports the window rather
|
||
than bracketing it.
|
||
|
||
## Incident bookkeeping
|
||
|
||
Open incidents after the storm: **8 `alerted`** + **2 `detected`** + 2 pre-existing = **12 open**
|
||
(9 alerted / 3 detected). Each clears **only on the next successful run of its own job**, which for the
|
||
weekly and twice-weekly jobs means days of visible noise:
|
||
`08eb86f02d37` 10-16 08:30 · `dca8482e4f76` 10-16 07:00 · `85edf6772bca` 10-16 07:00 ·
|
||
`ecc1fc13655a` 10-16 12:00 · `1a8e90ea2271` 10-16 03:00 · `0e519874d280` 10-19 (Mon) ·
|
||
`06a0a1c8704f` 10-22 (Thu) · `c8d52904e1d6` 10-22 (Thu).
|
||
Pre-existing and **not** storm-caused: `1b23c042e17d` (alerted since 10-09, next slot 10-16 08:00) and
|
||
`0e753e_e7f8194a7d06` (FB Birthday Monthly, detected since 10-01, next slot 11-01).
|
||
|
||
## Lessons
|
||
|
||
1. **The entire scheduled estate rides one OmniRoute node.** All 18 enabled model-calling jobs are pinned to
|
||
`custom:omni-101`; the only other model-capable cron pin, `custom:omni`, sits on a `no_agent` job. Two
|
||
live OmniRoute nodes exist (`omniroute` CT600 at 100.88.81.19, `omniroute-1` at 100.78.63.20) with **no
|
||
failover between them**, so one node's availability gates the whole estate's output. This is the second
|
||
multi-day storm from the same topology → [[incidents/2026-10-09-cron-provider-timeout-storm]].
|
||
2. **`provider` on a `no_agent` job is decorative.** Never read a script-only job's green row as evidence of
|
||
provider health — check `no_agent` before concluding anything from a clean run.
|
||
3. **Both Mnemosyne monitors sit on the same node as the memory system they watch** — both failed, so
|
||
coverage figures went unrefreshed for the whole window (last explicit figures remain 2026-10-12).
|
||
Monitoring that shares a blast radius with its subject is degraded monitoring.
|
||
4. **A log-sentinel watchdog multiplies failure counts.** 36 failure rows were really 24, across 12 workers.
|
||
Tally per-job, not per-row, or the scale reads wrong.
|
||
5. **A failed nightly audit leaves a hole, not a gap in coverage.** The 10-14 audit died on this same storm,
|
||
so this page covers **two** windows (10-14 → 10-16).
|
||
|
||
## Related
|
||
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — the earlier storm; same topology, same failure class.
|
||
- [[systems/omniroute]] — node layout, providers and pins.
|
||
- [[systems/scheduled-tasks]] — per-window fleet tally and incident ledger.
|
||
- [[systems/mnemosyne]] — the two coverage monitors that went dark.
|