Files
agent-estate-wiki/incidents/2026-10-14-omniroute-node-connection-refused-storm.md
T
Tony0410 2d5219b068 stocktake 2026-11-06 (pass 2): re-verify the one page sitting on the 21-day threshold
incidents/2026-10-14-omniroute-node-connection-refused-storm.md was at exactly 21d and
would have tripped the threshold on the next run. Its residual claim ("omniroute-1
answers again") was re-probed live and still holds: TCP connect to 100.78.63.20:20128
succeeds (no Errno 111), the Tailscale peer reads Online:true, and a keyed
POST /v1/chat/completions returns HTTP 200. Page re-verified (updated/verified_on ->
2026-11-06, review_after -> 2026-12-06) with the probe recorded on the page.

New trap recorded on the page and in systems/scheduled-tasks.md: on omniroute-1 the
RAW model id deepseek/deepseek-v4-flash returns HTTP 401 "No active credentials for
provider: deepseek", while the fleet's deepseek-paid-smart alias returns HTTP 200 and
resolves to that same model. The 401 is an alias-routing artefact, not an outage - probe
the alias the fleet pins, never the raw model id. (A first probe using the wrong
key_env also produced a 401; omni-101's key env is HERMES_CUSTOM_100_78_63_20_20128_API_KEY.)

Freshness re-run after the edits: 0 stale / 124 fresh / 3 no-date of 127 pages.
Quartz rebuilt and redeployed. Vault mirror synced to both conventions (sha-verified).
2026-11-06 04:07:12 +08:00

154 lines
10 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: OmniRoute `omni-101` Node Outage — Connection-Refused Storm
type: concept
status: resolved
created: 2026-10-16
updated: 2026-11-06
verified_on: 2026-11-06
review_after: 2026-12-06
confidence: high
tags: [cron, router, provider, infrastructure]
sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, hermes-errors-log, live-probe]
---
# OmniRoute `omni-101` Node Outage — Connection-Refused Storm (2026-10-14 → 2026-10-15)
**Window:** 2026-10-14 08:30 → 2026-10-15 18:07 AWST (**~34 hours**)
**Status:** **Resolved.** `omniroute-1` answers again; four scheduled jobs have since completed on it.
**Re-verified 2026-11-06 (stocktake) — the residual claim still holds, and the node is genuinely healthy.**
Live probe of the incident's own endpoint: TCP **connect to `100.78.63.20:20128` succeeds** (no
`Errno 111 Connection refused` — the incident's root cause has not recurred), the Tailscale peer
`omniroute-1` reads `Online: true`, and a keyed `POST /v1/chat/completions` returns **HTTP 200**.
`auto/best-chat` → `codestral-2508` (200), `auto/best-reasoning` → `deepseek-ai/deepseek-v4.1-flash` (200),
and the fleet's `deepseek-paid-smart` pin → `deepseek/deepseek-v4-flash` (**200**) — confirming the
[[systems/scheduled-tasks]] note that this pin dispatches on the `omniroute-1` node. ⚠ **One trap worth
recording:** requesting the **raw** model id `deepseek/deepseek-v4-flash` (rather than the
`deepseek-paid-smart` alias) returns **HTTP 401 `No active credentials for provider: deepseek`** — that is
an alias-routing artefact on this node, **not** an outage and **not** the incident recurring. Probe the
**alias** the fleet actually uses.
**Impact:** **36 failed execution rows across 13 jobs** — 24 genuine model-call failures on 12 jobs, plus
12 watchdog *symptom* reds. **Every model call attempted in the window failed** (see below). For scale:
10-12 recorded `0 failed` and 10-13 recorded `1`.
**Severity:** the largest failure count since the 2026-10-09 storm (15 in one day); this one ran **two days**
and silently suppressed most of the day's digests.
## Root cause
`TCP connect refused` on **`http://100.78.63.20:20128`** — the **`omniroute-1`** node (Tailscale peer
`omniroute-1`, 100.78.63.20), registered in Hermes as provider **`custom:omni-101`**
(`config.yaml`: custom provider `name: omni-101`, `base_url: http://100.78.63.20:20128`).
```
httpcore.ConnectError: [Errno 111] Connection refused
→ openai.APIConnectionError: Connection error.
→ RuntimeError: Connection error. (as recorded in executions.db)
```
`hermes-errors.log` carried **328 `Errno 111`** occurrences in the live file alone (504 across the
rotations during the window). Every failing run recorded the identical leaf string, so all 24 model-call
failures are **one fault**, not 24 different ones.
## Timeline (AWST)
| Time | Event |
|---|---|
| **10-14 08:30:27** | First failed run — Gmail Action Digest (`08eb86f02d37`). Isolated: 06:00 and 09:00–11:45 runs were clean. |
| **10-14 12:00** | Cascade — all three noon jobs fail (`b63ed5817212`, `ea79a2107e36`, `ecc1fc13655a`) and the watchdog reddens at 12:15/12:30. |
| **10-14 15:00 / 18:00 / 22:00** | Mosquito Tracker ×2, Embedding Coverage, Health Monitor all fail. |
| **10-14 23:59** | **The nightly Wiki Audit itself failed** — so the 10-14 window was never audited. |
| **10-15 03:00** | `wiki-stocktake-maintenance` (`1a8e90ea2271`) failed — no `freshness-20261015.txt` written. |
| **10-15 07:00–10:00** | Prime-time digests fail: Tech & AI, Daily Marketing Job Radar, Political News, AI Newsletter Curation for Patrick. |
| **10-15 12:00** | Noon jobs fail again. |
| **10-15 18:00:41** | **Last genuine failure** — Embedding Coverage Monitor. |
| **10-15 18:30:26** | Last watchdog red. |
| **10-15 22:01:56** | **Recovery marker** — Mosquito Tracker (`1b31187397a5`) completes on `omni-101`; streak 4 → 0. |
| **10-16 00:00–00:10** | Embedding Coverage (00:02:29), Health Monitor (00:09:48) and the Watchdog (00:01:36) all complete — three streaks cleared. |
| **10-16 00:1x** | **Live re-probe:** `GET /v1/models` on 100.78.63.20:20128 → **HTTP 401** in 10 ms (alive, auth required). Same on `omniroute:20128` / 100.88.81.19. |
## Blast radius
| Job | Name | Failures | First → last |
|---|---|---|---|
| `b63ed5817212` | Mnemosyne Embedding Coverage (6-hourly) | 6 | 10-14 12:00 → 10-15 18:00 |
| `1b31187397a5` | Mosquito Tracker (4×/day) | 4 | 10-14 15:00 → 10-15 15:00 |
| `ea79a2107e36` | Mnemosyne Health Monitor (12-hourly) | 3 | 10-14 12:00 → 10-15 12:00 |
| `08eb86f02d37` | Gmail Action Digest (daily 08:30) | 2 | 10-14 08:30 → 10-15 08:30 |
| `ecc1fc13655a` | FreshRSS Midday Briefing (daily 12:00) | 2 | 10-14 12:00 → 10-15 12:00 |
| `85edf6772bca` | Tech & AI Newsletter Digest (daily 07:00) | 1 | 10-15 07:00 |
| `dca8482e4f76` | Daily Marketing Job Radar (weekdays 07:00) | 1 | 10-15 07:00 |
| `0e519874d280` | Political News Digest (Mon/Thu 08:00) | 1 | 10-15 08:00 |
| `06a0a1c8704f` | AI Newsletter Curation for Patrick (Thu 10:00) | 1 | 10-15 10:00 |
| `c8d52904e1d6` | Perth New Restaurant (Thu 18:00) | 1 | 10-15 18:00 |
| `1a8e90ea2271` | wiki-stocktake-maintenance (daily 03:00) | 1 | 10-15 03:00 |
| `cbfe434a3b5e` | Wiki Audit (daily 23:59) | 1 | 10-14 23:59 |
| `96eb95ce12f2` | Mnemosyne Watchdog (every 15 min) | 12 | **symptom only** — see below |
Day tallies: **10-14 = 160 completed / 12 failed** (172), **10-15 = 151 completed / 24 failed / 1 running** (176).
**User-visible cost:** several digests simply never went out — Tech & AI, Daily Marketing Job Radar,
Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick, and the
weekly Perth New Restaurant (missed outright; next chance 10-22).
## What stayed green — and what that does *not* prove
**Script-only jobs kept running normally.** The fleet holds **10 `no_agent` jobs** (no LLM turn: Mnemosyne
Auto-Consolidate, DB Backup, Watchdog, Daily Birthday Checker, Job Tracker Dashboard Refresh, Hermes Config
Backup to Gitea, weekly-save-restricted-bot-resources, Evening Stand-Down Briefing, plus two disabled).
Completing runs inside the storm window by provider: **210 `no_agent` / no provider + 1 `no_agent` on
`custom:omni`** — that is the *entire* green of the fleet, and **zero model calls completed in the 34 hours.**
**Every model-calling job in the fleet is on the one node that failed.** Verified from `jobs.json`: **all 18
enabled agent-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2).
⚠ **Correction to a first pass at this window:** Evening Stand-Down Briefing (`e530c89f2be0`) carries
`provider: custom:omni` — but it is also **`no_agent: true`**, a script-only job whose provider/model fields
are vestigial. Its clean 18:45 runs on *both* 10-14 and 10-15 are therefore **not** evidence that CT600
stayed healthy; it never called a model. No cron job exercises `custom:omni` with an LLM turn
(`d40085da631c`, the only other `custom:omni` pin, is **disabled**), so the cron fleet **cannot** localise
this outage between the two OmniRoute nodes. What *is* reachable is the leaf error itself — `Connection
refused` to `100.78.63.20:20128` — which points at **`omniroute-1`**.
## The watchdog's reds are derivative
`96eb95ce12f2` (Mnemosyne Watchdog, `no_agent`, script-only) accounts for **12 of the 36 failure rows** — it
reddens when its log scan finds errors, so during a provider storm it is *reporting the weather, not the
fault*. All six of its incidents auto-resolved. Do not chase watchdog reds as a root cause; read
`executions.error` on the jobs that actually make model calls. It also reddens in **bursts** (12:15/12:30,
then 18:15/18:30), not continuously — its trigger is a log sentinel, so it under-reports the window rather
than bracketing it.
## Incident bookkeeping
Open incidents after the storm: **8 `alerted`** + **2 `detected`** + 2 pre-existing = **12 open**
(9 alerted / 3 detected). Each clears **only on the next successful run of its own job**, which for the
weekly and twice-weekly jobs means days of visible noise:
`08eb86f02d37` 10-16 08:30 · `dca8482e4f76` 10-16 07:00 · `85edf6772bca` 10-16 07:00 ·
`ecc1fc13655a` 10-16 12:00 · `1a8e90ea2271` 10-16 03:00 · `0e519874d280` 10-19 (Mon) ·
`06a0a1c8704f` 10-22 (Thu) · `c8d52904e1d6` 10-22 (Thu).
Pre-existing and **not** storm-caused: `1b23c042e17d` (alerted since 10-09, next slot 10-16 08:00) and
`0e753e_e7f8194a7d06` (FB Birthday Monthly, detected since 10-01, next slot 11-01).
## Lessons
1. **The entire scheduled estate rides one OmniRoute node.** All 18 enabled model-calling jobs are pinned to
`custom:omni-101`; the only other model-capable cron pin, `custom:omni`, sits on a `no_agent` job. Two
live OmniRoute nodes exist (`omniroute` CT600 at 100.88.81.19, `omniroute-1` at 100.78.63.20) with **no
failover between them**, so one node's availability gates the whole estate's output. This is the second
multi-day storm from the same topology → [[incidents/2026-10-09-cron-provider-timeout-storm]].
2. **`provider` on a `no_agent` job is decorative.** Never read a script-only job's green row as evidence of
provider health — check `no_agent` before concluding anything from a clean run.
3. **Both Mnemosyne monitors sit on the same node as the memory system they watch** — both failed, so
coverage figures went unrefreshed for the whole window (last explicit figures remain 2026-10-12).
Monitoring that shares a blast radius with its subject is degraded monitoring.
4. **A log-sentinel watchdog multiplies failure counts.** 36 failure rows were really 24, across 12 workers.
Tally per-job, not per-row, or the scale reads wrong.
5. **A failed nightly audit leaves a hole, not a gap in coverage.** The 10-14 audit died on this same storm,
so this page covers **two** windows (10-14 → 10-16).
## Related
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — the earlier storm; same topology, same failure class.
- [[systems/omniroute]] — node layout, providers and pins.
- [[systems/scheduled-tasks]] — per-window fleet tally and incident ledger.
- [[systems/mnemosyne]] — the two coverage monitors that went dark.