Files
agent-estate-wiki/incidents/2026-10-14-omniroute-node-connection-refused-storm.md
T
hermes 2061b8607b audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections
omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07
(httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error).
36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog
symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1).

STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable
pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls
completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence.

New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live).
Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute,
systems/mnemosyne (monitor blast-radius note), current-state, index, log.
2026-10-16 00:19:14 +08:00

142 lines
9.1 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: OmniRoute `omni-101` Node Outage — Connection-Refused Storm
type: concept
status: resolved
created: 2026-10-16
updated: 2026-10-16
verified_on: 2026-10-16
review_after: 2026-10-23
confidence: high
tags: [cron, router, provider, infrastructure]
sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, hermes-errors-log, live-probe]
---
# OmniRoute `omni-101` Node Outage — Connection-Refused Storm (2026-10-14 → 2026-10-15)
**Window:** 2026-10-14 08:30 → 2026-10-15 18:07 AWST (**~34 hours**)
**Status:** **Resolved.** `omniroute-1` answers again; four scheduled jobs have since completed on it.
**Impact:** **36 failed execution rows across 13 jobs** — 24 genuine model-call failures on 12 jobs, plus
12 watchdog *symptom* reds. **Every model call attempted in the window failed** (see below). For scale:
10-12 recorded `0 failed` and 10-13 recorded `1`.
**Severity:** the largest failure count since the 2026-10-09 storm (15 in one day); this one ran **two days**
and silently suppressed most of the day's digests.
## Root cause
`TCP connect refused` on **`http://100.78.63.20:20128`** — the **`omniroute-1`** node (Tailscale peer
`omniroute-1`, 100.78.63.20), registered in Hermes as provider **`custom:omni-101`**
(`config.yaml`: custom provider `name: omni-101`, `base_url: http://100.78.63.20:20128`).
```
httpcore.ConnectError: [Errno 111] Connection refused
→ openai.APIConnectionError: Connection error.
→ RuntimeError: Connection error. (as recorded in executions.db)
```
`hermes-errors.log` carried **328 `Errno 111`** occurrences in the live file alone (504 across the
rotations during the window). Every failing run recorded the identical leaf string, so all 24 model-call
failures are **one fault**, not 24 different ones.
## Timeline (AWST)
| Time | Event |
|---|---|
| **10-14 08:30:27** | First failed run — Gmail Action Digest (`08eb86f02d37`). Isolated: 06:00 and 09:00–11:45 runs were clean. |
| **10-14 12:00** | Cascade — all three noon jobs fail (`b63ed5817212`, `ea79a2107e36`, `ecc1fc13655a`) and the watchdog reddens at 12:15/12:30. |
| **10-14 15:00 / 18:00 / 22:00** | Mosquito Tracker ×2, Embedding Coverage, Health Monitor all fail. |
| **10-14 23:59** | **The nightly Wiki Audit itself failed** — so the 10-14 window was never audited. |
| **10-15 03:00** | `wiki-stocktake-maintenance` (`1a8e90ea2271`) failed — no `freshness-20261015.txt` written. |
| **10-15 07:00–10:00** | Prime-time digests fail: Tech & AI, Daily Marketing Job Radar, Political News, AI Newsletter Curation for Patrick. |
| **10-15 12:00** | Noon jobs fail again. |
| **10-15 18:00:41** | **Last genuine failure** — Embedding Coverage Monitor. |
| **10-15 18:30:26** | Last watchdog red. |
| **10-15 22:01:56** | **Recovery marker** — Mosquito Tracker (`1b31187397a5`) completes on `omni-101`; streak 4 → 0. |
| **10-16 00:00–00:10** | Embedding Coverage (00:02:29), Health Monitor (00:09:48) and the Watchdog (00:01:36) all complete — three streaks cleared. |
| **10-16 00:1x** | **Live re-probe:** `GET /v1/models` on 100.78.63.20:20128 → **HTTP 401** in 10 ms (alive, auth required). Same on `omniroute:20128` / 100.88.81.19. |
## Blast radius
| Job | Name | Failures | First → last |
|---|---|---|---|
| `b63ed5817212` | Mnemosyne Embedding Coverage (6-hourly) | 6 | 10-14 12:00 → 10-15 18:00 |
| `1b31187397a5` | Mosquito Tracker (4×/day) | 4 | 10-14 15:00 → 10-15 15:00 |
| `ea79a2107e36` | Mnemosyne Health Monitor (12-hourly) | 3 | 10-14 12:00 → 10-15 12:00 |
| `08eb86f02d37` | Gmail Action Digest (daily 08:30) | 2 | 10-14 08:30 → 10-15 08:30 |
| `ecc1fc13655a` | FreshRSS Midday Briefing (daily 12:00) | 2 | 10-14 12:00 → 10-15 12:00 |
| `85edf6772bca` | Tech & AI Newsletter Digest (daily 07:00) | 1 | 10-15 07:00 |
| `dca8482e4f76` | Daily Marketing Job Radar (weekdays 07:00) | 1 | 10-15 07:00 |
| `0e519874d280` | Political News Digest (Mon/Thu 08:00) | 1 | 10-15 08:00 |
| `06a0a1c8704f` | AI Newsletter Curation for Patrick (Thu 10:00) | 1 | 10-15 10:00 |
| `c8d52904e1d6` | Perth New Restaurant (Thu 18:00) | 1 | 10-15 18:00 |
| `1a8e90ea2271` | wiki-stocktake-maintenance (daily 03:00) | 1 | 10-15 03:00 |
| `cbfe434a3b5e` | Wiki Audit (daily 23:59) | 1 | 10-14 23:59 |
| `96eb95ce12f2` | Mnemosyne Watchdog (every 15 min) | 12 | **symptom only** — see below |
Day tallies: **10-14 = 160 completed / 12 failed** (172), **10-15 = 151 completed / 24 failed / 1 running** (176).
**User-visible cost:** several digests simply never went out — Tech & AI, Daily Marketing Job Radar,
Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick, and the
weekly Perth New Restaurant (missed outright; next chance 10-22).
## What stayed green — and what that does *not* prove
**Script-only jobs kept running normally.** The fleet holds **10 `no_agent` jobs** (no LLM turn: Mnemosyne
Auto-Consolidate, DB Backup, Watchdog, Daily Birthday Checker, Job Tracker Dashboard Refresh, Hermes Config
Backup to Gitea, weekly-save-restricted-bot-resources, Evening Stand-Down Briefing, plus two disabled).
Completing runs inside the storm window by provider: **210 `no_agent` / no provider + 1 `no_agent` on
`custom:omni`** — that is the *entire* green of the fleet, and **zero model calls completed in the 34 hours.**
**Every model-calling job in the fleet is on the one node that failed.** Verified from `jobs.json`: **all 18
enabled agent-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2).
⚠ **Correction to a first pass at this window:** Evening Stand-Down Briefing (`e530c89f2be0`) carries
`provider: custom:omni` — but it is also **`no_agent: true`**, a script-only job whose provider/model fields
are vestigial. Its clean 18:45 runs on *both* 10-14 and 10-15 are therefore **not** evidence that CT600
stayed healthy; it never called a model. No cron job exercises `custom:omni` with an LLM turn
(`d40085da631c`, the only other `custom:omni` pin, is **disabled**), so the cron fleet **cannot** localise
this outage between the two OmniRoute nodes. What *is* reachable is the leaf error itself — `Connection
refused` to `100.78.63.20:20128` — which points at **`omniroute-1`**.
## The watchdog's reds are derivative
`96eb95ce12f2` (Mnemosyne Watchdog, `no_agent`, script-only) accounts for **12 of the 36 failure rows** — it
reddens when its log scan finds errors, so during a provider storm it is *reporting the weather, not the
fault*. All six of its incidents auto-resolved. Do not chase watchdog reds as a root cause; read
`executions.error` on the jobs that actually make model calls. It also reddens in **bursts** (12:15/12:30,
then 18:15/18:30), not continuously — its trigger is a log sentinel, so it under-reports the window rather
than bracketing it.
## Incident bookkeeping
Open incidents after the storm: **8 `alerted`** + **2 `detected`** + 2 pre-existing = **12 open**
(9 alerted / 3 detected). Each clears **only on the next successful run of its own job**, which for the
weekly and twice-weekly jobs means days of visible noise:
`08eb86f02d37` 10-16 08:30 · `dca8482e4f76` 10-16 07:00 · `85edf6772bca` 10-16 07:00 ·
`ecc1fc13655a` 10-16 12:00 · `1a8e90ea2271` 10-16 03:00 · `0e519874d280` 10-19 (Mon) ·
`06a0a1c8704f` 10-22 (Thu) · `c8d52904e1d6` 10-22 (Thu).
Pre-existing and **not** storm-caused: `1b23c042e17d` (alerted since 10-09, next slot 10-16 08:00) and
`0e753e_e7f8194a7d06` (FB Birthday Monthly, detected since 10-01, next slot 11-01).
## Lessons
1. **The entire scheduled estate rides one OmniRoute node.** All 18 enabled model-calling jobs are pinned to
`custom:omni-101`; the only other model-capable cron pin, `custom:omni`, sits on a `no_agent` job. Two
live OmniRoute nodes exist (`omniroute` CT600 at 100.88.81.19, `omniroute-1` at 100.78.63.20) with **no
failover between them**, so one node's availability gates the whole estate's output. This is the second
multi-day storm from the same topology → [[incidents/2026-10-09-cron-provider-timeout-storm]].
2. **`provider` on a `no_agent` job is decorative.** Never read a script-only job's green row as evidence of
provider health — check `no_agent` before concluding anything from a clean run.
3. **Both Mnemosyne monitors sit on the same node as the memory system they watch** — both failed, so
coverage figures went unrefreshed for the whole window (last explicit figures remain 2026-10-12).
Monitoring that shares a blast radius with its subject is degraded monitoring.
4. **A log-sentinel watchdog multiplies failure counts.** 36 failure rows were really 24, across 12 workers.
Tally per-job, not per-row, or the scale reads wrong.
5. **A failed nightly audit leaves a hole, not a gap in coverage.** The 10-14 audit died on this same storm,
so this page covers **two** windows (10-14 → 10-16).
## Related
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — the earlier storm; same topology, same failure class.
- [[systems/omniroute]] — node layout, providers and pins.
- [[systems/scheduled-tasks]] — per-window fleet tally and incident ledger.
- [[systems/mnemosyne]] — the two coverage monitors that went dark.