audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections

omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07
(httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error).
36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog
symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1).

STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable
pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls
completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence.

New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live).
Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute,
systems/mnemosyne (monitor blast-radius note), current-state, index, log.
This commit is contained in:
hermes
2026-10-16 00:19:14 +08:00
parent deb6a8eec2
commit 2061b8607b
7 changed files with 328 additions and 11 deletions
+21 -2
View File
@@ -3,8 +3,8 @@ title: OmniRoute
type: system
status: active
created: 2026-07-22
updated: 2026-10-11
verified_on: 2026-10-11
updated: 2026-10-16
verified_on: 2026-10-16
confidence: high
tags: [routing, provider, model, inference]
sources: [raw/configs/hermes-config-sanitized.txt]
@@ -66,6 +66,25 @@ The wiki previously recorded CT600 as retired and CT601 as the live successor. *
- Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
- Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
## Recent Changes (2026-10-16) — `omniroute-1` connection-refused storm
- **`omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) was TCP-unreachable from 2026-10-14
08:30 to 2026-10-15 ~18:07 AWST** — `httpcore.ConnectError: [Errno 111] Connection refused` →
`openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` Result: **24 genuine
cron model-call failures across 12 jobs** plus 12 derivative watchdog rows, and **every scheduled digest
for ~34 hours was silently suppressed** (including the 10-14 23:59 Wiki Audit).
- **This node is a single point of failure for the whole scheduled estate.** All **18** enabled
`agent`-mode cron jobs are pinned to `custom:omni-101` (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2). The only other model-capable cron pin — **`custom:omni` (CT600,
`omniroute:20128`)** — is attached to a **`no_agent`** job (`e530c89f2be0`, Evening Stand-Down), so it is
**never exercised by a model call**; `d40085da631c`, the only other `custom:omni` pin, is **disabled**.
The cron fleet therefore has **no model-call failover between the two nodes**, and no job that can even
tell you whether the second node is up.
- **Recovered — verified 2026-10-16 00:1x:** `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in
10 ms** (up, auth required); same on `omniroute:20128` / `100.88.81.19`. Four jobs have completed on the
node since 2026-10-15 22:01.
- Full write-up: [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
## Recent Changes (2026-08-14)
- **OmniRoute factory reset on CT601.** Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (`auto/best-fast`, `auto/best-coding`, etc.). Old state backed up at `/opt/omniroute/factory-reset-backup-20260814-225524`.
- **Providers re-added from Notion API vault:** Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry.