audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections
omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07 (httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error). 36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1). STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence. New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live). Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute, systems/mnemosyne (monitor blast-radius note), current-state, index, log.
This commit is contained in:
+21
-2
@@ -3,8 +3,8 @@ title: OmniRoute
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
updated: 2026-10-16
|
||||
verified_on: 2026-10-16
|
||||
confidence: high
|
||||
tags: [routing, provider, model, inference]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -66,6 +66,25 @@ The wiki previously recorded CT600 as retired and CT601 as the live successor. *
|
||||
- Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
|
||||
- Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
|
||||
|
||||
## Recent Changes (2026-10-16) — `omniroute-1` connection-refused storm
|
||||
|
||||
- **`omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) was TCP-unreachable from 2026-10-14
|
||||
08:30 to 2026-10-15 ~18:07 AWST** — `httpcore.ConnectError: [Errno 111] Connection refused` →
|
||||
`openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` Result: **24 genuine
|
||||
cron model-call failures across 12 jobs** plus 12 derivative watchdog rows, and **every scheduled digest
|
||||
for ~34 hours was silently suppressed** (including the 10-14 23:59 Wiki Audit).
|
||||
- **This node is a single point of failure for the whole scheduled estate.** All **18** enabled
|
||||
`agent`-mode cron jobs are pinned to `custom:omni-101` (`smart-route` ×10, `free-stack` ×6,
|
||||
`deepseek-paid-smart` ×2). The only other model-capable cron pin — **`custom:omni` (CT600,
|
||||
`omniroute:20128`)** — is attached to a **`no_agent`** job (`e530c89f2be0`, Evening Stand-Down), so it is
|
||||
**never exercised by a model call**; `d40085da631c`, the only other `custom:omni` pin, is **disabled**.
|
||||
The cron fleet therefore has **no model-call failover between the two nodes**, and no job that can even
|
||||
tell you whether the second node is up.
|
||||
- **Recovered — verified 2026-10-16 00:1x:** `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in
|
||||
10 ms** (up, auth required); same on `omniroute:20128` / `100.88.81.19`. Four jobs have completed on the
|
||||
node since 2026-10-15 22:01.
|
||||
- Full write-up: [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
|
||||
|
||||
## Recent Changes (2026-08-14)
|
||||
- **OmniRoute factory reset on CT601.** Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (`auto/best-fast`, `auto/best-coding`, etc.). Old state backed up at `/opt/omniroute/factory-reset-backup-20260814-225524`.
|
||||
- **Providers re-added from Notion API vault:** Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry.
|
||||
|
||||
Reference in New Issue
Block a user