audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections
omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07 (httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error). 36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1). STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence. New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live). Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute, systems/mnemosyne (monitor blast-radius note), current-state, index, log.
This commit is contained in:
+14
-3
@@ -3,8 +3,8 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-14
|
||||
verified_on: 2026-10-14
|
||||
updated: 2026-10-16
|
||||
verified_on: 2026-10-16
|
||||
review_after: 2026-10-25
|
||||
confidence: high
|
||||
tags: [system]
|
||||
@@ -12,7 +12,18 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-14
|
||||
Updated: 2026-10-16
|
||||
|
||||
## Recent Changes (2026-10-16, nightly audit; live probes 2026-10-16 00:0x–00:2x AWST)
|
||||
|
||||
- **⚠ A fleet-wide 34-hour cron blackout just ended — `omniroute-1` was refusing TCP connections.** `10-14 = 160 completed / 12 failed` and `10-15 = 151 completed / 24 failed`, against a clean 10-10→10-12 (`175/0`, `172/0`, `175/0`) and 10-13 (`167/1`). Root cause: `httpcore.ConnectError: [Errno 111] Connection refused` against `http://100.78.63.20:20128` (provider `custom:omni-101`), surfacing in the cron DB as `openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` Window **10-14 08:30:27 → 10-15 18:07**, recovered by **10-15 22:01**. **24 genuine model-call failures on 12 jobs** + 12 derivative watchdog reds = 36 failure rows across 13 jobs. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
|
||||
- **Everything model-calling is pinned to that one node — there is no failover anywhere in the fleet.** All **18** enabled `agent`-mode cron jobs run on `custom:omni-101` (`smart-route` ×10, `free-stack` ×6, `deepseek-paid-smart` ×2). The only other model-capable pin, `custom:omni` (CT600 `omniroute:20128`), carries just the **`no_agent`** job `e530c89f2be0`, whose provider field is vestigial; `d40085da631c` is disabled. Completing runs inside the window: **211 — every single one script-only.** **Zero model calls succeeded for 34 hours.** → [[systems/omniroute]], [[systems/scheduled-tasks]]
|
||||
- **The cost was silence, not errors.** Digests that never went out: Tech & AI, Daily Marketing Job Radar, Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick; the weekly Perth New Restaurant was **missed outright** (next chance 10-22). The **10-14 23:59 Wiki Audit itself died on the same fault**, so the 10-14 window was never audited — this entry spans two days. → [[systems/scheduled-tasks]]
|
||||
- **Recovered, verified live.** `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in 10 ms** (up, auth required); same on `omniroute:20128` / `100.88.81.19`. First green model-call job: `1b31187397a5` at **10-15 22:01:56**; then `b63ed5817212` (**10-16 00:02:29**) and `ea79a2107e36` (**10-16 00:09:48**).
|
||||
- **The backlog unwinds on schedule rather than by hand.** **Non-clean `last_status`: 3 → 12** (9 new storm rows; three cleared as their slots came round), **open incidents 3 → 12** (9 `alerted` / 3 `detected`). The two longest-lived — `1b23c042e17d` (Fri 08:00 slot) and `0e753e9720fd` (1st 09:00) — **pre-date the storm**. → [[systems/scheduled-tasks]]
|
||||
- **Both Mnemosyne monitors sit inside the blast radius.** Health (12 h) and Embedding Coverage (6 h) both failed on every slot, so **no coverage figures were refreshed for ~34 h** — the last explicit read remains 2026-10-12 (working 96.7 %, episodic 99.4 %). The script-only Mnemosyne jobs were unaffected. → [[systems/mnemosyne]]
|
||||
- **Self-healing gap, not a manual repair.** The failed 10-15 03:00 stocktake left `raw/stocktake/freshness-20261015.txt` unwritten and `systems/skills-index.md` stamped 10-14; the 10-16 03:00 slot closes it, so nothing was regenerated by hand here.
|
||||
- **Fourth consecutive window with no interactive session** — **36 sessions** across the two days, all `cron`.
|
||||
|
||||
## Recent Changes (2026-10-14, 03:00 stocktake; live probes 03:01–03:1x AWST)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user