audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections

omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07
(httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error).
36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog
symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1).

STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable
pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls
completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence.

New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live).
Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute,
systems/mnemosyne (monitor blast-radius note), current-state, index, log.
This commit is contained in:
hermes
2026-10-16 00:19:14 +08:00
parent deb6a8eec2
commit 2061b8607b
7 changed files with 328 additions and 11 deletions
+65
View File
@@ -715,3 +715,68 @@ single-host in framing: [[systems/omniroute]], [[infrastructure/hosts]], [[syste
OmniRoute topology needs a dedicated pass before anyone relies on the current four-node story.
No secrets written; host names, IPs, VMIDs and job/incident IDs only.
---
## [2026-10-16] audit | nightly review of 2026-10-14 → 2026-10-15 (two days — the 10-14 audit died inside the same outage) — a 34-hour fleet blackout from one OmniRoute node
Ran 2026-10-16 00:0x–00:3x AWST. **Scope is two windows, not one:** the 2026-10-14 23:59 Wiki Audit
(`cbfe434a3b5e`) failed on the fault below, so the 10-14 window was never covered.
**Finding: fleet-wide cron blackout, 2026-10-14 08:30:27 → 2026-10-15 18:07 AWST (~34 h).**
- **Root cause.** `httpcore.ConnectError: [Errno 111] Connection refused` against
`http://100.78.63.20:20128` — the **`omniroute-1`** node, provider **`custom:omni-101`** (confirmed in
`config.yaml`: custom provider `name: omni-101`, `base_url: http://100.78.63.20:20128`). It surfaced as
`openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` — the exact string
recorded on every failing execution row. 328 `Errno 111` lines in the live `errors.log`, 504 across the
rotations.
- **Impact.** 10-14 = **160 completed / 12 failed**; 10-15 = **151 completed / 24 failed** (+1 in flight).
Against 10-12 `175/0`, 10-13 `167/1`. **36 failure rows across 13 jobs**, but only **24 are genuine
model-call failures** on **12 jobs** — the other 12 rows are the Mnemosyne Watchdog reddening on a log
sentinel (symptom, not fault). 10-15 is the worst cron day since 2026-10-09 (15).
- **The finding that matters most — there is no failover.** Verified from live `jobs.json`: **all 18
enabled `agent`-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2). The only other model-capable pin, `custom:omni` (CT600, `omniroute:20128`),
sits on `e530c89f2be0` — which is **`no_agent: true`**; `d40085da631c` is disabled. Completing runs inside
the window by provider: **210 `no_agent` / no provider + 1 `no_agent` on `custom:omni`**. So **zero model
calls completed in 34 hours**, and the fleet has no job that could even report the second node's health.
- **Correction made during this audit.** A first pass read Evening Stand-Down Briefing's clean 18:45 runs
(both days) as evidence CT600 stayed healthy. Wrong: it is `no_agent` and never calls a model. Recorded as
a standing rule — *never read a `no_agent` job's green row as provider-health evidence* — in the incident
page and in [[systems/scheduled-tasks]].
- **User-visible cost: silence.** Digests never delivered: Tech & AI, Daily Marketing Job Radar, Political
News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick. The **weekly Perth
New Restaurant was missed outright** (next slot 10-22).
- **Recovery, verified live (not assumed).** `1b31187397a5` completed **10-15 22:01:56** (streak 4 → 0);
`96eb95ce12f2` **10-16 00:01:36**; `b63ed5817212` **10-16 00:02:29**; `ea79a2107e36` **10-16 00:09:48**.
Probes 2026-10-16 00:1x: `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in 10 ms** (up, auth
required); same on `omniroute:20128` / `100.88.81.19`.
- **Fleet bookkeeping.** **Non-clean `last_status`: 3 → 12** (9 new: `08eb86f02d37`, `ecc1fc13655a`,
`dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`,
`cbfe434a3b5e`; three cleared in the window: `b63ed5817212`, `1b31187397a5`, `ea79a2107e36`).
**Open incidents 3 → 12** (9 `alerted` / 3 `detected`). Long-standing and pre-storm: `1b23c042e17d`
(Fri 08:00), `0e753e9720fd` (1st 09:00), `066bb5191280` (`delivery_failed`, dead Telegram thread 34795).
- **Mnemosyne.** Both monitors sit on the failing node, so **no coverage figures were refreshed for ~34 h**;
last explicit read remains 2026-10-12 12:02 (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %).
Script-only Mnemosyne jobs were unaffected.
- **Self-healing gap left alone deliberately.** The 10-15 03:00 stocktake failure left
`raw/stocktake/freshness-20261015.txt` unwritten and `systems/skills-index.md` stamped 10-14. Its next
slot (10-16 03:00) closes the gap, so nothing was regenerated by hand — regenerating would race the
scheduled run.
- **Sessions.** 36 sessions across the two days, **all `cron`, zero interactive** — fourth consecutive
window with no interactive session.
**Pages touched:** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` (**new**, 9,303 B);
`systems/scheduled-tasks.md` (frontmatter, **Count** current-state bullet, new Re-verified block);
`systems/omniroute.md` (frontmatter + new Recent Changes section); `systems/mnemosyne.md` (frontmatter +
blast-radius note on the two monitors); `current-state.md` (frontmatter + new top section, `Updated:` line);
`index.md` (`Last updated`, new incidents entry, scheduled-tasks bullet); `log.md` (this entry).
**Nothing removed or archived** — no retired job, no obsolete system, so `index.md` structure gains one page.
**Durable fix still open (needs Anthony's call):** the single-node topology. All 18 model-calling jobs on
one OmniRoute node means this recurs on the next `omniroute-1` blip — as it did on 2026-10-09. Options are
a second provider pin on a subset of jobs, or router-side failover. **Not applied** — that is a routing
decision, not an audit one.
No secrets written; host names, IPs, VMIDs and job/incident IDs only.