audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections

omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07
(httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error).
36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog
symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1).

STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable
pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls
completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence.

New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live).
Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute,
systems/mnemosyne (monitor blast-radius note), current-state, index, log.
This commit is contained in:
hermes
2026-10-16 00:19:14 +08:00
parent deb6a8eec2
commit 2061b8607b
7 changed files with 328 additions and 11 deletions
+12 -2
View File
@@ -3,8 +3,8 @@ title: Mnemosyne
type: system
status: active
created: 2026-07-22
updated: 2026-10-13
verified_on: 2026-10-13
updated: 2026-10-16
verified_on: 2026-10-16
confidence: high
tags: [memory, mnemosyne, backend]
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
@@ -94,6 +94,16 @@ Primary agent memory store for Hermes.
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
- ⚠ **Blast-radius note (2026-10-16).** Both monitors above (`ea79a2107e36` Health every 12 h,
`b63ed5817212` Embedding Coverage every 6 h) are pinned to **`custom:omni-101`** — the same node the
memory system's own reporting depends on. During the **2026-10-14 → 10-15 `omniroute-1` outage both
failed on every slot**, so **no coverage figures were refreshed for ~34 hours**; the last explicit read
remains the **2026-10-12 12:02** run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %), and the
**2026-10-16 00:09:48** Health Monitor run is the first clean read since. The two script-only jobs
(`consolidate_mnemosyne.sh`, `mnemosyne-backup.sh`) were **unaffected** — they make no model call.
Monitoring that shares a blast radius with its subject is degraded monitoring; a coverage alert that
cannot fire during a router outage is exactly the case this has to cover.
→ [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
## History and Evidence
- **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
+21 -2
View File
@@ -3,8 +3,8 @@ title: OmniRoute
type: system
status: active
created: 2026-07-22
updated: 2026-10-11
verified_on: 2026-10-11
updated: 2026-10-16
verified_on: 2026-10-16
confidence: high
tags: [routing, provider, model, inference]
sources: [raw/configs/hermes-config-sanitized.txt]
@@ -66,6 +66,25 @@ The wiki previously recorded CT600 as retired and CT601 as the live successor. *
- Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
- Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
## Recent Changes (2026-10-16) — `omniroute-1` connection-refused storm
- **`omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) was TCP-unreachable from 2026-10-14
08:30 to 2026-10-15 ~18:07 AWST** — `httpcore.ConnectError: [Errno 111] Connection refused` →
`openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` Result: **24 genuine
cron model-call failures across 12 jobs** plus 12 derivative watchdog rows, and **every scheduled digest
for ~34 hours was silently suppressed** (including the 10-14 23:59 Wiki Audit).
- **This node is a single point of failure for the whole scheduled estate.** All **18** enabled
`agent`-mode cron jobs are pinned to `custom:omni-101` (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2). The only other model-capable cron pin — **`custom:omni` (CT600,
`omniroute:20128`)** — is attached to a **`no_agent`** job (`e530c89f2be0`, Evening Stand-Down), so it is
**never exercised by a model call**; `d40085da631c`, the only other `custom:omni` pin, is **disabled**.
The cron fleet therefore has **no model-call failover between the two nodes**, and no job that can even
tell you whether the second node is up.
- **Recovered — verified 2026-10-16 00:1x:** `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in
10 ms** (up, auth required); same on `omniroute:20128` / `100.88.81.19`. Four jobs have completed on the
node since 2026-10-15 22:01.
- Full write-up: [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
## Recent Changes (2026-08-14)
- **OmniRoute factory reset on CT601.** Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (`auto/best-fast`, `auto/best-coding`, etc.). Old state backed up at `/opt/omniroute/factory-reset-backup-20260814-225524`.
- **Providers re-added from Notion API vault:** Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry.
+72 -2
View File
@@ -3,8 +3,8 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-14
verified_on: 2026-10-14
updated: 2026-10-16
verified_on: 2026-10-16
review_after: 2026-10-25
confidence: high
tags: [cron, scheduled, automation]
@@ -14,6 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks
## Count
- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
- **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, re-verified 2026-10-13 — unchanged; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10)
- **Non-clean `last_status`:** **3** (was 5 at the 2026-10-12 audit) — `0e753e9720fd` (`error`, streak 0), `1b23c042e17d` (`error`, streak 1), `066bb5191280` (`delivery_failed`, streak 0). `fad5a86979d4` and `dca8482e4f76` cleared on **2026-10-12** when both ran clean on schedule.
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]].
@@ -406,3 +407,72 @@ or repair OmniRoute's `opencode` provider credentials); GitHub Trending Radar to
Mon 09:00); Camofox containment vs. multi-tab workloads (2026-10-13); Quartz rebuild step inert;
skills-index off-by-one; **4** non-clean `last_status` rows (above); Mattermost platform unreachable;
gateway runtime env outside the 15-minute dependency guard.
## Re-verified 2026-10-15 / 2026-10-16 — the `omni-101` connection-refused storm
Window: **2026-10-14 00:00 → 2026-10-16 00:00 AWST** (two days — the 2026-10-14 23:59 audit died inside the
same outage, so the 10-14 window was never covered). Live probes 2026-10-16 00:0x–00:2x.
Fleet **29 jobs (26 enabled / 3 disabled)** — unchanged. **Open incidents 3 → 12** (9 `alerted` / 3 `detected`).
**The tally broke hard.** 2026-10-14 = **160 completed / 12 failed**; 2026-10-15 = **151 completed /
24 failed** / 1 running. Against a four-day clean run (10-10 `175/0`, 10-11 `172/0`, 10-12 `175/0`,
10-13 `167/1`). **36 failure rows across 13 jobs** — but only **24 are genuine model-call failures** on
**12 jobs**; the other **12 rows are the Mnemosyne Watchdog reddening on a log sentinel** (symptom, not
fault). Day-level, 10-15 is the worst cron day since 2026-10-09 (15).
**Root cause.** `httpcore.ConnectError: [Errno 111] Connection refused` against
`http://100.78.63.20:20128` — the **`omniroute-1`** node, provider **`custom:omni-101`**
(`config.yaml`: custom provider `name: omni-101`). Surfaced as `openai.APIConnectionError: Connection error.`
→ `RuntimeError: Connection error.` 328 `Errno 111` lines in the live `errors.log`, 504 across rotations.
Window 10-14 08:30:27 → 10-15 18:07, recovered by 10-15 22:01. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
**Structural finding — no model-call failover anywhere in the fleet.** Verified from `jobs.json`: **all 18
enabled agent-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2). The only other model-capable cron pin, `custom:omni` (CT600, `omniroute:20128`),
sits on `e530c89f2be0` — which is **`no_agent: true`**, a script-only job, so its provider field is
vestigial. Completing runs inside the window by provider: **210 `no_agent` / no provider + 1 `no_agent` on
`custom:omni`** — that is the entire green of the fleet, and **zero model calls completed in 34 hours**.
⚠ A first pass at this window read Evening Stand-Down's clean 18:45 runs as proof CT600 stayed up. It is
not: Stand-Down never calls a model. **Never read a `no_agent` job's green row as provider-health evidence.**
**Worst-affected jobs:** `b63ed5817212` Mnemosyne Embedding Coverage ×6 (every slot for 30 h),
`1b31187397a5` Mosquito Tracker ×4, `ea79a2107e36` Mnemosyne Health Monitor ×3, `08eb86f02d37` Gmail Action
Digest ×2, `ecc1fc13655a` FreshRSS Midday ×2, then one apiece for `85edf6772bca`, `dca8482e4f76`,
`0e519874d280`, `06a0a1c8704f`, `c8d52904e1d6`, `1a8e90ea2271`, `cbfe434a3b5e`. Full table in the incident
page. **User-visible cost:** Digests that never went out — Tech & AI, Daily Marketing Job Radar, Political
News, AI Newsletter Curation for Patrick, Gmail Action Digest ×2, FreshRSS Midday ×2, and the weekly Perth
New Restaurant (missed outright; next chance 10-22). The missed audit also meant the 10-14 window went
uncovered.
**Recovery verified by schedule, not by probe alone.** `1b31187397a5` completed 10-15 **22:01:56** (streak
4 → 0); `b63ed5817212` 10-16 **00:02:29**; `96eb95ce12f2` 10-16 **00:01:36**; `ea79a2107e36` 10-16
**00:09:48**. Live 10-16 00:1x: `GET /v1/models` on **100.78.63.20:20128 → HTTP 401 in 10 ms** (up, auth
required) and the same on `omniroute:20128` / 100.88.81.19.
**Non-clean `last_status`: 3 → 12.** Cleared during the window: `b63ed5817212`, `1b31187397a5`,
`ea79a2107e36` (all on successful post-storm runs). Added: the nine rows listed under **Count** above.
Unchanged long-standing: `1b23c042e17d` (10-09 error), `0e753e9720fd` (10-01 error), `066bb5191280`
(10-01 `delivery_failed`, dead Telegram `thread_id 34795`).
**Mnemosyne.** Both monitors sat on the failing node, so **no coverage figures were refreshed in the
window** — last explicit numbers remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic
1050/1056 = 99.4 %); the 10-16 00:09 Health Monitor run is the first clean read since 10-12.
→ [[systems/mnemosyne]]
**Stocktake gap (self-healing).** `1a8e90ea2271` failed 10-15 03:00 → `raw/stocktake/freshness-20261015.txt`
was never written and `systems/skills-index.md` still carries the 10-14 stamp. Its next slot is 10-16 03:00,
so the gap closes on its own — **not** regenerated by hand here, to avoid racing the scheduled run.
**Sessions.** **36 sessions in the two-day window, all `cron`, zero interactive** — fourth consecutive
window with no interactive session.
**Pages touched:** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` (new),
`systems/omniroute.md`, `systems/mnemosyne.md`, `systems/scheduled-tasks.md` (this block, frontmatter,
Count), `current-state.md`, `index.md`, `log.md`.
**Nothing removed or archived:** no retired job, no obsolete system → `index.md` structure gains one page.
**Still open after this cycle:** the **no-failover topology** itself (all 18 model-calling jobs on one
OmniRoute node — the durable fix is a second provider pin or a router-side failover, and it needs Anthony's
call); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab
workloads; Quartz rebuild step inert; skills-index off-by-one; **12** non-clean `last_status` rows;
Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.