audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections
omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07 (httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error). 36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1). STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence. New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live). Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute, systems/mnemosyne (monitor blast-radius note), current-state, index, log.
This commit is contained in:
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-14
|
||||
verified_on: 2026-10-14
|
||||
updated: 2026-10-16
|
||||
verified_on: 2026-10-16
|
||||
review_after: 2026-10-25
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
@@ -14,6 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
|
||||
- **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, re-verified 2026-10-13 — unchanged; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10)
|
||||
- **Non-clean `last_status`:** **3** (was 5 at the 2026-10-12 audit) — `0e753e9720fd` (`error`, streak 0), `1b23c042e17d` (`error`, streak 1), `066bb5191280` (`delivery_failed`, streak 0). `fad5a86979d4` and `dca8482e4f76` cleared on **2026-10-12** when both ran clean on schedule.
|
||||
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]].
|
||||
@@ -406,3 +407,72 @@ or repair OmniRoute's `opencode` provider credentials); GitHub Trending Radar to
|
||||
Mon 09:00); Camofox containment vs. multi-tab workloads (2026-10-13); Quartz rebuild step inert;
|
||||
skills-index off-by-one; **4** non-clean `last_status` rows (above); Mattermost platform unreachable;
|
||||
gateway runtime env outside the 15-minute dependency guard.
|
||||
|
||||
## Re-verified 2026-10-15 / 2026-10-16 — the `omni-101` connection-refused storm
|
||||
|
||||
Window: **2026-10-14 00:00 → 2026-10-16 00:00 AWST** (two days — the 2026-10-14 23:59 audit died inside the
|
||||
same outage, so the 10-14 window was never covered). Live probes 2026-10-16 00:0x–00:2x.
|
||||
Fleet **29 jobs (26 enabled / 3 disabled)** — unchanged. **Open incidents 3 → 12** (9 `alerted` / 3 `detected`).
|
||||
|
||||
**The tally broke hard.** 2026-10-14 = **160 completed / 12 failed**; 2026-10-15 = **151 completed /
|
||||
24 failed** / 1 running. Against a four-day clean run (10-10 `175/0`, 10-11 `172/0`, 10-12 `175/0`,
|
||||
10-13 `167/1`). **36 failure rows across 13 jobs** — but only **24 are genuine model-call failures** on
|
||||
**12 jobs**; the other **12 rows are the Mnemosyne Watchdog reddening on a log sentinel** (symptom, not
|
||||
fault). Day-level, 10-15 is the worst cron day since 2026-10-09 (15).
|
||||
|
||||
**Root cause.** `httpcore.ConnectError: [Errno 111] Connection refused` against
|
||||
`http://100.78.63.20:20128` — the **`omniroute-1`** node, provider **`custom:omni-101`**
|
||||
(`config.yaml`: custom provider `name: omni-101`). Surfaced as `openai.APIConnectionError: Connection error.`
|
||||
→ `RuntimeError: Connection error.` 328 `Errno 111` lines in the live `errors.log`, 504 across rotations.
|
||||
Window 10-14 08:30:27 → 10-15 18:07, recovered by 10-15 22:01. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
|
||||
|
||||
**Structural finding — no model-call failover anywhere in the fleet.** Verified from `jobs.json`: **all 18
|
||||
enabled agent-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6,
|
||||
`deepseek-paid-smart` ×2). The only other model-capable cron pin, `custom:omni` (CT600, `omniroute:20128`),
|
||||
sits on `e530c89f2be0` — which is **`no_agent: true`**, a script-only job, so its provider field is
|
||||
vestigial. Completing runs inside the window by provider: **210 `no_agent` / no provider + 1 `no_agent` on
|
||||
`custom:omni`** — that is the entire green of the fleet, and **zero model calls completed in 34 hours**.
|
||||
⚠ A first pass at this window read Evening Stand-Down's clean 18:45 runs as proof CT600 stayed up. It is
|
||||
not: Stand-Down never calls a model. **Never read a `no_agent` job's green row as provider-health evidence.**
|
||||
|
||||
**Worst-affected jobs:** `b63ed5817212` Mnemosyne Embedding Coverage ×6 (every slot for 30 h),
|
||||
`1b31187397a5` Mosquito Tracker ×4, `ea79a2107e36` Mnemosyne Health Monitor ×3, `08eb86f02d37` Gmail Action
|
||||
Digest ×2, `ecc1fc13655a` FreshRSS Midday ×2, then one apiece for `85edf6772bca`, `dca8482e4f76`,
|
||||
`0e519874d280`, `06a0a1c8704f`, `c8d52904e1d6`, `1a8e90ea2271`, `cbfe434a3b5e`. Full table in the incident
|
||||
page. **User-visible cost:** Digests that never went out — Tech & AI, Daily Marketing Job Radar, Political
|
||||
News, AI Newsletter Curation for Patrick, Gmail Action Digest ×2, FreshRSS Midday ×2, and the weekly Perth
|
||||
New Restaurant (missed outright; next chance 10-22). The missed audit also meant the 10-14 window went
|
||||
uncovered.
|
||||
|
||||
**Recovery verified by schedule, not by probe alone.** `1b31187397a5` completed 10-15 **22:01:56** (streak
|
||||
4 → 0); `b63ed5817212` 10-16 **00:02:29**; `96eb95ce12f2` 10-16 **00:01:36**; `ea79a2107e36` 10-16
|
||||
**00:09:48**. Live 10-16 00:1x: `GET /v1/models` on **100.78.63.20:20128 → HTTP 401 in 10 ms** (up, auth
|
||||
required) and the same on `omniroute:20128` / 100.88.81.19.
|
||||
|
||||
**Non-clean `last_status`: 3 → 12.** Cleared during the window: `b63ed5817212`, `1b31187397a5`,
|
||||
`ea79a2107e36` (all on successful post-storm runs). Added: the nine rows listed under **Count** above.
|
||||
Unchanged long-standing: `1b23c042e17d` (10-09 error), `0e753e9720fd` (10-01 error), `066bb5191280`
|
||||
(10-01 `delivery_failed`, dead Telegram `thread_id 34795`).
|
||||
|
||||
**Mnemosyne.** Both monitors sat on the failing node, so **no coverage figures were refreshed in the
|
||||
window** — last explicit numbers remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic
|
||||
1050/1056 = 99.4 %); the 10-16 00:09 Health Monitor run is the first clean read since 10-12.
|
||||
→ [[systems/mnemosyne]]
|
||||
|
||||
**Stocktake gap (self-healing).** `1a8e90ea2271` failed 10-15 03:00 → `raw/stocktake/freshness-20261015.txt`
|
||||
was never written and `systems/skills-index.md` still carries the 10-14 stamp. Its next slot is 10-16 03:00,
|
||||
so the gap closes on its own — **not** regenerated by hand here, to avoid racing the scheduled run.
|
||||
|
||||
**Sessions.** **36 sessions in the two-day window, all `cron`, zero interactive** — fourth consecutive
|
||||
window with no interactive session.
|
||||
|
||||
**Pages touched:** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` (new),
|
||||
`systems/omniroute.md`, `systems/mnemosyne.md`, `systems/scheduled-tasks.md` (this block, frontmatter,
|
||||
Count), `current-state.md`, `index.md`, `log.md`.
|
||||
**Nothing removed or archived:** no retired job, no obsolete system → `index.md` structure gains one page.
|
||||
|
||||
**Still open after this cycle:** the **no-failover topology** itself (all 18 model-calling jobs on one
|
||||
OmniRoute node — the durable fix is a second provider pin or a router-side failover, and it needs Anthony's
|
||||
call); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab
|
||||
workloads; Quartz rebuild step inert; skills-index off-by-one; **12** non-clean `last_status` rows;
|
||||
Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.
|
||||
|
||||
Reference in New Issue
Block a user