audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections

omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07
(httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error).
36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog
symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1).

STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable
pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls
completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence.

New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live).
Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute,
systems/mnemosyne (monitor blast-radius note), current-state, index, log.
This commit is contained in:
hermes
2026-10-16 00:19:14 +08:00
parent deb6a8eec2
commit 2061b8607b
7 changed files with 328 additions and 11 deletions
+14 -3
View File
@@ -3,8 +3,8 @@ title: Current State
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-14 updated: 2026-10-16
verified_on: 2026-10-14 verified_on: 2026-10-16
review_after: 2026-10-25 review_after: 2026-10-25
confidence: high confidence: high
tags: [system] tags: [system]
@@ -12,7 +12,18 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
--- ---
# Current State # Current State
Updated: 2026-10-14 Updated: 2026-10-16
## Recent Changes (2026-10-16, nightly audit; live probes 2026-10-16 00:0x–00:2x AWST)
- **⚠ A fleet-wide 34-hour cron blackout just ended — `omniroute-1` was refusing TCP connections.** `10-14 = 160 completed / 12 failed` and `10-15 = 151 completed / 24 failed`, against a clean 10-10→10-12 (`175/0`, `172/0`, `175/0`) and 10-13 (`167/1`). Root cause: `httpcore.ConnectError: [Errno 111] Connection refused` against `http://100.78.63.20:20128` (provider `custom:omni-101`), surfacing in the cron DB as `openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` Window **10-14 08:30:27 → 10-15 18:07**, recovered by **10-15 22:01**. **24 genuine model-call failures on 12 jobs** + 12 derivative watchdog reds = 36 failure rows across 13 jobs. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
- **Everything model-calling is pinned to that one node — there is no failover anywhere in the fleet.** All **18** enabled `agent`-mode cron jobs run on `custom:omni-101` (`smart-route` ×10, `free-stack` ×6, `deepseek-paid-smart` ×2). The only other model-capable pin, `custom:omni` (CT600 `omniroute:20128`), carries just the **`no_agent`** job `e530c89f2be0`, whose provider field is vestigial; `d40085da631c` is disabled. Completing runs inside the window: **211 — every single one script-only.** **Zero model calls succeeded for 34 hours.** → [[systems/omniroute]], [[systems/scheduled-tasks]]
- **The cost was silence, not errors.** Digests that never went out: Tech & AI, Daily Marketing Job Radar, Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick; the weekly Perth New Restaurant was **missed outright** (next chance 10-22). The **10-14 23:59 Wiki Audit itself died on the same fault**, so the 10-14 window was never audited — this entry spans two days. → [[systems/scheduled-tasks]]
- **Recovered, verified live.** `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in 10 ms** (up, auth required); same on `omniroute:20128` / `100.88.81.19`. First green model-call job: `1b31187397a5` at **10-15 22:01:56**; then `b63ed5817212` (**10-16 00:02:29**) and `ea79a2107e36` (**10-16 00:09:48**).
- **The backlog unwinds on schedule rather than by hand.** **Non-clean `last_status`: 3 → 12** (9 new storm rows; three cleared as their slots came round), **open incidents 3 → 12** (9 `alerted` / 3 `detected`). The two longest-lived — `1b23c042e17d` (Fri 08:00 slot) and `0e753e9720fd` (1st 09:00) — **pre-date the storm**. → [[systems/scheduled-tasks]]
- **Both Mnemosyne monitors sit inside the blast radius.** Health (12 h) and Embedding Coverage (6 h) both failed on every slot, so **no coverage figures were refreshed for ~34 h** — the last explicit read remains 2026-10-12 (working 96.7 %, episodic 99.4 %). The script-only Mnemosyne jobs were unaffected. → [[systems/mnemosyne]]
- **Self-healing gap, not a manual repair.** The failed 10-15 03:00 stocktake left `raw/stocktake/freshness-20261015.txt` unwritten and `systems/skills-index.md` stamped 10-14; the 10-16 03:00 slot closes it, so nothing was regenerated by hand here.
- **Fourth consecutive window with no interactive session** — **36 sessions** across the two days, all `cron`.
## Recent Changes (2026-10-14, 03:00 stocktake; live probes 03:01–03:1x AWST) ## Recent Changes (2026-10-14, 03:00 stocktake; live probes 03:01–03:1x AWST)
@@ -0,0 +1,141 @@
---
title: OmniRoute `omni-101` Node Outage — Connection-Refused Storm
type: concept
status: resolved
created: 2026-10-16
updated: 2026-10-16
verified_on: 2026-10-16
review_after: 2026-10-23
confidence: high
tags: [cron, router, provider, infrastructure]
sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, hermes-errors-log, live-probe]
---
# OmniRoute `omni-101` Node Outage — Connection-Refused Storm (2026-10-14 → 2026-10-15)
**Window:** 2026-10-14 08:30 → 2026-10-15 18:07 AWST (**~34 hours**)
**Status:** **Resolved.** `omniroute-1` answers again; four scheduled jobs have since completed on it.
**Impact:** **36 failed execution rows across 13 jobs** — 24 genuine model-call failures on 12 jobs, plus
12 watchdog *symptom* reds. **Every model call attempted in the window failed** (see below). For scale:
10-12 recorded `0 failed` and 10-13 recorded `1`.
**Severity:** the largest failure count since the 2026-10-09 storm (15 in one day); this one ran **two days**
and silently suppressed most of the day's digests.
## Root cause
`TCP connect refused` on **`http://100.78.63.20:20128`** — the **`omniroute-1`** node (Tailscale peer
`omniroute-1`, 100.78.63.20), registered in Hermes as provider **`custom:omni-101`**
(`config.yaml`: custom provider `name: omni-101`, `base_url: http://100.78.63.20:20128`).
```
httpcore.ConnectError: [Errno 111] Connection refused
→ openai.APIConnectionError: Connection error.
→ RuntimeError: Connection error. (as recorded in executions.db)
```
`hermes-errors.log` carried **328 `Errno 111`** occurrences in the live file alone (504 across the
rotations during the window). Every failing run recorded the identical leaf string, so all 24 model-call
failures are **one fault**, not 24 different ones.
## Timeline (AWST)
| Time | Event |
|---|---|
| **10-14 08:30:27** | First failed run — Gmail Action Digest (`08eb86f02d37`). Isolated: 06:00 and 09:00–11:45 runs were clean. |
| **10-14 12:00** | Cascade — all three noon jobs fail (`b63ed5817212`, `ea79a2107e36`, `ecc1fc13655a`) and the watchdog reddens at 12:15/12:30. |
| **10-14 15:00 / 18:00 / 22:00** | Mosquito Tracker ×2, Embedding Coverage, Health Monitor all fail. |
| **10-14 23:59** | **The nightly Wiki Audit itself failed** — so the 10-14 window was never audited. |
| **10-15 03:00** | `wiki-stocktake-maintenance` (`1a8e90ea2271`) failed — no `freshness-20261015.txt` written. |
| **10-15 07:00–10:00** | Prime-time digests fail: Tech & AI, Daily Marketing Job Radar, Political News, AI Newsletter Curation for Patrick. |
| **10-15 12:00** | Noon jobs fail again. |
| **10-15 18:00:41** | **Last genuine failure** — Embedding Coverage Monitor. |
| **10-15 18:30:26** | Last watchdog red. |
| **10-15 22:01:56** | **Recovery marker** — Mosquito Tracker (`1b31187397a5`) completes on `omni-101`; streak 4 → 0. |
| **10-16 00:00–00:10** | Embedding Coverage (00:02:29), Health Monitor (00:09:48) and the Watchdog (00:01:36) all complete — three streaks cleared. |
| **10-16 00:1x** | **Live re-probe:** `GET /v1/models` on 100.78.63.20:20128 → **HTTP 401** in 10 ms (alive, auth required). Same on `omniroute:20128` / 100.88.81.19. |
## Blast radius
| Job | Name | Failures | First → last |
|---|---|---|---|
| `b63ed5817212` | Mnemosyne Embedding Coverage (6-hourly) | 6 | 10-14 12:00 → 10-15 18:00 |
| `1b31187397a5` | Mosquito Tracker (4×/day) | 4 | 10-14 15:00 → 10-15 15:00 |
| `ea79a2107e36` | Mnemosyne Health Monitor (12-hourly) | 3 | 10-14 12:00 → 10-15 12:00 |
| `08eb86f02d37` | Gmail Action Digest (daily 08:30) | 2 | 10-14 08:30 → 10-15 08:30 |
| `ecc1fc13655a` | FreshRSS Midday Briefing (daily 12:00) | 2 | 10-14 12:00 → 10-15 12:00 |
| `85edf6772bca` | Tech & AI Newsletter Digest (daily 07:00) | 1 | 10-15 07:00 |
| `dca8482e4f76` | Daily Marketing Job Radar (weekdays 07:00) | 1 | 10-15 07:00 |
| `0e519874d280` | Political News Digest (Mon/Thu 08:00) | 1 | 10-15 08:00 |
| `06a0a1c8704f` | AI Newsletter Curation for Patrick (Thu 10:00) | 1 | 10-15 10:00 |
| `c8d52904e1d6` | Perth New Restaurant (Thu 18:00) | 1 | 10-15 18:00 |
| `1a8e90ea2271` | wiki-stocktake-maintenance (daily 03:00) | 1 | 10-15 03:00 |
| `cbfe434a3b5e` | Wiki Audit (daily 23:59) | 1 | 10-14 23:59 |
| `96eb95ce12f2` | Mnemosyne Watchdog (every 15 min) | 12 | **symptom only** — see below |
Day tallies: **10-14 = 160 completed / 12 failed** (172), **10-15 = 151 completed / 24 failed / 1 running** (176).
**User-visible cost:** several digests simply never went out — Tech & AI, Daily Marketing Job Radar,
Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick, and the
weekly Perth New Restaurant (missed outright; next chance 10-22).
## What stayed green — and what that does *not* prove
**Script-only jobs kept running normally.** The fleet holds **10 `no_agent` jobs** (no LLM turn: Mnemosyne
Auto-Consolidate, DB Backup, Watchdog, Daily Birthday Checker, Job Tracker Dashboard Refresh, Hermes Config
Backup to Gitea, weekly-save-restricted-bot-resources, Evening Stand-Down Briefing, plus two disabled).
Completing runs inside the storm window by provider: **210 `no_agent` / no provider + 1 `no_agent` on
`custom:omni`** — that is the *entire* green of the fleet, and **zero model calls completed in the 34 hours.**
**Every model-calling job in the fleet is on the one node that failed.** Verified from `jobs.json`: **all 18
enabled agent-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2).
⚠ **Correction to a first pass at this window:** Evening Stand-Down Briefing (`e530c89f2be0`) carries
`provider: custom:omni` — but it is also **`no_agent: true`**, a script-only job whose provider/model fields
are vestigial. Its clean 18:45 runs on *both* 10-14 and 10-15 are therefore **not** evidence that CT600
stayed healthy; it never called a model. No cron job exercises `custom:omni` with an LLM turn
(`d40085da631c`, the only other `custom:omni` pin, is **disabled**), so the cron fleet **cannot** localise
this outage between the two OmniRoute nodes. What *is* reachable is the leaf error itself — `Connection
refused` to `100.78.63.20:20128` — which points at **`omniroute-1`**.
## The watchdog's reds are derivative
`96eb95ce12f2` (Mnemosyne Watchdog, `no_agent`, script-only) accounts for **12 of the 36 failure rows** — it
reddens when its log scan finds errors, so during a provider storm it is *reporting the weather, not the
fault*. All six of its incidents auto-resolved. Do not chase watchdog reds as a root cause; read
`executions.error` on the jobs that actually make model calls. It also reddens in **bursts** (12:15/12:30,
then 18:15/18:30), not continuously — its trigger is a log sentinel, so it under-reports the window rather
than bracketing it.
## Incident bookkeeping
Open incidents after the storm: **8 `alerted`** + **2 `detected`** + 2 pre-existing = **12 open**
(9 alerted / 3 detected). Each clears **only on the next successful run of its own job**, which for the
weekly and twice-weekly jobs means days of visible noise:
`08eb86f02d37` 10-16 08:30 · `dca8482e4f76` 10-16 07:00 · `85edf6772bca` 10-16 07:00 ·
`ecc1fc13655a` 10-16 12:00 · `1a8e90ea2271` 10-16 03:00 · `0e519874d280` 10-19 (Mon) ·
`06a0a1c8704f` 10-22 (Thu) · `c8d52904e1d6` 10-22 (Thu).
Pre-existing and **not** storm-caused: `1b23c042e17d` (alerted since 10-09, next slot 10-16 08:00) and
`0e753e_e7f8194a7d06` (FB Birthday Monthly, detected since 10-01, next slot 11-01).
## Lessons
1. **The entire scheduled estate rides one OmniRoute node.** All 18 enabled model-calling jobs are pinned to
`custom:omni-101`; the only other model-capable cron pin, `custom:omni`, sits on a `no_agent` job. Two
live OmniRoute nodes exist (`omniroute` CT600 at 100.88.81.19, `omniroute-1` at 100.78.63.20) with **no
failover between them**, so one node's availability gates the whole estate's output. This is the second
multi-day storm from the same topology → [[incidents/2026-10-09-cron-provider-timeout-storm]].
2. **`provider` on a `no_agent` job is decorative.** Never read a script-only job's green row as evidence of
provider health — check `no_agent` before concluding anything from a clean run.
3. **Both Mnemosyne monitors sit on the same node as the memory system they watch** — both failed, so
coverage figures went unrefreshed for the whole window (last explicit figures remain 2026-10-12).
Monitoring that shares a blast radius with its subject is degraded monitoring.
4. **A log-sentinel watchdog multiplies failure counts.** 36 failure rows were really 24, across 12 workers.
Tally per-job, not per-row, or the scale reads wrong.
5. **A failed nightly audit leaves a hole, not a gap in coverage.** The 10-14 audit died on this same storm,
so this page covers **two** windows (10-14 → 10-16).
## Related
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — the earlier storm; same topology, same failure class.
- [[systems/omniroute]] — node layout, providers and pins.
- [[systems/scheduled-tasks]] — per-window fleet tally and incident ledger.
- [[systems/mnemosyne]] — the two coverage monitors that went dark.
+3 -2
View File
@@ -1,6 +1,6 @@
# Anthony's Systems Wiki # Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure, > Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-10-14 > model routing and automation. Last updated: 2026-10-16
> >
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
@@ -33,10 +33,11 @@
### Integrations ### Integrations
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **4 carry a non-clean `last_status`** as of 2026-10-14 00:15. **The clean streak broke on 2026-10-13: `167 completed` / 1 `failed`**, after 10-12 finalised `175 / 0` (10-11 `172 / 0`, 10-10 `175 / 0`) — `1a8e90ea2271` (wiki-stocktake-maintenance, 03:00) hit `HTTP 404: No active credentials for provider: opencode`, its **third failure in six days**, so **open incidents rose 2 → 3**. Two rows cleared: `fad5a86979d4` (10-12 03:05), `dca8482e4f76` (10-13 07:02). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. ⚠ A green cron row is not proof of delivery: GitHub Trending Radar's Telegram token is revoked (`getMe` → 401) while the row stays green — it next runs Mon 09:00. See [[incidents/2026-10-09-cron-provider-timeout-storm]] - [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-16 00:1x — the **`omniroute-1` connection-refused storm (2026-10-14 → 10-15)** cost **24 model-call failures across 12 jobs** and blanked ~34 h of digests; **all 18 model-calling jobs sit on `custom:omni-101` with no failover**. See [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]. Prior audit: **4 carried a non-clean `last_status`** as of 2026-10-14 00:15. **The clean streak broke on 2026-10-13: `167 completed` / 1 `failed`**, after 10-12 finalised `175 / 0` (10-11 `172 / 0`, 10-10 `175 / 0`) — `1a8e90ea2271` (wiki-stocktake-maintenance, 03:00) hit `HTTP 404: No active credentials for provider: opencode`, its **third failure in six days**, so **open incidents rose 2 → 3**. Two rows cleared: `fad5a86979d4` (10-12 03:05), `dca8482e4f76` (10-13 07:02). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. ⚠ A green cron row is not proof of delivery: GitHub Trending Radar's Telegram token is revoked (`getMe` → 401) while the row stays green — it next runs Mon 09:00. See [[incidents/2026-10-09-cron-provider-timeout-storm]]
- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13). - [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13).
## Incidents ## Incidents
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved)
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00) - [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00)
- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved) - [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved)
- [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05) - [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05)
+65
View File
@@ -715,3 +715,68 @@ single-host in framing: [[systems/omniroute]], [[infrastructure/hosts]], [[syste
OmniRoute topology needs a dedicated pass before anyone relies on the current four-node story. OmniRoute topology needs a dedicated pass before anyone relies on the current four-node story.
No secrets written; host names, IPs, VMIDs and job/incident IDs only. No secrets written; host names, IPs, VMIDs and job/incident IDs only.
---
## [2026-10-16] audit | nightly review of 2026-10-14 → 2026-10-15 (two days — the 10-14 audit died inside the same outage) — a 34-hour fleet blackout from one OmniRoute node
Ran 2026-10-16 00:0x–00:3x AWST. **Scope is two windows, not one:** the 2026-10-14 23:59 Wiki Audit
(`cbfe434a3b5e`) failed on the fault below, so the 10-14 window was never covered.
**Finding: fleet-wide cron blackout, 2026-10-14 08:30:27 → 2026-10-15 18:07 AWST (~34 h).**
- **Root cause.** `httpcore.ConnectError: [Errno 111] Connection refused` against
`http://100.78.63.20:20128` — the **`omniroute-1`** node, provider **`custom:omni-101`** (confirmed in
`config.yaml`: custom provider `name: omni-101`, `base_url: http://100.78.63.20:20128`). It surfaced as
`openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` — the exact string
recorded on every failing execution row. 328 `Errno 111` lines in the live `errors.log`, 504 across the
rotations.
- **Impact.** 10-14 = **160 completed / 12 failed**; 10-15 = **151 completed / 24 failed** (+1 in flight).
Against 10-12 `175/0`, 10-13 `167/1`. **36 failure rows across 13 jobs**, but only **24 are genuine
model-call failures** on **12 jobs** — the other 12 rows are the Mnemosyne Watchdog reddening on a log
sentinel (symptom, not fault). 10-15 is the worst cron day since 2026-10-09 (15).
- **The finding that matters most — there is no failover.** Verified from live `jobs.json`: **all 18
enabled `agent`-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2). The only other model-capable pin, `custom:omni` (CT600, `omniroute:20128`),
sits on `e530c89f2be0` — which is **`no_agent: true`**; `d40085da631c` is disabled. Completing runs inside
the window by provider: **210 `no_agent` / no provider + 1 `no_agent` on `custom:omni`**. So **zero model
calls completed in 34 hours**, and the fleet has no job that could even report the second node's health.
- **Correction made during this audit.** A first pass read Evening Stand-Down Briefing's clean 18:45 runs
(both days) as evidence CT600 stayed healthy. Wrong: it is `no_agent` and never calls a model. Recorded as
a standing rule — *never read a `no_agent` job's green row as provider-health evidence* — in the incident
page and in [[systems/scheduled-tasks]].
- **User-visible cost: silence.** Digests never delivered: Tech & AI, Daily Marketing Job Radar, Political
News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick. The **weekly Perth
New Restaurant was missed outright** (next slot 10-22).
- **Recovery, verified live (not assumed).** `1b31187397a5` completed **10-15 22:01:56** (streak 4 → 0);
`96eb95ce12f2` **10-16 00:01:36**; `b63ed5817212` **10-16 00:02:29**; `ea79a2107e36` **10-16 00:09:48**.
Probes 2026-10-16 00:1x: `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in 10 ms** (up, auth
required); same on `omniroute:20128` / `100.88.81.19`.
- **Fleet bookkeeping.** **Non-clean `last_status`: 3 → 12** (9 new: `08eb86f02d37`, `ecc1fc13655a`,
`dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`,
`cbfe434a3b5e`; three cleared in the window: `b63ed5817212`, `1b31187397a5`, `ea79a2107e36`).
**Open incidents 3 → 12** (9 `alerted` / 3 `detected`). Long-standing and pre-storm: `1b23c042e17d`
(Fri 08:00), `0e753e9720fd` (1st 09:00), `066bb5191280` (`delivery_failed`, dead Telegram thread 34795).
- **Mnemosyne.** Both monitors sit on the failing node, so **no coverage figures were refreshed for ~34 h**;
last explicit read remains 2026-10-12 12:02 (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %).
Script-only Mnemosyne jobs were unaffected.
- **Self-healing gap left alone deliberately.** The 10-15 03:00 stocktake failure left
`raw/stocktake/freshness-20261015.txt` unwritten and `systems/skills-index.md` stamped 10-14. Its next
slot (10-16 03:00) closes the gap, so nothing was regenerated by hand — regenerating would race the
scheduled run.
- **Sessions.** 36 sessions across the two days, **all `cron`, zero interactive** — fourth consecutive
window with no interactive session.
**Pages touched:** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` (**new**, 9,303 B);
`systems/scheduled-tasks.md` (frontmatter, **Count** current-state bullet, new Re-verified block);
`systems/omniroute.md` (frontmatter + new Recent Changes section); `systems/mnemosyne.md` (frontmatter +
blast-radius note on the two monitors); `current-state.md` (frontmatter + new top section, `Updated:` line);
`index.md` (`Last updated`, new incidents entry, scheduled-tasks bullet); `log.md` (this entry).
**Nothing removed or archived** — no retired job, no obsolete system, so `index.md` structure gains one page.
**Durable fix still open (needs Anthony's call):** the single-node topology. All 18 model-calling jobs on
one OmniRoute node means this recurs on the next `omniroute-1` blip — as it did on 2026-10-09. Options are
a second provider pin on a subset of jobs, or router-side failover. **Not applied** — that is a routing
decision, not an audit one.
No secrets written; host names, IPs, VMIDs and job/incident IDs only.
+12 -2
View File
@@ -3,8 +3,8 @@ title: Mnemosyne
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-13 updated: 2026-10-16
verified_on: 2026-10-13 verified_on: 2026-10-16
confidence: high confidence: high
tags: [memory, mnemosyne, backend] tags: [memory, mnemosyne, backend]
sources: [live-probe, mnemosyne-db, config, ops-profile-config] sources: [live-probe, mnemosyne-db, config, ops-profile-config]
@@ -94,6 +94,16 @@ Primary agent memory store for Hermes.
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception - Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95% - Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent - Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
- ⚠ **Blast-radius note (2026-10-16).** Both monitors above (`ea79a2107e36` Health every 12 h,
`b63ed5817212` Embedding Coverage every 6 h) are pinned to **`custom:omni-101`** — the same node the
memory system's own reporting depends on. During the **2026-10-14 → 10-15 `omniroute-1` outage both
failed on every slot**, so **no coverage figures were refreshed for ~34 hours**; the last explicit read
remains the **2026-10-12 12:02** run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %), and the
**2026-10-16 00:09:48** Health Monitor run is the first clean read since. The two script-only jobs
(`consolidate_mnemosyne.sh`, `mnemosyne-backup.sh`) were **unaffected** — they make no model call.
Monitoring that shares a blast radius with its subject is degraded monitoring; a coverage alert that
cannot fire during a router outage is exactly the case this has to cover.
→ [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
## History and Evidence ## History and Evidence
- **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]]. - **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
+21 -2
View File
@@ -3,8 +3,8 @@ title: OmniRoute
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-11 updated: 2026-10-16
verified_on: 2026-10-11 verified_on: 2026-10-16
confidence: high confidence: high
tags: [routing, provider, model, inference] tags: [routing, provider, model, inference]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -66,6 +66,25 @@ The wiki previously recorded CT600 as retired and CT601 as the live successor. *
- Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request. - Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
- Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map. - Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
## Recent Changes (2026-10-16) — `omniroute-1` connection-refused storm
- **`omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) was TCP-unreachable from 2026-10-14
08:30 to 2026-10-15 ~18:07 AWST** — `httpcore.ConnectError: [Errno 111] Connection refused` →
`openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` Result: **24 genuine
cron model-call failures across 12 jobs** plus 12 derivative watchdog rows, and **every scheduled digest
for ~34 hours was silently suppressed** (including the 10-14 23:59 Wiki Audit).
- **This node is a single point of failure for the whole scheduled estate.** All **18** enabled
`agent`-mode cron jobs are pinned to `custom:omni-101` (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2). The only other model-capable cron pin — **`custom:omni` (CT600,
`omniroute:20128`)** — is attached to a **`no_agent`** job (`e530c89f2be0`, Evening Stand-Down), so it is
**never exercised by a model call**; `d40085da631c`, the only other `custom:omni` pin, is **disabled**.
The cron fleet therefore has **no model-call failover between the two nodes**, and no job that can even
tell you whether the second node is up.
- **Recovered — verified 2026-10-16 00:1x:** `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in
10 ms** (up, auth required); same on `omniroute:20128` / `100.88.81.19`. Four jobs have completed on the
node since 2026-10-15 22:01.
- Full write-up: [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
## Recent Changes (2026-08-14) ## Recent Changes (2026-08-14)
- **OmniRoute factory reset on CT601.** Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (`auto/best-fast`, `auto/best-coding`, etc.). Old state backed up at `/opt/omniroute/factory-reset-backup-20260814-225524`. - **OmniRoute factory reset on CT601.** Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (`auto/best-fast`, `auto/best-coding`, etc.). Old state backed up at `/opt/omniroute/factory-reset-backup-20260814-225524`.
- **Providers re-added from Notion API vault:** Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry. - **Providers re-added from Notion API vault:** Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry.
+72 -2
View File
@@ -3,8 +3,8 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-14 updated: 2026-10-16
verified_on: 2026-10-14 verified_on: 2026-10-16
review_after: 2026-10-25 review_after: 2026-10-25
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
@@ -14,6 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
- **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, re-verified 2026-10-13 — unchanged; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10) - **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, re-verified 2026-10-13 — unchanged; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10)
- **Non-clean `last_status`:** **3** (was 5 at the 2026-10-12 audit) — `0e753e9720fd` (`error`, streak 0), `1b23c042e17d` (`error`, streak 1), `066bb5191280` (`delivery_failed`, streak 0). `fad5a86979d4` and `dca8482e4f76` cleared on **2026-10-12** when both ran clean on schedule. - **Non-clean `last_status`:** **3** (was 5 at the 2026-10-12 audit) — `0e753e9720fd` (`error`, streak 0), `1b23c042e17d` (`error`, streak 1), `066bb5191280` (`delivery_failed`, streak 0). `fad5a86979d4` and `dca8482e4f76` cleared on **2026-10-12** when both ran clean on schedule.
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]]. - **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]].
@@ -406,3 +407,72 @@ or repair OmniRoute's `opencode` provider credentials); GitHub Trending Radar to
Mon 09:00); Camofox containment vs. multi-tab workloads (2026-10-13); Quartz rebuild step inert; Mon 09:00); Camofox containment vs. multi-tab workloads (2026-10-13); Quartz rebuild step inert;
skills-index off-by-one; **4** non-clean `last_status` rows (above); Mattermost platform unreachable; skills-index off-by-one; **4** non-clean `last_status` rows (above); Mattermost platform unreachable;
gateway runtime env outside the 15-minute dependency guard. gateway runtime env outside the 15-minute dependency guard.
## Re-verified 2026-10-15 / 2026-10-16 — the `omni-101` connection-refused storm
Window: **2026-10-14 00:00 → 2026-10-16 00:00 AWST** (two days — the 2026-10-14 23:59 audit died inside the
same outage, so the 10-14 window was never covered). Live probes 2026-10-16 00:0x–00:2x.
Fleet **29 jobs (26 enabled / 3 disabled)** — unchanged. **Open incidents 3 → 12** (9 `alerted` / 3 `detected`).
**The tally broke hard.** 2026-10-14 = **160 completed / 12 failed**; 2026-10-15 = **151 completed /
24 failed** / 1 running. Against a four-day clean run (10-10 `175/0`, 10-11 `172/0`, 10-12 `175/0`,
10-13 `167/1`). **36 failure rows across 13 jobs** — but only **24 are genuine model-call failures** on
**12 jobs**; the other **12 rows are the Mnemosyne Watchdog reddening on a log sentinel** (symptom, not
fault). Day-level, 10-15 is the worst cron day since 2026-10-09 (15).
**Root cause.** `httpcore.ConnectError: [Errno 111] Connection refused` against
`http://100.78.63.20:20128` — the **`omniroute-1`** node, provider **`custom:omni-101`**
(`config.yaml`: custom provider `name: omni-101`). Surfaced as `openai.APIConnectionError: Connection error.`
→ `RuntimeError: Connection error.` 328 `Errno 111` lines in the live `errors.log`, 504 across rotations.
Window 10-14 08:30:27 → 10-15 18:07, recovered by 10-15 22:01. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
**Structural finding — no model-call failover anywhere in the fleet.** Verified from `jobs.json`: **all 18
enabled agent-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2). The only other model-capable cron pin, `custom:omni` (CT600, `omniroute:20128`),
sits on `e530c89f2be0` — which is **`no_agent: true`**, a script-only job, so its provider field is
vestigial. Completing runs inside the window by provider: **210 `no_agent` / no provider + 1 `no_agent` on
`custom:omni`** — that is the entire green of the fleet, and **zero model calls completed in 34 hours**.
⚠ A first pass at this window read Evening Stand-Down's clean 18:45 runs as proof CT600 stayed up. It is
not: Stand-Down never calls a model. **Never read a `no_agent` job's green row as provider-health evidence.**
**Worst-affected jobs:** `b63ed5817212` Mnemosyne Embedding Coverage ×6 (every slot for 30 h),
`1b31187397a5` Mosquito Tracker ×4, `ea79a2107e36` Mnemosyne Health Monitor ×3, `08eb86f02d37` Gmail Action
Digest ×2, `ecc1fc13655a` FreshRSS Midday ×2, then one apiece for `85edf6772bca`, `dca8482e4f76`,
`0e519874d280`, `06a0a1c8704f`, `c8d52904e1d6`, `1a8e90ea2271`, `cbfe434a3b5e`. Full table in the incident
page. **User-visible cost:** Digests that never went out — Tech & AI, Daily Marketing Job Radar, Political
News, AI Newsletter Curation for Patrick, Gmail Action Digest ×2, FreshRSS Midday ×2, and the weekly Perth
New Restaurant (missed outright; next chance 10-22). The missed audit also meant the 10-14 window went
uncovered.
**Recovery verified by schedule, not by probe alone.** `1b31187397a5` completed 10-15 **22:01:56** (streak
4 → 0); `b63ed5817212` 10-16 **00:02:29**; `96eb95ce12f2` 10-16 **00:01:36**; `ea79a2107e36` 10-16
**00:09:48**. Live 10-16 00:1x: `GET /v1/models` on **100.78.63.20:20128 → HTTP 401 in 10 ms** (up, auth
required) and the same on `omniroute:20128` / 100.88.81.19.
**Non-clean `last_status`: 3 → 12.** Cleared during the window: `b63ed5817212`, `1b31187397a5`,
`ea79a2107e36` (all on successful post-storm runs). Added: the nine rows listed under **Count** above.
Unchanged long-standing: `1b23c042e17d` (10-09 error), `0e753e9720fd` (10-01 error), `066bb5191280`
(10-01 `delivery_failed`, dead Telegram `thread_id 34795`).
**Mnemosyne.** Both monitors sat on the failing node, so **no coverage figures were refreshed in the
window** — last explicit numbers remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic
1050/1056 = 99.4 %); the 10-16 00:09 Health Monitor run is the first clean read since 10-12.
→ [[systems/mnemosyne]]
**Stocktake gap (self-healing).** `1a8e90ea2271` failed 10-15 03:00 → `raw/stocktake/freshness-20261015.txt`
was never written and `systems/skills-index.md` still carries the 10-14 stamp. Its next slot is 10-16 03:00,
so the gap closes on its own — **not** regenerated by hand here, to avoid racing the scheduled run.
**Sessions.** **36 sessions in the two-day window, all `cron`, zero interactive** — fourth consecutive
window with no interactive session.
**Pages touched:** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` (new),
`systems/omniroute.md`, `systems/mnemosyne.md`, `systems/scheduled-tasks.md` (this block, frontmatter,
Count), `current-state.md`, `index.md`, `log.md`.
**Nothing removed or archived:** no retired job, no obsolete system → `index.md` structure gains one page.
**Still open after this cycle:** the **no-failover topology** itself (all 18 model-calling jobs on one
OmniRoute node — the durable fix is a second provider pin or a router-side failover, and it needs Anthony's
call); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab
workloads; Quartz rebuild step inert; skills-index off-by-one; **12** non-clean `last_status` rows;
Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.