diff --git a/current-state.md b/current-state.md index fce9e0f..d8c0633 100644 --- a/current-state.md +++ b/current-state.md @@ -3,8 +3,8 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-14 -verified_on: 2026-10-14 +updated: 2026-10-16 +verified_on: 2026-10-16 review_after: 2026-10-25 confidence: high tags: [system] @@ -12,7 +12,18 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv --- # Current State -Updated: 2026-10-14 +Updated: 2026-10-16 + +## Recent Changes (2026-10-16, nightly audit; live probes 2026-10-16 00:0x–00:2x AWST) + +- **⚠ A fleet-wide 34-hour cron blackout just ended — `omniroute-1` was refusing TCP connections.** `10-14 = 160 completed / 12 failed` and `10-15 = 151 completed / 24 failed`, against a clean 10-10→10-12 (`175/0`, `172/0`, `175/0`) and 10-13 (`167/1`). Root cause: `httpcore.ConnectError: [Errno 111] Connection refused` against `http://100.78.63.20:20128` (provider `custom:omni-101`), surfacing in the cron DB as `openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` Window **10-14 08:30:27 → 10-15 18:07**, recovered by **10-15 22:01**. **24 genuine model-call failures on 12 jobs** + 12 derivative watchdog reds = 36 failure rows across 13 jobs. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] +- **Everything model-calling is pinned to that one node — there is no failover anywhere in the fleet.** All **18** enabled `agent`-mode cron jobs run on `custom:omni-101` (`smart-route` ×10, `free-stack` ×6, `deepseek-paid-smart` ×2). The only other model-capable pin, `custom:omni` (CT600 `omniroute:20128`), carries just the **`no_agent`** job `e530c89f2be0`, whose provider field is vestigial; `d40085da631c` is disabled. Completing runs inside the window: **211 — every single one script-only.** **Zero model calls succeeded for 34 hours.** → [[systems/omniroute]], [[systems/scheduled-tasks]] +- **The cost was silence, not errors.** Digests that never went out: Tech & AI, Daily Marketing Job Radar, Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick; the weekly Perth New Restaurant was **missed outright** (next chance 10-22). The **10-14 23:59 Wiki Audit itself died on the same fault**, so the 10-14 window was never audited — this entry spans two days. → [[systems/scheduled-tasks]] +- **Recovered, verified live.** `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in 10 ms** (up, auth required); same on `omniroute:20128` / `100.88.81.19`. First green model-call job: `1b31187397a5` at **10-15 22:01:56**; then `b63ed5817212` (**10-16 00:02:29**) and `ea79a2107e36` (**10-16 00:09:48**). +- **The backlog unwinds on schedule rather than by hand.** **Non-clean `last_status`: 3 → 12** (9 new storm rows; three cleared as their slots came round), **open incidents 3 → 12** (9 `alerted` / 3 `detected`). The two longest-lived — `1b23c042e17d` (Fri 08:00 slot) and `0e753e9720fd` (1st 09:00) — **pre-date the storm**. → [[systems/scheduled-tasks]] +- **Both Mnemosyne monitors sit inside the blast radius.** Health (12 h) and Embedding Coverage (6 h) both failed on every slot, so **no coverage figures were refreshed for ~34 h** — the last explicit read remains 2026-10-12 (working 96.7 %, episodic 99.4 %). The script-only Mnemosyne jobs were unaffected. → [[systems/mnemosyne]] +- **Self-healing gap, not a manual repair.** The failed 10-15 03:00 stocktake left `raw/stocktake/freshness-20261015.txt` unwritten and `systems/skills-index.md` stamped 10-14; the 10-16 03:00 slot closes it, so nothing was regenerated by hand here. +- **Fourth consecutive window with no interactive session** — **36 sessions** across the two days, all `cron`. ## Recent Changes (2026-10-14, 03:00 stocktake; live probes 03:01–03:1x AWST) diff --git a/incidents/2026-10-14-omniroute-node-connection-refused-storm.md b/incidents/2026-10-14-omniroute-node-connection-refused-storm.md new file mode 100644 index 0000000..beb1181 --- /dev/null +++ b/incidents/2026-10-14-omniroute-node-connection-refused-storm.md @@ -0,0 +1,141 @@ +--- +title: OmniRoute `omni-101` Node Outage — Connection-Refused Storm +type: concept +status: resolved +created: 2026-10-16 +updated: 2026-10-16 +verified_on: 2026-10-16 +review_after: 2026-10-23 +confidence: high +tags: [cron, router, provider, infrastructure] +sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, hermes-errors-log, live-probe] +--- + +# OmniRoute `omni-101` Node Outage — Connection-Refused Storm (2026-10-14 → 2026-10-15) + +**Window:** 2026-10-14 08:30 → 2026-10-15 18:07 AWST (**~34 hours**) +**Status:** **Resolved.** `omniroute-1` answers again; four scheduled jobs have since completed on it. +**Impact:** **36 failed execution rows across 13 jobs** — 24 genuine model-call failures on 12 jobs, plus +12 watchdog *symptom* reds. **Every model call attempted in the window failed** (see below). For scale: +10-12 recorded `0 failed` and 10-13 recorded `1`. +**Severity:** the largest failure count since the 2026-10-09 storm (15 in one day); this one ran **two days** +and silently suppressed most of the day's digests. + +## Root cause + +`TCP connect refused` on **`http://100.78.63.20:20128`** — the **`omniroute-1`** node (Tailscale peer +`omniroute-1`, 100.78.63.20), registered in Hermes as provider **`custom:omni-101`** +(`config.yaml`: custom provider `name: omni-101`, `base_url: http://100.78.63.20:20128`). + +``` +httpcore.ConnectError: [Errno 111] Connection refused + → openai.APIConnectionError: Connection error. + → RuntimeError: Connection error. (as recorded in executions.db) +``` +`hermes-errors.log` carried **328 `Errno 111`** occurrences in the live file alone (504 across the +rotations during the window). Every failing run recorded the identical leaf string, so all 24 model-call +failures are **one fault**, not 24 different ones. + +## Timeline (AWST) + +| Time | Event | +|---|---| +| **10-14 08:30:27** | First failed run — Gmail Action Digest (`08eb86f02d37`). Isolated: 06:00 and 09:00–11:45 runs were clean. | +| **10-14 12:00** | Cascade — all three noon jobs fail (`b63ed5817212`, `ea79a2107e36`, `ecc1fc13655a`) and the watchdog reddens at 12:15/12:30. | +| **10-14 15:00 / 18:00 / 22:00** | Mosquito Tracker ×2, Embedding Coverage, Health Monitor all fail. | +| **10-14 23:59** | **The nightly Wiki Audit itself failed** — so the 10-14 window was never audited. | +| **10-15 03:00** | `wiki-stocktake-maintenance` (`1a8e90ea2271`) failed — no `freshness-20261015.txt` written. | +| **10-15 07:00–10:00** | Prime-time digests fail: Tech & AI, Daily Marketing Job Radar, Political News, AI Newsletter Curation for Patrick. | +| **10-15 12:00** | Noon jobs fail again. | +| **10-15 18:00:41** | **Last genuine failure** — Embedding Coverage Monitor. | +| **10-15 18:30:26** | Last watchdog red. | +| **10-15 22:01:56** | **Recovery marker** — Mosquito Tracker (`1b31187397a5`) completes on `omni-101`; streak 4 → 0. | +| **10-16 00:00–00:10** | Embedding Coverage (00:02:29), Health Monitor (00:09:48) and the Watchdog (00:01:36) all complete — three streaks cleared. | +| **10-16 00:1x** | **Live re-probe:** `GET /v1/models` on 100.78.63.20:20128 → **HTTP 401** in 10 ms (alive, auth required). Same on `omniroute:20128` / 100.88.81.19. | + +## Blast radius + +| Job | Name | Failures | First → last | +|---|---|---|---| +| `b63ed5817212` | Mnemosyne Embedding Coverage (6-hourly) | 6 | 10-14 12:00 → 10-15 18:00 | +| `1b31187397a5` | Mosquito Tracker (4×/day) | 4 | 10-14 15:00 → 10-15 15:00 | +| `ea79a2107e36` | Mnemosyne Health Monitor (12-hourly) | 3 | 10-14 12:00 → 10-15 12:00 | +| `08eb86f02d37` | Gmail Action Digest (daily 08:30) | 2 | 10-14 08:30 → 10-15 08:30 | +| `ecc1fc13655a` | FreshRSS Midday Briefing (daily 12:00) | 2 | 10-14 12:00 → 10-15 12:00 | +| `85edf6772bca` | Tech & AI Newsletter Digest (daily 07:00) | 1 | 10-15 07:00 | +| `dca8482e4f76` | Daily Marketing Job Radar (weekdays 07:00) | 1 | 10-15 07:00 | +| `0e519874d280` | Political News Digest (Mon/Thu 08:00) | 1 | 10-15 08:00 | +| `06a0a1c8704f` | AI Newsletter Curation for Patrick (Thu 10:00) | 1 | 10-15 10:00 | +| `c8d52904e1d6` | Perth New Restaurant (Thu 18:00) | 1 | 10-15 18:00 | +| `1a8e90ea2271` | wiki-stocktake-maintenance (daily 03:00) | 1 | 10-15 03:00 | +| `cbfe434a3b5e` | Wiki Audit (daily 23:59) | 1 | 10-14 23:59 | +| `96eb95ce12f2` | Mnemosyne Watchdog (every 15 min) | 12 | **symptom only** — see below | + +Day tallies: **10-14 = 160 completed / 12 failed** (172), **10-15 = 151 completed / 24 failed / 1 running** (176). + +**User-visible cost:** several digests simply never went out — Tech & AI, Daily Marketing Job Radar, +Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick, and the +weekly Perth New Restaurant (missed outright; next chance 10-22). + +## What stayed green — and what that does *not* prove + +**Script-only jobs kept running normally.** The fleet holds **10 `no_agent` jobs** (no LLM turn: Mnemosyne +Auto-Consolidate, DB Backup, Watchdog, Daily Birthday Checker, Job Tracker Dashboard Refresh, Hermes Config +Backup to Gitea, weekly-save-restricted-bot-resources, Evening Stand-Down Briefing, plus two disabled). +Completing runs inside the storm window by provider: **210 `no_agent` / no provider + 1 `no_agent` on +`custom:omni`** — that is the *entire* green of the fleet, and **zero model calls completed in the 34 hours.** + +**Every model-calling job in the fleet is on the one node that failed.** Verified from `jobs.json`: **all 18 +enabled agent-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6, +`deepseek-paid-smart` ×2). + +⚠ **Correction to a first pass at this window:** Evening Stand-Down Briefing (`e530c89f2be0`) carries +`provider: custom:omni` — but it is also **`no_agent: true`**, a script-only job whose provider/model fields +are vestigial. Its clean 18:45 runs on *both* 10-14 and 10-15 are therefore **not** evidence that CT600 +stayed healthy; it never called a model. No cron job exercises `custom:omni` with an LLM turn +(`d40085da631c`, the only other `custom:omni` pin, is **disabled**), so the cron fleet **cannot** localise +this outage between the two OmniRoute nodes. What *is* reachable is the leaf error itself — `Connection +refused` to `100.78.63.20:20128` — which points at **`omniroute-1`**. + +## The watchdog's reds are derivative + +`96eb95ce12f2` (Mnemosyne Watchdog, `no_agent`, script-only) accounts for **12 of the 36 failure rows** — it +reddens when its log scan finds errors, so during a provider storm it is *reporting the weather, not the +fault*. All six of its incidents auto-resolved. Do not chase watchdog reds as a root cause; read +`executions.error` on the jobs that actually make model calls. It also reddens in **bursts** (12:15/12:30, +then 18:15/18:30), not continuously — its trigger is a log sentinel, so it under-reports the window rather +than bracketing it. + +## Incident bookkeeping + +Open incidents after the storm: **8 `alerted`** + **2 `detected`** + 2 pre-existing = **12 open** +(9 alerted / 3 detected). Each clears **only on the next successful run of its own job**, which for the +weekly and twice-weekly jobs means days of visible noise: +`08eb86f02d37` 10-16 08:30 · `dca8482e4f76` 10-16 07:00 · `85edf6772bca` 10-16 07:00 · +`ecc1fc13655a` 10-16 12:00 · `1a8e90ea2271` 10-16 03:00 · `0e519874d280` 10-19 (Mon) · +`06a0a1c8704f` 10-22 (Thu) · `c8d52904e1d6` 10-22 (Thu). +Pre-existing and **not** storm-caused: `1b23c042e17d` (alerted since 10-09, next slot 10-16 08:00) and +`0e753e_e7f8194a7d06` (FB Birthday Monthly, detected since 10-01, next slot 11-01). + +## Lessons + +1. **The entire scheduled estate rides one OmniRoute node.** All 18 enabled model-calling jobs are pinned to + `custom:omni-101`; the only other model-capable cron pin, `custom:omni`, sits on a `no_agent` job. Two + live OmniRoute nodes exist (`omniroute` CT600 at 100.88.81.19, `omniroute-1` at 100.78.63.20) with **no + failover between them**, so one node's availability gates the whole estate's output. This is the second + multi-day storm from the same topology → [[incidents/2026-10-09-cron-provider-timeout-storm]]. +2. **`provider` on a `no_agent` job is decorative.** Never read a script-only job's green row as evidence of + provider health — check `no_agent` before concluding anything from a clean run. +3. **Both Mnemosyne monitors sit on the same node as the memory system they watch** — both failed, so + coverage figures went unrefreshed for the whole window (last explicit figures remain 2026-10-12). + Monitoring that shares a blast radius with its subject is degraded monitoring. +4. **A log-sentinel watchdog multiplies failure counts.** 36 failure rows were really 24, across 12 workers. + Tally per-job, not per-row, or the scale reads wrong. +5. **A failed nightly audit leaves a hole, not a gap in coverage.** The 10-14 audit died on this same storm, + so this page covers **two** windows (10-14 → 10-16). + +## Related +- [[incidents/2026-10-09-cron-provider-timeout-storm]] — the earlier storm; same topology, same failure class. +- [[systems/omniroute]] — node layout, providers and pins. +- [[systems/scheduled-tasks]] — per-window fleet tally and incident ledger. +- [[systems/mnemosyne]] — the two coverage monitors that went dark. diff --git a/index.md b/index.md index 186b41d..157e2d6 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-14 +> model routing and automation. Last updated: 2026-10-16 > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. @@ -33,10 +33,11 @@ ### Integrations - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. -- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **4 carry a non-clean `last_status`** as of 2026-10-14 00:15. **The clean streak broke on 2026-10-13: `167 completed` / 1 `failed`**, after 10-12 finalised `175 / 0` (10-11 `172 / 0`, 10-10 `175 / 0`) — `1a8e90ea2271` (wiki-stocktake-maintenance, 03:00) hit `HTTP 404: No active credentials for provider: opencode`, its **third failure in six days**, so **open incidents rose 2 → 3**. Two rows cleared: `fad5a86979d4` (10-12 03:05), `dca8482e4f76` (10-13 07:02). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. ⚠ A green cron row is not proof of delivery: GitHub Trending Radar's Telegram token is revoked (`getMe` → 401) while the row stays green — it next runs Mon 09:00. See [[incidents/2026-10-09-cron-provider-timeout-storm]] +- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-16 00:1x — the **`omniroute-1` connection-refused storm (2026-10-14 → 10-15)** cost **24 model-call failures across 12 jobs** and blanked ~34 h of digests; **all 18 model-calling jobs sit on `custom:omni-101` with no failover**. See [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]. Prior audit: **4 carried a non-clean `last_status`** as of 2026-10-14 00:15. **The clean streak broke on 2026-10-13: `167 completed` / 1 `failed`**, after 10-12 finalised `175 / 0` (10-11 `172 / 0`, 10-10 `175 / 0`) — `1a8e90ea2271` (wiki-stocktake-maintenance, 03:00) hit `HTTP 404: No active credentials for provider: opencode`, its **third failure in six days**, so **open incidents rose 2 → 3**. Two rows cleared: `fad5a86979d4` (10-12 03:05), `dca8482e4f76` (10-13 07:02). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. ⚠ A green cron row is not proof of delivery: GitHub Trending Radar's Telegram token is revoked (`getMe` → 401) while the row stays green — it next runs Mon 09:00. See [[incidents/2026-10-09-cron-provider-timeout-storm]] - [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13). ## Incidents +- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved) - [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00) - [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved) - [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05) diff --git a/log.md b/log.md index 97479a6..74aa2c9 100644 --- a/log.md +++ b/log.md @@ -715,3 +715,68 @@ single-host in framing: [[systems/omniroute]], [[infrastructure/hosts]], [[syste OmniRoute topology needs a dedicated pass before anyone relies on the current four-node story. No secrets written; host names, IPs, VMIDs and job/incident IDs only. + +--- + +## [2026-10-16] audit | nightly review of 2026-10-14 → 2026-10-15 (two days — the 10-14 audit died inside the same outage) — a 34-hour fleet blackout from one OmniRoute node + +Ran 2026-10-16 00:0x–00:3x AWST. **Scope is two windows, not one:** the 2026-10-14 23:59 Wiki Audit +(`cbfe434a3b5e`) failed on the fault below, so the 10-14 window was never covered. + +**Finding: fleet-wide cron blackout, 2026-10-14 08:30:27 → 2026-10-15 18:07 AWST (~34 h).** + +- **Root cause.** `httpcore.ConnectError: [Errno 111] Connection refused` against + `http://100.78.63.20:20128` — the **`omniroute-1`** node, provider **`custom:omni-101`** (confirmed in + `config.yaml`: custom provider `name: omni-101`, `base_url: http://100.78.63.20:20128`). It surfaced as + `openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` — the exact string + recorded on every failing execution row. 328 `Errno 111` lines in the live `errors.log`, 504 across the + rotations. +- **Impact.** 10-14 = **160 completed / 12 failed**; 10-15 = **151 completed / 24 failed** (+1 in flight). + Against 10-12 `175/0`, 10-13 `167/1`. **36 failure rows across 13 jobs**, but only **24 are genuine + model-call failures** on **12 jobs** — the other 12 rows are the Mnemosyne Watchdog reddening on a log + sentinel (symptom, not fault). 10-15 is the worst cron day since 2026-10-09 (15). +- **The finding that matters most — there is no failover.** Verified from live `jobs.json`: **all 18 + enabled `agent`-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6, + `deepseek-paid-smart` ×2). The only other model-capable pin, `custom:omni` (CT600, `omniroute:20128`), + sits on `e530c89f2be0` — which is **`no_agent: true`**; `d40085da631c` is disabled. Completing runs inside + the window by provider: **210 `no_agent` / no provider + 1 `no_agent` on `custom:omni`**. So **zero model + calls completed in 34 hours**, and the fleet has no job that could even report the second node's health. +- **Correction made during this audit.** A first pass read Evening Stand-Down Briefing's clean 18:45 runs + (both days) as evidence CT600 stayed healthy. Wrong: it is `no_agent` and never calls a model. Recorded as + a standing rule — *never read a `no_agent` job's green row as provider-health evidence* — in the incident + page and in [[systems/scheduled-tasks]]. +- **User-visible cost: silence.** Digests never delivered: Tech & AI, Daily Marketing Job Radar, Political + News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick. The **weekly Perth + New Restaurant was missed outright** (next slot 10-22). +- **Recovery, verified live (not assumed).** `1b31187397a5` completed **10-15 22:01:56** (streak 4 → 0); + `96eb95ce12f2` **10-16 00:01:36**; `b63ed5817212` **10-16 00:02:29**; `ea79a2107e36` **10-16 00:09:48**. + Probes 2026-10-16 00:1x: `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in 10 ms** (up, auth + required); same on `omniroute:20128` / `100.88.81.19`. +- **Fleet bookkeeping.** **Non-clean `last_status`: 3 → 12** (9 new: `08eb86f02d37`, `ecc1fc13655a`, + `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, + `cbfe434a3b5e`; three cleared in the window: `b63ed5817212`, `1b31187397a5`, `ea79a2107e36`). + **Open incidents 3 → 12** (9 `alerted` / 3 `detected`). Long-standing and pre-storm: `1b23c042e17d` + (Fri 08:00), `0e753e9720fd` (1st 09:00), `066bb5191280` (`delivery_failed`, dead Telegram thread 34795). +- **Mnemosyne.** Both monitors sit on the failing node, so **no coverage figures were refreshed for ~34 h**; + last explicit read remains 2026-10-12 12:02 (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %). + Script-only Mnemosyne jobs were unaffected. +- **Self-healing gap left alone deliberately.** The 10-15 03:00 stocktake failure left + `raw/stocktake/freshness-20261015.txt` unwritten and `systems/skills-index.md` stamped 10-14. Its next + slot (10-16 03:00) closes the gap, so nothing was regenerated by hand — regenerating would race the + scheduled run. +- **Sessions.** 36 sessions across the two days, **all `cron`, zero interactive** — fourth consecutive + window with no interactive session. + +**Pages touched:** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` (**new**, 9,303 B); +`systems/scheduled-tasks.md` (frontmatter, **Count** current-state bullet, new Re-verified block); +`systems/omniroute.md` (frontmatter + new Recent Changes section); `systems/mnemosyne.md` (frontmatter + +blast-radius note on the two monitors); `current-state.md` (frontmatter + new top section, `Updated:` line); +`index.md` (`Last updated`, new incidents entry, scheduled-tasks bullet); `log.md` (this entry). +**Nothing removed or archived** — no retired job, no obsolete system, so `index.md` structure gains one page. + +**Durable fix still open (needs Anthony's call):** the single-node topology. All 18 model-calling jobs on +one OmniRoute node means this recurs on the next `omniroute-1` blip — as it did on 2026-10-09. Options are +a second provider pin on a subset of jobs, or router-side failover. **Not applied** — that is a routing +decision, not an audit one. + +No secrets written; host names, IPs, VMIDs and job/incident IDs only. diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index 4bde911..ee7d547 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -3,8 +3,8 @@ title: Mnemosyne type: system status: active created: 2026-07-22 -updated: 2026-10-13 -verified_on: 2026-10-13 +updated: 2026-10-16 +verified_on: 2026-10-16 confidence: high tags: [memory, mnemosyne, backend] sources: [live-probe, mnemosyne-db, config, ops-profile-config] @@ -94,6 +94,16 @@ Primary agent memory store for Hermes. - Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception - Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95% - Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent +- ⚠ **Blast-radius note (2026-10-16).** Both monitors above (`ea79a2107e36` Health every 12 h, + `b63ed5817212` Embedding Coverage every 6 h) are pinned to **`custom:omni-101`** — the same node the + memory system's own reporting depends on. During the **2026-10-14 → 10-15 `omniroute-1` outage both + failed on every slot**, so **no coverage figures were refreshed for ~34 hours**; the last explicit read + remains the **2026-10-12 12:02** run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %), and the + **2026-10-16 00:09:48** Health Monitor run is the first clean read since. The two script-only jobs + (`consolidate_mnemosyne.sh`, `mnemosyne-backup.sh`) were **unaffected** — they make no model call. + Monitoring that shares a blast radius with its subject is degraded monitoring; a coverage alert that + cannot fire during a router outage is exactly the case this has to cover. + → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] ## History and Evidence - **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]]. diff --git a/systems/omniroute.md b/systems/omniroute.md index 857327c..cec0b5a 100644 --- a/systems/omniroute.md +++ b/systems/omniroute.md @@ -3,8 +3,8 @@ title: OmniRoute type: system status: active created: 2026-07-22 -updated: 2026-10-11 -verified_on: 2026-10-11 +updated: 2026-10-16 +verified_on: 2026-10-16 confidence: high tags: [routing, provider, model, inference] sources: [raw/configs/hermes-config-sanitized.txt] @@ -66,6 +66,25 @@ The wiki previously recorded CT600 as retired and CT601 as the live successor. * - Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request. - Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map. +## Recent Changes (2026-10-16) — `omniroute-1` connection-refused storm + +- **`omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) was TCP-unreachable from 2026-10-14 + 08:30 to 2026-10-15 ~18:07 AWST** — `httpcore.ConnectError: [Errno 111] Connection refused` → + `openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` Result: **24 genuine + cron model-call failures across 12 jobs** plus 12 derivative watchdog rows, and **every scheduled digest + for ~34 hours was silently suppressed** (including the 10-14 23:59 Wiki Audit). +- **This node is a single point of failure for the whole scheduled estate.** All **18** enabled + `agent`-mode cron jobs are pinned to `custom:omni-101` (`smart-route` ×10, `free-stack` ×6, + `deepseek-paid-smart` ×2). The only other model-capable cron pin — **`custom:omni` (CT600, + `omniroute:20128`)** — is attached to a **`no_agent`** job (`e530c89f2be0`, Evening Stand-Down), so it is + **never exercised by a model call**; `d40085da631c`, the only other `custom:omni` pin, is **disabled**. + The cron fleet therefore has **no model-call failover between the two nodes**, and no job that can even + tell you whether the second node is up. +- **Recovered — verified 2026-10-16 00:1x:** `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in + 10 ms** (up, auth required); same on `omniroute:20128` / `100.88.81.19`. Four jobs have completed on the + node since 2026-10-15 22:01. +- Full write-up: [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] + ## Recent Changes (2026-08-14) - **OmniRoute factory reset on CT601.** Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (`auto/best-fast`, `auto/best-coding`, etc.). Old state backed up at `/opt/omniroute/factory-reset-backup-20260814-225524`. - **Providers re-added from Notion API vault:** Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 71e55a7..fda80f4 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,8 +3,8 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-14 -verified_on: 2026-10-14 +updated: 2026-10-16 +verified_on: 2026-10-16 review_after: 2026-10-25 confidence: high tags: [cron, scheduled, automation] @@ -14,6 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count +- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] - **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, re-verified 2026-10-13 — unchanged; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10) - **Non-clean `last_status`:** **3** (was 5 at the 2026-10-12 audit) — `0e753e9720fd` (`error`, streak 0), `1b23c042e17d` (`error`, streak 1), `066bb5191280` (`delivery_failed`, streak 0). `fad5a86979d4` and `dca8482e4f76` cleared on **2026-10-12** when both ran clean on schedule. - **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]]. @@ -406,3 +407,72 @@ or repair OmniRoute's `opencode` provider credentials); GitHub Trending Radar to Mon 09:00); Camofox containment vs. multi-tab workloads (2026-10-13); Quartz rebuild step inert; skills-index off-by-one; **4** non-clean `last_status` rows (above); Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard. + +## Re-verified 2026-10-15 / 2026-10-16 — the `omni-101` connection-refused storm + +Window: **2026-10-14 00:00 → 2026-10-16 00:00 AWST** (two days — the 2026-10-14 23:59 audit died inside the +same outage, so the 10-14 window was never covered). Live probes 2026-10-16 00:0x–00:2x. +Fleet **29 jobs (26 enabled / 3 disabled)** — unchanged. **Open incidents 3 → 12** (9 `alerted` / 3 `detected`). + +**The tally broke hard.** 2026-10-14 = **160 completed / 12 failed**; 2026-10-15 = **151 completed / +24 failed** / 1 running. Against a four-day clean run (10-10 `175/0`, 10-11 `172/0`, 10-12 `175/0`, +10-13 `167/1`). **36 failure rows across 13 jobs** — but only **24 are genuine model-call failures** on +**12 jobs**; the other **12 rows are the Mnemosyne Watchdog reddening on a log sentinel** (symptom, not +fault). Day-level, 10-15 is the worst cron day since 2026-10-09 (15). + +**Root cause.** `httpcore.ConnectError: [Errno 111] Connection refused` against +`http://100.78.63.20:20128` — the **`omniroute-1`** node, provider **`custom:omni-101`** +(`config.yaml`: custom provider `name: omni-101`). Surfaced as `openai.APIConnectionError: Connection error.` +→ `RuntimeError: Connection error.` 328 `Errno 111` lines in the live `errors.log`, 504 across rotations. +Window 10-14 08:30:27 → 10-15 18:07, recovered by 10-15 22:01. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] + +**Structural finding — no model-call failover anywhere in the fleet.** Verified from `jobs.json`: **all 18 +enabled agent-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6, +`deepseek-paid-smart` ×2). The only other model-capable cron pin, `custom:omni` (CT600, `omniroute:20128`), +sits on `e530c89f2be0` — which is **`no_agent: true`**, a script-only job, so its provider field is +vestigial. Completing runs inside the window by provider: **210 `no_agent` / no provider + 1 `no_agent` on +`custom:omni`** — that is the entire green of the fleet, and **zero model calls completed in 34 hours**. +⚠ A first pass at this window read Evening Stand-Down's clean 18:45 runs as proof CT600 stayed up. It is +not: Stand-Down never calls a model. **Never read a `no_agent` job's green row as provider-health evidence.** + +**Worst-affected jobs:** `b63ed5817212` Mnemosyne Embedding Coverage ×6 (every slot for 30 h), +`1b31187397a5` Mosquito Tracker ×4, `ea79a2107e36` Mnemosyne Health Monitor ×3, `08eb86f02d37` Gmail Action +Digest ×2, `ecc1fc13655a` FreshRSS Midday ×2, then one apiece for `85edf6772bca`, `dca8482e4f76`, +`0e519874d280`, `06a0a1c8704f`, `c8d52904e1d6`, `1a8e90ea2271`, `cbfe434a3b5e`. Full table in the incident +page. **User-visible cost:** Digests that never went out — Tech & AI, Daily Marketing Job Radar, Political +News, AI Newsletter Curation for Patrick, Gmail Action Digest ×2, FreshRSS Midday ×2, and the weekly Perth +New Restaurant (missed outright; next chance 10-22). The missed audit also meant the 10-14 window went +uncovered. + +**Recovery verified by schedule, not by probe alone.** `1b31187397a5` completed 10-15 **22:01:56** (streak +4 → 0); `b63ed5817212` 10-16 **00:02:29**; `96eb95ce12f2` 10-16 **00:01:36**; `ea79a2107e36` 10-16 +**00:09:48**. Live 10-16 00:1x: `GET /v1/models` on **100.78.63.20:20128 → HTTP 401 in 10 ms** (up, auth +required) and the same on `omniroute:20128` / 100.88.81.19. + +**Non-clean `last_status`: 3 → 12.** Cleared during the window: `b63ed5817212`, `1b31187397a5`, +`ea79a2107e36` (all on successful post-storm runs). Added: the nine rows listed under **Count** above. +Unchanged long-standing: `1b23c042e17d` (10-09 error), `0e753e9720fd` (10-01 error), `066bb5191280` +(10-01 `delivery_failed`, dead Telegram `thread_id 34795`). + +**Mnemosyne.** Both monitors sat on the failing node, so **no coverage figures were refreshed in the +window** — last explicit numbers remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic +1050/1056 = 99.4 %); the 10-16 00:09 Health Monitor run is the first clean read since 10-12. +→ [[systems/mnemosyne]] + +**Stocktake gap (self-healing).** `1a8e90ea2271` failed 10-15 03:00 → `raw/stocktake/freshness-20261015.txt` +was never written and `systems/skills-index.md` still carries the 10-14 stamp. Its next slot is 10-16 03:00, +so the gap closes on its own — **not** regenerated by hand here, to avoid racing the scheduled run. + +**Sessions.** **36 sessions in the two-day window, all `cron`, zero interactive** — fourth consecutive +window with no interactive session. + +**Pages touched:** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` (new), +`systems/omniroute.md`, `systems/mnemosyne.md`, `systems/scheduled-tasks.md` (this block, frontmatter, +Count), `current-state.md`, `index.md`, `log.md`. +**Nothing removed or archived:** no retired job, no obsolete system → `index.md` structure gains one page. + +**Still open after this cycle:** the **no-failover topology** itself (all 18 model-calling jobs on one +OmniRoute node — the durable fix is a second provider pin or a router-side failover, and it needs Anthony's +call); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab +workloads; Quartz rebuild step inert; skills-index off-by-one; **12** non-clean `last_status` rows; +Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.