audit(2026-10-20): omni-101 503 burst 2 — incident back to active; scheduled-tasks rebuilt from jobs.json

- burst 2 (2026-10-19): 9 model calls across 8 jobs, 159/13 day
- burst-2 table names/schedules/aliases corrected from live jobs.json
- non-clean last_status 7 -> 12; unclosed incident rows 13 (query stated)
- 2026-10-17 tail corrected 162/5 -> 163/5
This commit is contained in:
Hermes Rhino
2026-10-20 00:12:39 +08:00
parent 93c55efc94
commit 95fd7d33f1
6 changed files with 195 additions and 40 deletions
+31 -4
View File
@@ -3,16 +3,43 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-10-19
verified_on: 2026-10-19
review_after: 2026-10-25
updated: 2026-10-20
verified_on: 2026-10-20
review_after: 2026-10-27
confidence: high
tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
---
# Current State
Updated: 2026-10-19
Updated: 2026-10-20
## Recent Changes (2026-10-20, nightly audit; live probes 2026-10-20 00:03-00:12 AWST)
- **The `omni-101` 503 resource-pressure fault returned 44 hours after the incident page was downgraded to
`monitoring` — same node, same error string, same blast radius.** 2026-10-19 closed **159 completed / 13
failed** against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all read
`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on
`custom:omni-101`, in two clusters — 06:01 (one job) and 07:01 → 12:01 (seven more runs, seven jobs). The
remaining 4 failed rows are the Mnemosyne Watchdog quoting the coverage monitor's error: the same derivative
double-count as burst 1. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] (status back to **`active`**)
- **Single pin, verified this time.** Every job that failed in burst 2 reads `provider: custom:omni-101`,
`model: smart-route` in `jobs.json` as of 2026-10-20 — there is no alias diversity left to fail over to. ⚠ The
burst-1 note recording a wider mix (`deepseek-paid-smart`, `free-IA`, `dark`) no longer matches `jobs.json`;
it needs confirming with Anthony before either reading is relied on.
- **The node stayed green through all of it.** `omniroute-1` (`100.78.63.20:20128`) answered
`/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` at 2026-10-20 00:03, before, during and after
the burst. **Reachability probes cannot see this failure class** — only job-level error text can.
- **`c0a9fa314542` (GitHub Trending Radar) lost its scheduled Monday slot** (`0 9 * * 1`, 09:01) to the 503, so
this week's roundup never ran and its revoked-token status is **untested for a second consecutive Monday**.
Its next real test is Mon **2026-10-26 09:00**.
- **Non-clean `last_status` moved 7 → 12** (10 `error`, 2 `delivery_failed`) as of 2026-10-20 00:09. Newly
dirtied: `dca8482e4f76`, `85edf6772bca`, `08eb86f02d37`, `c0a9fa314542`; deepened: `0e519874d280` and
`ecc1fc13655a` (both streak 2). `fad5a86979d4` added a `delivery_failed` (bare-`telegram` target, thread
34795 not found — no incident row created for it). `b63ed5817212`, `96eb95ce12f2` and `ea79a2107e36` all
cleared on their next slots. **Unclosed incident rows: 13** (12 `alerted` / 1 `detected`). → [[systems/scheduled-tasks]]
- **Unchanged otherwise:** fleet still **28 jobs / 25 enabled / 3 disabled**; CT255 Mattermost
(`100.96.100.255:8065`) still unreachable; **15 sessions in the window, all `cron` — no interactive sessions.**
## Recent Changes (2026-10-19, nightly audit; live probes 2026-10-19 00:04-00:2x AWST)