audit(2026-10-20): omni-101 503 burst 2 — incident back to active; scheduled-tasks rebuilt from jobs.json
- burst 2 (2026-10-19): 9 model calls across 8 jobs, 159/13 day - burst-2 table names/schedules/aliases corrected from live jobs.json - non-clean last_status 7 -> 12; unclosed incident rows 13 (query stated) - 2026-10-17 tail corrected 162/5 -> 163/5
This commit is contained in:
+31
-4
@@ -3,16 +3,43 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-19
|
||||
verified_on: 2026-10-19
|
||||
review_after: 2026-10-25
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
review_after: 2026-10-27
|
||||
confidence: high
|
||||
tags: [system]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-19
|
||||
Updated: 2026-10-20
|
||||
|
||||
## Recent Changes (2026-10-20, nightly audit; live probes 2026-10-20 00:03-00:12 AWST)
|
||||
|
||||
- **The `omni-101` 503 resource-pressure fault returned 44 hours after the incident page was downgraded to
|
||||
`monitoring` — same node, same error string, same blast radius.** 2026-10-19 closed **159 completed / 13
|
||||
failed** against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all read
|
||||
`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on
|
||||
`custom:omni-101`, in two clusters — 06:01 (one job) and 07:01 → 12:01 (seven more runs, seven jobs). The
|
||||
remaining 4 failed rows are the Mnemosyne Watchdog quoting the coverage monitor's error: the same derivative
|
||||
double-count as burst 1. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] (status back to **`active`**)
|
||||
- **Single pin, verified this time.** Every job that failed in burst 2 reads `provider: custom:omni-101`,
|
||||
`model: smart-route` in `jobs.json` as of 2026-10-20 — there is no alias diversity left to fail over to. ⚠ The
|
||||
burst-1 note recording a wider mix (`deepseek-paid-smart`, `free-IA`, `dark`) no longer matches `jobs.json`;
|
||||
it needs confirming with Anthony before either reading is relied on.
|
||||
- **The node stayed green through all of it.** `omniroute-1` (`100.78.63.20:20128`) answered
|
||||
`/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` at 2026-10-20 00:03, before, during and after
|
||||
the burst. **Reachability probes cannot see this failure class** — only job-level error text can.
|
||||
- **`c0a9fa314542` (GitHub Trending Radar) lost its scheduled Monday slot** (`0 9 * * 1`, 09:01) to the 503, so
|
||||
this week's roundup never ran and its revoked-token status is **untested for a second consecutive Monday**.
|
||||
Its next real test is Mon **2026-10-26 09:00**.
|
||||
- **Non-clean `last_status` moved 7 → 12** (10 `error`, 2 `delivery_failed`) as of 2026-10-20 00:09. Newly
|
||||
dirtied: `dca8482e4f76`, `85edf6772bca`, `08eb86f02d37`, `c0a9fa314542`; deepened: `0e519874d280` and
|
||||
`ecc1fc13655a` (both streak 2). `fad5a86979d4` added a `delivery_failed` (bare-`telegram` target, thread
|
||||
34795 not found — no incident row created for it). `b63ed5817212`, `96eb95ce12f2` and `ea79a2107e36` all
|
||||
cleared on their next slots. **Unclosed incident rows: 13** (12 `alerted` / 1 `detected`). → [[systems/scheduled-tasks]]
|
||||
- **Unchanged otherwise:** fleet still **28 jobs / 25 enabled / 3 disabled**; CT255 Mattermost
|
||||
(`100.96.100.255:8065`) still unreachable; **15 sessions in the window, all `cron` — no interactive sessions.**
|
||||
|
||||
## Recent Changes (2026-10-19, nightly audit; live probes 2026-10-19 00:04-00:2x AWST)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user