audit(2026-10-20): omni-101 503 burst 2 — incident back to active; scheduled-tasks rebuilt from jobs.json

- burst 2 (2026-10-19): 9 model calls across 8 jobs, 159/13 day
- burst-2 table names/schedules/aliases corrected from live jobs.json
- non-clean last_status 7 -> 12; unclosed incident rows 13 (query stated)
- 2026-10-17 tail corrected 162/5 -> 163/5
This commit is contained in:
Hermes Rhino
2026-10-20 00:12:39 +08:00
parent 93c55efc94
commit 95fd7d33f1
6 changed files with 195 additions and 40 deletions
+22 -2
View File
@@ -3,8 +3,8 @@ title: OmniRoute
type: system
status: active
created: 2026-07-22
updated: 2026-10-18
verified_on: 2026-10-18
updated: 2026-10-20
verified_on: 2026-10-20
confidence: high
tags: [routing, provider, model, inference]
sources: [raw/configs/hermes-config-sanitized.txt]
@@ -12,6 +12,26 @@ sources: [raw/configs/hermes-config-sanitized.txt]
# OmniRoute
## Recent Changes (2026-10-20)
- **Recurring `503 … resource pressure` load-shedding on `omniroute-1` — second burst in four days.** On
**2026-10-19** the node failed **9 model calls across 8 jobs** (06:01, then 07:01 → 12:01, plus 4 derivative
Mnemosyne Watchdog reds) with
`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.`, while
`/api/health` answered `200 {"status":"ok"}` and `/v1/models` answered `401` throughout. Burst 1 ran
**2026-10-16 08:00 → 10-17 03:00** with the identical error string and the identical 9-call blast radius.
Both bursts **self-cleared**; nothing was changed on CT600 or `omniroute-1` in between.
**Root cause unknown — no host telemetry was captured during either burst.**
→ [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **Every failing job is pinned to the same alias on the same node.** All eight read
`provider: custom:omni-101`, `model: smart-route` in `~/.hermes/cron/jobs.json` (verified 2026-10-20).
With no second provider pin and no router-side failover between `omniroute-1` and CT600 `omniroute`, one
node's load-shedding blanks the fleet's digest slots.
- **Probe properly.** A green `/api/health` proves nothing about the model path — it stayed `200 ok` through a
~19 h burst and a second 7 h burst. To catch the next one, run a **real completion** against
`custom:omni-101` while jobs are still failing, and capture the node's CPU / RAM / queue depth *during* the
burst, not afterwards.
## Purpose
Secondary provider/router for Hermes auxiliary roles and some agent routing.