audit(2026-10-20): omni-101 503 burst 2 — incident back to active; scheduled-tasks rebuilt from jobs.json
- burst 2 (2026-10-19): 9 model calls across 8 jobs, 159/13 day - burst-2 table names/schedules/aliases corrected from live jobs.json - non-clean last_status 7 -> 12; unclosed incident rows 13 (query stated) - 2026-10-17 tail corrected 162/5 -> 163/5
This commit is contained in:
+22
-2
@@ -3,8 +3,8 @@ title: OmniRoute
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-18
|
||||
verified_on: 2026-10-18
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
confidence: high
|
||||
tags: [routing, provider, model, inference]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -12,6 +12,26 @@ sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
|
||||
# OmniRoute
|
||||
|
||||
## Recent Changes (2026-10-20)
|
||||
|
||||
- **Recurring `503 … resource pressure` load-shedding on `omniroute-1` — second burst in four days.** On
|
||||
**2026-10-19** the node failed **9 model calls across 8 jobs** (06:01, then 07:01 → 12:01, plus 4 derivative
|
||||
Mnemosyne Watchdog reds) with
|
||||
`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.`, while
|
||||
`/api/health` answered `200 {"status":"ok"}` and `/v1/models` answered `401` throughout. Burst 1 ran
|
||||
**2026-10-16 08:00 → 10-17 03:00** with the identical error string and the identical 9-call blast radius.
|
||||
Both bursts **self-cleared**; nothing was changed on CT600 or `omniroute-1` in between.
|
||||
**Root cause unknown — no host telemetry was captured during either burst.**
|
||||
→ [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
- **Every failing job is pinned to the same alias on the same node.** All eight read
|
||||
`provider: custom:omni-101`, `model: smart-route` in `~/.hermes/cron/jobs.json` (verified 2026-10-20).
|
||||
With no second provider pin and no router-side failover between `omniroute-1` and CT600 `omniroute`, one
|
||||
node's load-shedding blanks the fleet's digest slots.
|
||||
- **Probe properly.** A green `/api/health` proves nothing about the model path — it stayed `200 ok` through a
|
||||
~19 h burst and a second 7 h burst. To catch the next one, run a **real completion** against
|
||||
`custom:omni-101` while jobs are still failing, and capture the node's CPU / RAM / queue depth *during* the
|
||||
burst, not afterwards.
|
||||
|
||||
## Purpose
|
||||
Secondary provider/router for Hermes auxiliary roles and some agent routing.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user