audit 2026-10-18: new incident page (omniroute-1 503 resource pressure, 9 calls / 7 jobs), fleet re-verified 28 jobs / 25 enabled / 8 non-clean, Mosquito Tracker removed, radar bot token dead
This commit is contained in:
+17
-2
@@ -3,8 +3,8 @@ title: OmniRoute
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-16
|
||||
verified_on: 2026-10-16
|
||||
updated: 2026-10-18
|
||||
verified_on: 2026-10-18
|
||||
confidence: high
|
||||
tags: [routing, provider, model, inference]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -66,6 +66,21 @@ The wiki previously recorded CT600 as retired and CT601 as the live successor. *
|
||||
- Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
|
||||
- Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
|
||||
|
||||
## Recent Changes (2026-10-18) — `omniroute-1` 503 resource pressure (2026-10-16 -> 2026-10-17)
|
||||
|
||||
- **A second, different failure mode from the same node.** Three days after the 34-hour TCP blackout,
|
||||
`omniroute-1` answered model calls with **`HTTP 503: Service temporarily unavailable due to resource
|
||||
pressure. Retry shortly.`** — **9 model calls across 7 jobs**, between **10-16 08:00:37** and
|
||||
**10-17 03:00:28** (~19 h). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
- **The control plane looked healthy throughout.** `/api/health` returned `200 {"status":"ok"}` and
|
||||
`/v1/models` returned `401` (auth-required) before, during and after the window — re-probed 2026-10-18
|
||||
00:33. A green health endpoint does **not** mean the model path is serving: 503 here is load-shedding
|
||||
under resource pressure, not a dead process, so process checks miss it.
|
||||
- **It recovered on its own; nothing was changed on the node.** Last 503 **10-17 03:00:28**; `10-17`
|
||||
closed **162 completed / 5 failed** (all five before 03:01) and `10-18` is green so far. Every affected
|
||||
job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01, `08eb86f02d37` 10-17 08:31,
|
||||
`ecc1fc13655a` and `ea79a2107e36` both 10-17 12:02, auto-closing their own incidents.
|
||||
|
||||
## Recent Changes (2026-10-16) — `omniroute-1` connection-refused storm
|
||||
|
||||
- **`omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) was TCP-unreachable from 2026-10-14
|
||||
|
||||
Reference in New Issue
Block a user