audit 2026-10-18: new incident page (omniroute-1 503 resource pressure, 9 calls / 7 jobs), fleet re-verified 28 jobs / 25 enabled / 8 non-clean, Mosquito Tracker removed, radar bot token dead

This commit is contained in:
hermes-audit
2026-10-18 00:41:38 +08:00
parent 76de919e6a
commit 4137dd6945
7 changed files with 276 additions and 23 deletions
+17 -2
View File
@@ -3,8 +3,8 @@ title: OmniRoute
type: system
status: active
created: 2026-07-22
updated: 2026-10-16
verified_on: 2026-10-16
updated: 2026-10-18
verified_on: 2026-10-18
confidence: high
tags: [routing, provider, model, inference]
sources: [raw/configs/hermes-config-sanitized.txt]
@@ -66,6 +66,21 @@ The wiki previously recorded CT600 as retired and CT601 as the live successor. *
- Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
- Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
## Recent Changes (2026-10-18) — `omniroute-1` 503 resource pressure (2026-10-16 -> 2026-10-17)
- **A second, different failure mode from the same node.** Three days after the 34-hour TCP blackout,
`omniroute-1` answered model calls with **`HTTP 503: Service temporarily unavailable due to resource
pressure. Retry shortly.`** — **9 model calls across 7 jobs**, between **10-16 08:00:37** and
**10-17 03:00:28** (~19 h). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **The control plane looked healthy throughout.** `/api/health` returned `200 {"status":"ok"}` and
`/v1/models` returned `401` (auth-required) before, during and after the window — re-probed 2026-10-18
00:33. A green health endpoint does **not** mean the model path is serving: 503 here is load-shedding
under resource pressure, not a dead process, so process checks miss it.
- **It recovered on its own; nothing was changed on the node.** Last 503 **10-17 03:00:28**; `10-17`
closed **162 completed / 5 failed** (all five before 03:01) and `10-18` is green so far. Every affected
job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01, `08eb86f02d37` 10-17 08:31,
`ecc1fc13655a` and `ea79a2107e36` both 10-17 12:02, auto-closing their own incidents.
## Recent Changes (2026-10-16) — `omniroute-1` connection-refused storm
- **`omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) was TCP-unreachable from 2026-10-14