Files
agent-estate-wiki/systems/omniroute.md
T
Hermes Rhino 95fd7d33f1 audit(2026-10-20): omni-101 503 burst 2 — incident back to active; scheduled-tasks rebuilt from jobs.json
- burst 2 (2026-10-19): 9 model calls across 8 jobs, 159/13 day
- burst-2 table names/schedules/aliases corrected from live jobs.json
- non-clean last_status 7 -> 12; unclosed incident rows 13 (query stated)
- 2026-10-17 tail corrected 162/5 -> 163/5
2026-10-20 00:12:39 +08:00

163 lines
14 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: OmniRoute
type: system
status: active
created: 2026-07-22
updated: 2026-10-20
verified_on: 2026-10-20
confidence: high
tags: [routing, provider, model, inference]
sources: [raw/configs/hermes-config-sanitized.txt]
---
# OmniRoute
## Recent Changes (2026-10-20)
- **Recurring `503 … resource pressure` load-shedding on `omniroute-1` — second burst in four days.** On
**2026-10-19** the node failed **9 model calls across 8 jobs** (06:01, then 07:01 → 12:01, plus 4 derivative
Mnemosyne Watchdog reds) with
`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.`, while
`/api/health` answered `200 {"status":"ok"}` and `/v1/models` answered `401` throughout. Burst 1 ran
**2026-10-16 08:00 → 10-17 03:00** with the identical error string and the identical 9-call blast radius.
Both bursts **self-cleared**; nothing was changed on CT600 or `omniroute-1` in between.
**Root cause unknown — no host telemetry was captured during either burst.**
→ [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **Every failing job is pinned to the same alias on the same node.** All eight read
`provider: custom:omni-101`, `model: smart-route` in `~/.hermes/cron/jobs.json` (verified 2026-10-20).
With no second provider pin and no router-side failover between `omniroute-1` and CT600 `omniroute`, one
node's load-shedding blanks the fleet's digest slots.
- **Probe properly.** A green `/api/health` proves nothing about the model path — it stayed `200 ok` through a
~19 h burst and a second 7 h burst. To catch the next one, run a **real completion** against
`custom:omni-101` while jobs are still failing, and capture the node's CPU / RAM / queue depth *during* the
burst, not afterwards.
## Purpose
Secondary provider/router for Hermes auxiliary roles and some agent routing.
## Current Configuration
- **Hermes provider name:** `custom:omni` (maps to provider `omniroute:20128`)
- **Base URL:** `http://omniroute:20128/v1`
- **Host reachability:** bare name `omniroute` resolves to `100.88.81.19`; direct HTTP on `:20128` answers `401` (auth-required, service live)
- **Runs on:** Proxmox CT600 (`omniroute`, Tailscale `100.88.81.19`, onboot=1) — verified running 2026-10-11
- **DB path:** `/opt/omniroute/storage.sqlite` (was `/var/lib/omniroute/storage.sqlite`)
- **Backup:** pre-migration config saved at `~/.hermes/config.yaml.bak.omniroute-removal`
- **Miss:** Hermes default provider path does not use OmniRoute for main chat; used for auxiliary roles instead
- **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp`
- **Key prefix visible in config:** `sk-5c67fbdc0...` — full value not stored in wiki
## CT600 / CT601 / omniroute-1 — actual state (2026-10-11)
The wiki previously recorded CT600 as retired and CT601 as the live successor. **That is no longer true.** Live PVE and Tailscale checks return:
| Node | Status (live 2026-10-11) | Tailscale IP | Notes |
|---|---|---|---|
| CT600 `omniroute` | **running**, onboot=1 | `100.88.81.19` | OmniRoute listening on `0.0.0.0:20128`; `/v1/models` returns `401` (alive, auth required) |
| CT601 `omni` | **stopped**, onboot=0 | `100.93.204.4` | Tailscale peer last seen 54d ago |
| `omniroute-1` | **active** (not in PVE `pct list`) | `100.78.63.20` | Third OmniRoute node; `/v1/models` returns `401`. Registered in Hermes config as custom provider `omni-101`, which carries `skills_hub` (model `free-stack`, 5384 models, default `smart-route`) |
- Hermes config resolving to the omniroute hostname (`custom:omni` → `omniroute:20128`) therefore points at **CT600**, not CT601.
- CT600's combo API is reachable with its local key and returns combos (e.g. `go-pro`), confirming it is the serving instance.
- **Do not treat CT601 as the production OmniRoute node.** It is stopped with `onboot=0`.
- Treat `omniroute-1` (`100.78.63.20`) as a second live instance used by the `omni-101` provider. It is not visible in `pct list`, so it is likely a VM or off-PVE host; its identity was not confirmed.
## Catalog and Routing Facts
- Catalog reachable: yes, 47 combos returned at `/api/combos`
- Combo schema observed: `name`, `strategy`, `models`, `id`; some combos have `config`, `isHidden`, `uuid`
- Combo-level context can override individual models
- `auto`/`lkgp` strategies may silently change chosen models
- Model mappings can substitute models silently
- Session affinity caches last successful provider in `session_model_history`
## Catalog Evidence (2026-07-22)
- Advertised context for `gemini/gemini-3.6-flash`: `1048576`
- Related models in catalog: `kc/google/gemini-3.6-flash`, `kilocode/google/gemini-3.6-flash`, `nous/google/gemini-3.6-flash`, `opencode-zen/gemini-3.6-flash`, `vertex/gemini-3.6-flash`
- No live combo in the returned 47 combos referenced `gemini-3.6-flash`
## Known Issues
- opencode-zen proxy pitfall documented in omniroute-ops skill
- Web search proxy failures documented in omniroute-ops skill
- Combo discovery can differ from resolution;
## AutoRoute variants (2026-08-12)
- `auto/best-free` (Auto Cheap) and `auto/coding-free` are **virtual** AutoRoute variants — they do NOT appear in the persisted combo list (`/api/combos` returns 54 combos, none named these).
- The LKGP engine resolves them dynamically: it scores the operator's **live connected provider accounts** (e.g. `antigravity`, `opencode-zen`, `nvidia`, `openrouter`) rather than a hardcoded candidate list.
- Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
- Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
## Recent Changes (2026-10-18) — `omniroute-1` 503 resource pressure (2026-10-16 -> 2026-10-17)
- **A second, different failure mode from the same node.** Three days after the 34-hour TCP blackout,
`omniroute-1` answered model calls with **`HTTP 503: Service temporarily unavailable due to resource
pressure. Retry shortly.`** — **9 model calls across 7 jobs**, between **10-16 08:00:37** and
**10-17 03:00:28** (~19 h). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **The control plane looked healthy throughout.** `/api/health` returned `200 {"status":"ok"}` and
`/v1/models` returned `401` (auth-required) before, during and after the window — re-probed 2026-10-18
00:33. A green health endpoint does **not** mean the model path is serving: 503 here is load-shedding
under resource pressure, not a dead process, so process checks miss it.
- **It recovered on its own; nothing was changed on the node.** Last 503 **10-17 03:00:28**; `10-17`
closed **162 completed / 5 failed** (all five before 03:01) and `10-18` is green so far. Every affected
job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01, `08eb86f02d37` 10-17 08:31,
`ecc1fc13655a` and `ea79a2107e36` both 10-17 12:02, auto-closing their own incidents.
## Recent Changes (2026-10-16) — `omniroute-1` connection-refused storm
- **`omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) was TCP-unreachable from 2026-10-14
08:30 to 2026-10-15 ~18:07 AWST** — `httpcore.ConnectError: [Errno 111] Connection refused` →
`openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` Result: **24 genuine
cron model-call failures across 12 jobs** plus 12 derivative watchdog rows, and **every scheduled digest
for ~34 hours was silently suppressed** (including the 10-14 23:59 Wiki Audit).
- **This node is a single point of failure for the whole scheduled estate.** All **18** enabled
`agent`-mode cron jobs are pinned to `custom:omni-101` (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2). The only other model-capable cron pin — **`custom:omni` (CT600,
`omniroute:20128`)** — is attached to a **`no_agent`** job (`e530c89f2be0`, Evening Stand-Down), so it is
**never exercised by a model call**; `d40085da631c`, the only other `custom:omni` pin, is **disabled**.
The cron fleet therefore has **no model-call failover between the two nodes**, and no job that can even
tell you whether the second node is up.
- **Recovered — verified 2026-10-16 00:1x:** `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in
10 ms** (up, auth required); same on `omniroute:20128` / `100.88.81.19`. Four jobs have completed on the
node since 2026-10-15 22:01.
- Full write-up: [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
## Recent Changes (2026-08-14)
- **OmniRoute factory reset on CT601.** Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (`auto/best-fast`, `auto/best-coding`, etc.). Old state backed up at `/opt/omniroute/factory-reset-backup-20260814-225524`.
- **Providers re-added from Notion API vault:** Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry.
- **Hermes config default model changed:** `auto/best-free` → `resilient-free` (via `~/.hermes/config.yaml`). `auto/best-free` was misconfigured — had no combo definition and routed to `gpt-5.6-sol` (paid) instead of free models. Switched to `resilient-free` which is a working free combo.
- **Multiple free combos broken by upstream model ID changes:**
- OpenRouter: `poolside/laguna-s-2-1:free` (dash) → `poolside/laguna-s-2.1:free` (dot) — model ID format changed
- Groq: `meta-llama/llama-4-scout-17b-16e-instruct` — **retired**, only llama-3.3/3.1 remain
- Affected combos: `resilient-free`, `free-IA`, `free-stack`, `good-free`, `deepseek-flash-free`, `mimo-free`
- All route through OpenRouter → fail with 400/404 when trying these model IDs
- **Working free combos as of 2026-08-14:** `free-lkgp` (DeepSeek V4 Flash), `mega-free` (StepFun 3.7 Flash), `smart-route` / `smart-route-ia` (nested combos)
- **LiteLLM routing issue identified:** Hermes was previously pointing at `http://litellm:4000/v1` with `free-auto` model, causing 6–26s per call. OmniRoute's `auto/best-free` tested at 265ms. Config now points at OmniRoute.
- **GPT-5.6 Luna research:** Community consensus — Luna is cleaner/more polished; DeepSeek V4 Flash is more persistent/cheaper. Luna better for visual/UI work and final review; DeepSeek better for automation and high-volume executor work. Provider route matters more than model name.
## Recent Changes (2026-08-13)
- **OmniRoute migrated CT600 → CT601.** Provider renamed `custom:omniroute` → `custom:omni`; base URL `http://omniroute:20128/v1` → `http://omni:20128/v1`. Host `omni.kangaroo-eel.ts.net` (100.93.204.4, active) replaces `omniroute.kangaroo-eel.ts.net` (100.88.81.19, offline). CT600 stopped, CT601 running. Database path `/var/lib/omniroute/storage.sqlite` → `/opt/omniroute/storage.sqlite`. API key unchanged. All 14 config + 22 cron-job `custom:omniroute` references updated to `custom:omni`. Pre-migration backup at `~/.hermes/config.yaml.bak.omniroute-removal`. Verified live 2026-08-13 (`pct list`: 600 stopped, 601 running; `grep`: 0 remaining `custom:omniroute` refs).
## Recent Changes (2026-07-25)
- **free-stack combo updated:** Nemotron 3 Ultra 550B model at index 4 now configured with NVIDIA-specific parameters:
- `max_tokens`: 8192 (was default 65536, exceeding NVIDIA NIM limit of 16384)
- `reasoning_effort`: medium (was high, reducing long internal monologues)
- `chat_template_kwargs.force_nonempty_content`: true (prevents empty responses when tool calling + reasoning)
- **Combo ID:** `23d25f8f-5eb0-4b6f-b966-e260ca3905b3`
- **Hermes config.yaml updated:** `agent.reasoning_effort: medium`, `agent.max_tokens: 8192`, `agent.chat_template_kwargs.force_nonempty_content: true`
- **Rationale:** NVIDIA NIM API documents max output of 16384 tokens; medium reasoning uses more efficient chat-template mode; force_nonempty_content resolves empty responses with tool calls combo in picker may still route
## Recent Changes (2026-08-07)
- **Three-layer root cause for slowness identified and fixed:**
1. **npm package .env override:** OmniRoute loads three .env files (`/var/lib/omniroute/.env`, `/root/.omniroute/.env`, `/usr/lib/node_modules/omniroute/.env`). The npm package's `.env` ships with `ENABLE_SOCKS5_PROXY=true` and `npm update` re-installs it with that default, silently overriding the `/var/lib` config. Fix: flip all three to `false`, then restart. After ANY upgrade, check all three.
2. **Dead provider connection:** Connection `58debd59` (prefix `nanobot`) points at `clawtest:8900/v1` — nanobot on clawtest is DOWN. Every model sync + credential health check retried it, burning event-loop time. Disabled it.
3. **PVE host load leaking into CT600:** CT600 has `nesting=1`, so `uptime`/`top` inside the container mirror PVE's host load exactly (verified: identical 10.30/9.23/10.37 on both). Same trap as CT241. Judge CT600 CPU from `pct exec 600 -- ps -o %cpu` or host-side `ps`, never from in-container load average.
- **First-request-after-cache-expiry stall:** `/v1/models` hangs 10-15s+ on the FIRST hit after the model catalog cache expires (rebuild enumerates all connections, including slow dead ones), then answers in ~0.02s from cache. Don't judge health on one curl — run a burst of 5 and check if later tries are sub-second.
- **`is_active=0` does NOT stop CredentialHealth probes:** Disabling a dead connection stops it from ModelSync but the health checker still probes it and logs `[ProxyFetch] ECONNREFUSED` noise. That noise is benign once the endpoint is unreachable.
- **Lessons banked in `omniroute-ops` skill:** `references/omniroute-upgrade-procedure.md` updated with all three pitfalls.
## Recent Changes (2026-08-04)
- **OmniRoute free-DeepSeek silent-stop mitigation:** Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances. Reworked both combos: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort. See [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]].
## Related
- [[model-providers]]
- [[model-routing]]