--- title: OmniRoute type: system status: active created: 2026-07-22 updated: 2026-11-10 verified_on: 2026-11-10 confidence: high tags: [routing, provider, model, inference] sources: [raw/configs/hermes-config-sanitized.txt] --- # OmniRoute ## Recent Changes (2026-10-21) - **Third failure mode on the same node, and it is not the 503 string.** `dca8482e4f76` (Daily Marketing Job Radar) failed on 2026-10-21 at 07:40:48 after 40 m 36 s with `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s`. **Zero `HTTP 503 … resource pressure` rows exist for that day** (last: 2026-10-19 09:02) and the sibling 07:00 job `85edf6772bca` completed normally, so the node was serving work at that hour. → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] - **One pin, three signatures in eight days:** connection-refused storm (10-14/15), resource-pressure 503s (10-16/17 and 10-19), pre-dispatch filter skip (10-21) — all on `custom:omni-101`, all with `model: smart-route`. No job in the fleet has a second provider pin. - **Router-side telemetry is still missing for every one of them** — `omni-101` host logs have not been captured during a failure, which is why none of the three has a confirmed root cause. ## Recent Changes (2026-10-20) - **Recurring `503 … resource pressure` load-shedding on `omniroute-1` — second burst in four days.** On **2026-10-19** the node failed **9 model calls across 8 jobs** (06:01, then 07:01 → 12:01, plus 4 derivative Mnemosyne Watchdog reds) with `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.`, while `/api/health` answered `200 {"status":"ok"}` and `/v1/models` answered `401` throughout. Burst 1 ran **2026-10-16 08:00 → 10-17 03:00** with the identical error string and the identical 9-call blast radius. Both bursts **self-cleared**; nothing was changed on CT600 or `omniroute-1` in between. **Root cause unknown — no host telemetry was captured during either burst.** → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **Every failing job is pinned to the same alias on the same node.** All eight read `provider: custom:omni-101`, `model: smart-route` in `~/.hermes/cron/jobs.json` (verified 2026-10-20). With no second provider pin and no router-side failover between `omniroute-1` and CT600 `omniroute`, one node's load-shedding blanks the fleet's digest slots. - **Probe properly.** A green `/api/health` proves nothing about the model path — it stayed `200 ok` through a ~19 h burst and a second 7 h burst. To catch the next one, run a **real completion** against `custom:omni-101` while jobs are still failing, and capture the node's CPU / RAM / queue depth *during* the burst, not afterwards. ## Purpose Provider/router for Hermes — ⚠ **updated 2026-10-26: the `omniroute-1` node is the DEFAULT chat path again.** Hermes's default provider is `custom:omni-101` → `http://100.78.63.20:20128` with model `smart-route` (reverted 2026-10-24 19:06; the 2026-10-20 → 10-24 window pointed the default at `nous`). The CT600 provider key `custom:omni` (`http://omniroute:20128/v1`) remains auxiliary-only. → [[systems/current-profile]] ## Current Configuration - **Hermes default provider:** `custom:omni-101` → base URL `http://100.78.63.20:20128` (CT101, `m93p`), model default **`smart-route`** — **re-verified live 2026-10-31** from `~/.hermes/config.yaml` (`model.default: smart-route`, `model.provider: custom:omni-101`, `model.base_url: http://100.78.63.20:20128`) - **Secondary provider entry:** `custom:Omniroute:20128` → base URL `http://omniroute:20128/v1` (CT600, `pve`), referenced by the `moa` reference-model set (`vertex/gemini-3.6-flash`, `kilo-gateway/kilo-auto/free`, `nvidia/deepseek-ai/deepseek-v4-pro`, `freeapi/nemotron-3-ultra-550b`) — auxiliary/MoA use, **not** the main chat default - **Host reachability:** bare name `omniroute` resolves to `100.88.81.19`; direct HTTP on `:20128` answers `401` (auth-required, service live) - **Runs on:** two live nodes — CT600 (`omniroute`, Tailscale `100.88.81.19`, on `pve`) and CT101 (`omniroute`, Tailscale `100.78.63.20`, on `m93p`) — **both verified running and listening on `0.0.0.0:20128` on 2026-10-31** - **DB path:** `/opt/omniroute/storage.sqlite` (was `/var/lib/omniroute/storage.sqlite`) - **Backup:** pre-migration config saved at `~/.hermes/config.yaml.bak.omniroute-removal` - **Correction (2026-10-26):** the earlier *"Hermes default provider path does not use OmniRoute for main chat"* note is **stale** — the default is OmniRoute again (`custom:omni-101`, model `smart-route`). The **CT600** key `custom:Omniroute:20128` is still auxiliary-only (MoA reference models). - **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp` — all on `custom:omni-101` with model `no-think/freeapi/auto` (live-verified 2026-10-31) - **Key prefix visible in config:** `sk-5c67fbdc0...` — full value not stored in wiki ## CT600 / CT601 / omniroute-1 — actual state (re-verified 2026-10-31) The wiki previously recorded CT600 as retired and CT601 as the live successor, then CT601 as stopped. **All three framings are now superseded.** Live `pct list` on **both** hypervisors and live HTTP probes return: | Node | Status (live 2026-10-31) | Tailscale IP | Notes | |---|---|---|---| | CT600 `omniroute` | **running** | `100.88.81.19` | On `pve`. OmniRoute listening on `0.0.0.0:20128`; `/v1/models` and `/api/combos` both return `401` (alive, auth required) | | CT101 `omniroute-1` | **running** | `100.78.63.20` | On `m93p`. Live `pct config 101` → `hostname: omniroute`; `omniroute` listening on `0.0.0.0:20128`; `/v1/models` and `/api/combos` both return `401`. Registered in Hermes config as `custom:omni-101` — **the live default provider** | | CT601 `omni` | **does not exist** | — | `/etc/pve/lxc/601.conf` absent on **both** `pve` and `m93p`. The `omni` peer (`100.93.204.4`) has **expired out of the tailnet entirely** — live `tailscale status --json` has no peer named `omni` and no peer at that IP | - Hermes config resolving to the omniroute hostname (`custom:Omniroute:20128` → `omniroute:20128`) therefore points at **CT600**; the live default `custom:omni-101` points at **CT101**. - **CT601 was deleted, not merely stopped** — the earlier *"stopped, onboot=0"* entry was wrong. - **The `omniroute-1` identity is now confirmed** — it is **CT101 on `m93p`**, not "a VM or off-PVE host". The 2026-10-11 note that it was "not visible in `pct list`" was an artifact of probing only `pve`. - **Two live OmniRoute nodes remain** (CT600, CT101), both serving `:20128` with separate credentials — a pin's usability must be probed on the node that actually serves it. ## Catalog and Routing Facts - Catalog reachable: yes, 47 combos returned at `/api/combos` - Combo schema observed: `name`, `strategy`, `models`, `id`; some combos have `config`, `isHidden`, `uuid` - Combo-level context can override individual models - `auto`/`lkgp` strategies may silently change chosen models - Model mappings can substitute models silently - Session affinity caches last successful provider in `session_model_history` ## Catalog Evidence (2026-07-22) - Advertised context for `gemini/gemini-3.6-flash`: `1048576` - Related models in catalog: `kc/google/gemini-3.6-flash`, `kilocode/google/gemini-3.6-flash`, `nous/google/gemini-3.6-flash`, `opencode-zen/gemini-3.6-flash`, `vertex/gemini-3.6-flash` - No live combo in the returned 47 combos referenced `gemini-3.6-flash` ## Known Issues - opencode-zen proxy pitfall documented in omniroute-ops skill - Web search proxy failures documented in omniroute-ops skill - Combo discovery can differ from resolution; ## AutoRoute variants (2026-08-12) - `auto/best-free` (Auto Cheap) and `auto/coding-free` are **virtual** AutoRoute variants — they do NOT appear in the persisted combo list (`/api/combos` returns 54 combos, none named these). - **Live re-read 2026-11-10 00:4x AWST (nightly audit):** `omniroute-1` (`100.78.63.20:20128`) `/v1/combos` → **HTTP 200, 55 combos**, `free-stack` present — one ahead of the last reading above. `/v1/models` on **both** nodes and CT600's `/v1/combos` return **HTTP 401** (endpoint live, auth required), so the **6,480-model catalog figure dated 2026-10-26 stands unverified this window and is unchanged.** - The LKGP engine resolves them dynamically: it scores the operator's **live connected provider accounts** (e.g. `antigravity`, `opencode-zen`, `nvidia`, `openrouter`) rather than a hardcoded candidate list. - Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request. - Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map. ## Recent Changes (2026-10-18) — `omniroute-1` 503 resource pressure (2026-10-16 -> 2026-10-17) - **A second, different failure mode from the same node.** Three days after the 34-hour TCP blackout, `omniroute-1` answered model calls with **`HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.`** — **9 model calls across 7 jobs**, between **10-16 08:00:37** and **10-17 03:00:28** (~19 h). → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **The control plane looked healthy throughout.** `/api/health` returned `200 {"status":"ok"}` and `/v1/models` returned `401` (auth-required) before, during and after the window — re-probed 2026-10-18 00:33. A green health endpoint does **not** mean the model path is serving: 503 here is load-shedding under resource pressure, not a dead process, so process checks miss it. - **It recovered on its own; nothing was changed on the node.** Last 503 **10-17 03:00:28**; `10-17` closed **162 completed / 5 failed** (all five before 03:01) and `10-18` is green so far. Every affected job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01, `08eb86f02d37` 10-17 08:31, `ecc1fc13655a` and `ea79a2107e36` both 10-17 12:02, auto-closing their own incidents. ## Recent Changes (2026-10-16) — `omniroute-1` connection-refused storm - **`omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) was TCP-unreachable from 2026-10-14 08:30 to 2026-10-15 ~18:07 AWST** — `httpcore.ConnectError: [Errno 111] Connection refused` → `openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` Result: **24 genuine cron model-call failures across 12 jobs** plus 12 derivative watchdog rows, and **every scheduled digest for ~34 hours was silently suppressed** (including the 10-14 23:59 Wiki Audit). - **This node is a single point of failure for the whole scheduled estate.** All **18** enabled `agent`-mode cron jobs are pinned to `custom:omni-101` (`smart-route` ×10, `free-stack` ×6, `deepseek-paid-smart` ×2). The only other model-capable cron pin — **`custom:omni` (CT600, `omniroute:20128`)** — is attached to a **`no_agent`** job (`e530c89f2be0`, Evening Stand-Down), so it is **never exercised by a model call**; `d40085da631c`, the only other `custom:omni` pin, is **disabled**. The cron fleet therefore has **no model-call failover between the two nodes**, and no job that can even tell you whether the second node is up. - **Recovered — verified 2026-10-16 00:1x:** `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in 10 ms** (up, auth required); same on `omniroute:20128` / `100.88.81.19`. Four jobs have completed on the node since 2026-10-15 22:01. - Full write-up: [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] ## Recent Changes (2026-08-14) - **OmniRoute factory reset on CT601.** Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (`auto/best-fast`, `auto/best-coding`, etc.). Old state backed up at `/opt/omniroute/factory-reset-backup-20260814-225524`. - **Providers re-added from Notion API vault:** Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry. - **Hermes config default model changed:** `auto/best-free` → `resilient-free` (via `~/.hermes/config.yaml`). `auto/best-free` was misconfigured — had no combo definition and routed to `gpt-5.6-sol` (paid) instead of free models. Switched to `resilient-free` which is a working free combo. - **Multiple free combos broken by upstream model ID changes:** - OpenRouter: `poolside/laguna-s-2-1:free` (dash) → `poolside/laguna-s-2.1:free` (dot) — model ID format changed - Groq: `meta-llama/llama-4-scout-17b-16e-instruct` — **retired**, only llama-3.3/3.1 remain - Affected combos: `resilient-free`, `free-IA`, `free-stack`, `good-free`, `deepseek-flash-free`, `mimo-free` - All route through OpenRouter → fail with 400/404 when trying these model IDs - **Working free combos as of 2026-08-14:** `free-lkgp` (DeepSeek V4 Flash), `mega-free` (StepFun 3.7 Flash), `smart-route` / `smart-route-ia` (nested combos) - **LiteLLM routing issue identified:** Hermes was previously pointing at `http://litellm:4000/v1` with `free-auto` model, causing 6–26s per call. OmniRoute's `auto/best-free` tested at 265ms. Config now points at OmniRoute. - **GPT-5.6 Luna research:** Community consensus — Luna is cleaner/more polished; DeepSeek V4 Flash is more persistent/cheaper. Luna better for visual/UI work and final review; DeepSeek better for automation and high-volume executor work. Provider route matters more than model name. ## Recent Changes (2026-08-13) - **OmniRoute migrated CT600 → CT601.** Provider renamed `custom:omniroute` → `custom:omni`; base URL `http://omniroute:20128/v1` → `http://omni:20128/v1`. Host `omni.kangaroo-eel.ts.net` (100.93.204.4, active) replaces `omniroute.kangaroo-eel.ts.net` (100.88.81.19, offline). CT600 stopped, CT601 running. Database path `/var/lib/omniroute/storage.sqlite` → `/opt/omniroute/storage.sqlite`. API key unchanged. All 14 config + 22 cron-job `custom:omniroute` references updated to `custom:omni`. Pre-migration backup at `~/.hermes/config.yaml.bak.omniroute-removal`. Verified live 2026-08-13 (`pct list`: 600 stopped, 601 running; `grep`: 0 remaining `custom:omniroute` refs). ## Recent Changes (2026-07-25) - **free-stack combo updated:** Nemotron 3 Ultra 550B model at index 4 now configured with NVIDIA-specific parameters: - `max_tokens`: 8192 (was default 65536, exceeding NVIDIA NIM limit of 16384) - `reasoning_effort`: medium (was high, reducing long internal monologues) - `chat_template_kwargs.force_nonempty_content`: true (prevents empty responses when tool calling + reasoning) - **Combo ID:** `23d25f8f-5eb0-4b6f-b966-e260ca3905b3` - **Hermes config.yaml updated:** `agent.reasoning_effort: medium`, `agent.max_tokens: 8192`, `agent.chat_template_kwargs.force_nonempty_content: true` - **Rationale:** NVIDIA NIM API documents max output of 16384 tokens; medium reasoning uses more efficient chat-template mode; force_nonempty_content resolves empty responses with tool calls combo in picker may still route ## Recent Changes (2026-08-07) - **Three-layer root cause for slowness identified and fixed:** 1. **npm package .env override:** OmniRoute loads three .env files (`/var/lib/omniroute/.env`, `/root/.omniroute/.env`, `/usr/lib/node_modules/omniroute/.env`). The npm package's `.env` ships with `ENABLE_SOCKS5_PROXY=true` and `npm update` re-installs it with that default, silently overriding the `/var/lib` config. Fix: flip all three to `false`, then restart. After ANY upgrade, check all three. 2. **Dead provider connection:** Connection `58debd59` (prefix `nanobot`) points at `clawtest:8900/v1` — nanobot on clawtest is DOWN. Every model sync + credential health check retried it, burning event-loop time. Disabled it. 3. **PVE host load leaking into CT600:** CT600 has `nesting=1`, so `uptime`/`top` inside the container mirror PVE's host load exactly (verified: identical 10.30/9.23/10.37 on both). Same trap as CT241. Judge CT600 CPU from `pct exec 600 -- ps -o %cpu` or host-side `ps`, never from in-container load average. - **First-request-after-cache-expiry stall:** `/v1/models` hangs 10-15s+ on the FIRST hit after the model catalog cache expires (rebuild enumerates all connections, including slow dead ones), then answers in ~0.02s from cache. Don't judge health on one curl — run a burst of 5 and check if later tries are sub-second. - **`is_active=0` does NOT stop CredentialHealth probes:** Disabling a dead connection stops it from ModelSync but the health checker still probes it and logs `[ProxyFetch] ECONNREFUSED` noise. That noise is benign once the endpoint is unreachable. - **Lessons banked in `omniroute-ops` skill:** `references/omniroute-upgrade-procedure.md` updated with all three pitfalls. ## Recent Changes (2026-08-04) - **OmniRoute free-DeepSeek silent-stop mitigation:** Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances. Reworked both combos: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort. See [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]]. ## Related - [[model-providers]] - [[model-routing]]