Files
agent-estate-wiki/systems/omniroute.md
T
Tony0410 a0b3734c95 stocktake(2026-11-01): fix the Quartz build (YAML title), close the "unfixed memory ranking" incident, and correct live drift in tailscale/docker/proxmox
BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic.
Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an
unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a
mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)".
A scan of all 117 pages found exactly one such line; it is now properly quoted.
Quartz now builds cleanly: 690 files emitted and deployed.

RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page).
The page claimed the recall-ranking fix "was not present in live config" and that
conflict 152 was "the only open conflict". Both claims were false:
- The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5,
  fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 —
  not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the
  2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks.
- The original regression test now passes: "mnemosyne recall 'Are you up to date'"
  returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale
  Uber Eats episode no longer outranks anything.
- Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise.
All three open items closed; page moved to status: resolved.

LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01):
- infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping
  times out); m93p peer record reads false for the 4th consecutive time while the
  host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages
  refreshed; no IP changes, no missing peers.
- infrastructure/docker-services.md: CT201's Tailscale IP was wrong
  (100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale
  worldmonitor-seeder-fast entry removed from the CT200 list, which is now
  regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to
  thirsty_payne.
- infrastructure/proxmox.md: storage percentages refreshed on both hosts from live
  pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an
  elevated load average recorded.
- systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md:
  re-probed and verified_on bumped.
- systems/skills-index.md: regenerated — 301 unique skills in 68 categories,
  46 archived. (301 is the unique (category, skill_name) count; find returns 302
  files due to the known nested obsidian-git-sync alias duplicate.)

Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
2026-11-01 04:48:30 +08:00

182 lines
17 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: OmniRoute
type: system
status: active
created: 2026-07-22
updated: 2026-11-01
verified_on: 2026-11-01
confidence: high
tags: [routing, provider, model, inference]
sources: [raw/configs/hermes-config-sanitized.txt]
---
# OmniRoute
## Recent Changes (2026-10-21)
- **Third failure mode on the same node, and it is not the 503 string.** `dca8482e4f76` (Daily Marketing Job
Radar) failed on 2026-10-21 at 07:40:48 after 40 m 36 s with
`RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after
19m 20s`. **Zero `HTTP 503 … resource pressure` rows exist for that day** (last: 2026-10-19 09:02) and the
sibling 07:00 job `85edf6772bca` completed normally, so the node was serving work at that hour.
→ [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]
- **One pin, three signatures in eight days:** connection-refused storm (10-14/15), resource-pressure 503s
(10-16/17 and 10-19), pre-dispatch filter skip (10-21) — all on `custom:omni-101`, all with
`model: smart-route`. No job in the fleet has a second provider pin.
- **Router-side telemetry is still missing for every one of them** — `omni-101` host logs have not been
captured during a failure, which is why none of the three has a confirmed root cause.
## Recent Changes (2026-10-20)
- **Recurring `503 … resource pressure` load-shedding on `omniroute-1` — second burst in four days.** On
**2026-10-19** the node failed **9 model calls across 8 jobs** (06:01, then 07:01 → 12:01, plus 4 derivative
Mnemosyne Watchdog reds) with
`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.`, while
`/api/health` answered `200 {"status":"ok"}` and `/v1/models` answered `401` throughout. Burst 1 ran
**2026-10-16 08:00 → 10-17 03:00** with the identical error string and the identical 9-call blast radius.
Both bursts **self-cleared**; nothing was changed on CT600 or `omniroute-1` in between.
**Root cause unknown — no host telemetry was captured during either burst.**
→ [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **Every failing job is pinned to the same alias on the same node.** All eight read
`provider: custom:omni-101`, `model: smart-route` in `~/.hermes/cron/jobs.json` (verified 2026-10-20).
With no second provider pin and no router-side failover between `omniroute-1` and CT600 `omniroute`, one
node's load-shedding blanks the fleet's digest slots.
- **Probe properly.** A green `/api/health` proves nothing about the model path — it stayed `200 ok` through a
~19 h burst and a second 7 h burst. To catch the next one, run a **real completion** against
`custom:omni-101` while jobs are still failing, and capture the node's CPU / RAM / queue depth *during* the
burst, not afterwards.
## Purpose
Provider/router for Hermes — ⚠ **updated 2026-10-26: the `omniroute-1` node is the DEFAULT chat path again.**
Hermes's default provider is `custom:omni-101` → `http://100.78.63.20:20128` with model `smart-route`
(reverted 2026-10-24 19:06; the 2026-10-20 → 10-24 window pointed the default at `nous`).
The CT600 provider key `custom:omni` (`http://omniroute:20128/v1`) remains auxiliary-only.
→ [[systems/current-profile]]
## Current Configuration
- **Hermes default provider:** `custom:omni-101` → base URL `http://100.78.63.20:20128` (CT101, `m93p`), model default **`smart-route`** — **re-verified live 2026-10-31** from `~/.hermes/config.yaml` (`model.default: smart-route`, `model.provider: custom:omni-101`, `model.base_url: http://100.78.63.20:20128`)
- **Secondary provider entry:** `custom:Omniroute:20128` → base URL `http://omniroute:20128/v1` (CT600, `pve`), referenced by the `moa` reference-model set (`vertex/gemini-3.6-flash`, `kilo-gateway/kilo-auto/free`, `nvidia/deepseek-ai/deepseek-v4-pro`, `freeapi/nemotron-3-ultra-550b`) — auxiliary/MoA use, **not** the main chat default
- **Host reachability:** bare name `omniroute` resolves to `100.88.81.19`; direct HTTP on `:20128` answers `401` (auth-required, service live)
- **Runs on:** two live nodes — CT600 (`omniroute`, Tailscale `100.88.81.19`, on `pve`) and CT101 (`omniroute`, Tailscale `100.78.63.20`, on `m93p`) — **both verified running and listening on `0.0.0.0:20128` on 2026-10-31**
- **DB path:** `/opt/omniroute/storage.sqlite` (was `/var/lib/omniroute/storage.sqlite`)
- **Backup:** pre-migration config saved at `~/.hermes/config.yaml.bak.omniroute-removal`
- **Correction (2026-10-26):** the earlier *"Hermes default provider path does not use OmniRoute for main chat"* note is **stale** — the default is OmniRoute again (`custom:omni-101`, model `smart-route`). The **CT600** key `custom:Omniroute:20128` is still auxiliary-only (MoA reference models).
- **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp` — all on `custom:omni-101` with model `no-think/freeapi/auto` (live-verified 2026-10-31)
- **Key prefix visible in config:** `sk-5c67fbdc0...` — full value not stored in wiki
## CT600 / CT601 / omniroute-1 — actual state (re-verified 2026-10-31)
The wiki previously recorded CT600 as retired and CT601 as the live successor, then CT601 as stopped.
**All three framings are now superseded.** Live `pct list` on **both** hypervisors and live HTTP probes return:
| Node | Status (live 2026-10-31) | Tailscale IP | Notes |
|---|---|---|---|
| CT600 `omniroute` | **running** | `100.88.81.19` | On `pve`. OmniRoute listening on `0.0.0.0:20128`; `/v1/models` and `/api/combos` both return `401` (alive, auth required) |
| CT101 `omniroute-1` | **running** | `100.78.63.20` | On `m93p`. Live `pct config 101` → `hostname: omniroute`; `omniroute` listening on `0.0.0.0:20128`; `/v1/models` and `/api/combos` both return `401`. Registered in Hermes config as `custom:omni-101` — **the live default provider** |
| CT601 `omni` | **does not exist** | — | `/etc/pve/lxc/601.conf` absent on **both** `pve` and `m93p`. The `omni` peer (`100.93.204.4`) has **expired out of the tailnet entirely** — live `tailscale status --json` has no peer named `omni` and no peer at that IP |
- Hermes config resolving to the omniroute hostname (`custom:Omniroute:20128` → `omniroute:20128`) therefore points at **CT600**; the live default `custom:omni-101` points at **CT101**.
- **CT601 was deleted, not merely stopped** — the earlier *"stopped, onboot=0"* entry was wrong.
- **The `omniroute-1` identity is now confirmed** — it is **CT101 on `m93p`**, not "a VM or off-PVE host". The 2026-10-11 note that it was "not visible in `pct list`" was an artifact of probing only `pve`.
- **Two live OmniRoute nodes remain** (CT600, CT101), both serving `:20128` with separate credentials — a pin's usability must be probed on the node that actually serves it.
## Catalog and Routing Facts
- Catalog reachable: yes, 47 combos returned at `/api/combos`
- Combo schema observed: `name`, `strategy`, `models`, `id`; some combos have `config`, `isHidden`, `uuid`
- Combo-level context can override individual models
- `auto`/`lkgp` strategies may silently change chosen models
- Model mappings can substitute models silently
- Session affinity caches last successful provider in `session_model_history`
## Catalog Evidence (2026-07-22)
- Advertised context for `gemini/gemini-3.6-flash`: `1048576`
- Related models in catalog: `kc/google/gemini-3.6-flash`, `kilocode/google/gemini-3.6-flash`, `nous/google/gemini-3.6-flash`, `opencode-zen/gemini-3.6-flash`, `vertex/gemini-3.6-flash`
- No live combo in the returned 47 combos referenced `gemini-3.6-flash`
## Known Issues
- opencode-zen proxy pitfall documented in omniroute-ops skill
- Web search proxy failures documented in omniroute-ops skill
- Combo discovery can differ from resolution;
## AutoRoute variants (2026-08-12)
- `auto/best-free` (Auto Cheap) and `auto/coding-free` are **virtual** AutoRoute variants — they do NOT appear in the persisted combo list (`/api/combos` returns 54 combos, none named these).
- The LKGP engine resolves them dynamically: it scores the operator's **live connected provider accounts** (e.g. `antigravity`, `opencode-zen`, `nvidia`, `openrouter`) rather than a hardcoded candidate list.
- Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
- Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
## Recent Changes (2026-10-18) — `omniroute-1` 503 resource pressure (2026-10-16 -> 2026-10-17)
- **A second, different failure mode from the same node.** Three days after the 34-hour TCP blackout,
`omniroute-1` answered model calls with **`HTTP 503: Service temporarily unavailable due to resource
pressure. Retry shortly.`** — **9 model calls across 7 jobs**, between **10-16 08:00:37** and
**10-17 03:00:28** (~19 h). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **The control plane looked healthy throughout.** `/api/health` returned `200 {"status":"ok"}` and
`/v1/models` returned `401` (auth-required) before, during and after the window — re-probed 2026-10-18
00:33. A green health endpoint does **not** mean the model path is serving: 503 here is load-shedding
under resource pressure, not a dead process, so process checks miss it.
- **It recovered on its own; nothing was changed on the node.** Last 503 **10-17 03:00:28**; `10-17`
closed **162 completed / 5 failed** (all five before 03:01) and `10-18` is green so far. Every affected
job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01, `08eb86f02d37` 10-17 08:31,
`ecc1fc13655a` and `ea79a2107e36` both 10-17 12:02, auto-closing their own incidents.
## Recent Changes (2026-10-16) — `omniroute-1` connection-refused storm
- **`omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) was TCP-unreachable from 2026-10-14
08:30 to 2026-10-15 ~18:07 AWST** — `httpcore.ConnectError: [Errno 111] Connection refused` →
`openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` Result: **24 genuine
cron model-call failures across 12 jobs** plus 12 derivative watchdog rows, and **every scheduled digest
for ~34 hours was silently suppressed** (including the 10-14 23:59 Wiki Audit).
- **This node is a single point of failure for the whole scheduled estate.** All **18** enabled
`agent`-mode cron jobs are pinned to `custom:omni-101` (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2). The only other model-capable cron pin — **`custom:omni` (CT600,
`omniroute:20128`)** — is attached to a **`no_agent`** job (`e530c89f2be0`, Evening Stand-Down), so it is
**never exercised by a model call**; `d40085da631c`, the only other `custom:omni` pin, is **disabled**.
The cron fleet therefore has **no model-call failover between the two nodes**, and no job that can even
tell you whether the second node is up.
- **Recovered — verified 2026-10-16 00:1x:** `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in
10 ms** (up, auth required); same on `omniroute:20128` / `100.88.81.19`. Four jobs have completed on the
node since 2026-10-15 22:01.
- Full write-up: [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
## Recent Changes (2026-08-14)
- **OmniRoute factory reset on CT601.** Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (`auto/best-fast`, `auto/best-coding`, etc.). Old state backed up at `/opt/omniroute/factory-reset-backup-20260814-225524`.
- **Providers re-added from Notion API vault:** Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry.
- **Hermes config default model changed:** `auto/best-free` → `resilient-free` (via `~/.hermes/config.yaml`). `auto/best-free` was misconfigured — had no combo definition and routed to `gpt-5.6-sol` (paid) instead of free models. Switched to `resilient-free` which is a working free combo.
- **Multiple free combos broken by upstream model ID changes:**
- OpenRouter: `poolside/laguna-s-2-1:free` (dash) → `poolside/laguna-s-2.1:free` (dot) — model ID format changed
- Groq: `meta-llama/llama-4-scout-17b-16e-instruct` — **retired**, only llama-3.3/3.1 remain
- Affected combos: `resilient-free`, `free-IA`, `free-stack`, `good-free`, `deepseek-flash-free`, `mimo-free`
- All route through OpenRouter → fail with 400/404 when trying these model IDs
- **Working free combos as of 2026-08-14:** `free-lkgp` (DeepSeek V4 Flash), `mega-free` (StepFun 3.7 Flash), `smart-route` / `smart-route-ia` (nested combos)
- **LiteLLM routing issue identified:** Hermes was previously pointing at `http://litellm:4000/v1` with `free-auto` model, causing 6–26s per call. OmniRoute's `auto/best-free` tested at 265ms. Config now points at OmniRoute.
- **GPT-5.6 Luna research:** Community consensus — Luna is cleaner/more polished; DeepSeek V4 Flash is more persistent/cheaper. Luna better for visual/UI work and final review; DeepSeek better for automation and high-volume executor work. Provider route matters more than model name.
## Recent Changes (2026-08-13)
- **OmniRoute migrated CT600 → CT601.** Provider renamed `custom:omniroute` → `custom:omni`; base URL `http://omniroute:20128/v1` → `http://omni:20128/v1`. Host `omni.kangaroo-eel.ts.net` (100.93.204.4, active) replaces `omniroute.kangaroo-eel.ts.net` (100.88.81.19, offline). CT600 stopped, CT601 running. Database path `/var/lib/omniroute/storage.sqlite` → `/opt/omniroute/storage.sqlite`. API key unchanged. All 14 config + 22 cron-job `custom:omniroute` references updated to `custom:omni`. Pre-migration backup at `~/.hermes/config.yaml.bak.omniroute-removal`. Verified live 2026-08-13 (`pct list`: 600 stopped, 601 running; `grep`: 0 remaining `custom:omniroute` refs).
## Recent Changes (2026-07-25)
- **free-stack combo updated:** Nemotron 3 Ultra 550B model at index 4 now configured with NVIDIA-specific parameters:
- `max_tokens`: 8192 (was default 65536, exceeding NVIDIA NIM limit of 16384)
- `reasoning_effort`: medium (was high, reducing long internal monologues)
- `chat_template_kwargs.force_nonempty_content`: true (prevents empty responses when tool calling + reasoning)
- **Combo ID:** `23d25f8f-5eb0-4b6f-b966-e260ca3905b3`
- **Hermes config.yaml updated:** `agent.reasoning_effort: medium`, `agent.max_tokens: 8192`, `agent.chat_template_kwargs.force_nonempty_content: true`
- **Rationale:** NVIDIA NIM API documents max output of 16384 tokens; medium reasoning uses more efficient chat-template mode; force_nonempty_content resolves empty responses with tool calls combo in picker may still route
## Recent Changes (2026-08-07)
- **Three-layer root cause for slowness identified and fixed:**
1. **npm package .env override:** OmniRoute loads three .env files (`/var/lib/omniroute/.env`, `/root/.omniroute/.env`, `/usr/lib/node_modules/omniroute/.env`). The npm package's `.env` ships with `ENABLE_SOCKS5_PROXY=true` and `npm update` re-installs it with that default, silently overriding the `/var/lib` config. Fix: flip all three to `false`, then restart. After ANY upgrade, check all three.
2. **Dead provider connection:** Connection `58debd59` (prefix `nanobot`) points at `clawtest:8900/v1` — nanobot on clawtest is DOWN. Every model sync + credential health check retried it, burning event-loop time. Disabled it.
3. **PVE host load leaking into CT600:** CT600 has `nesting=1`, so `uptime`/`top` inside the container mirror PVE's host load exactly (verified: identical 10.30/9.23/10.37 on both). Same trap as CT241. Judge CT600 CPU from `pct exec 600 -- ps -o %cpu` or host-side `ps`, never from in-container load average.
- **First-request-after-cache-expiry stall:** `/v1/models` hangs 10-15s+ on the FIRST hit after the model catalog cache expires (rebuild enumerates all connections, including slow dead ones), then answers in ~0.02s from cache. Don't judge health on one curl — run a burst of 5 and check if later tries are sub-second.
- **`is_active=0` does NOT stop CredentialHealth probes:** Disabling a dead connection stops it from ModelSync but the health checker still probes it and logs `[ProxyFetch] ECONNREFUSED` noise. That noise is benign once the endpoint is unreachable.
- **Lessons banked in `omniroute-ops` skill:** `references/omniroute-upgrade-procedure.md` updated with all three pitfalls.
## Recent Changes (2026-08-04)
- **OmniRoute free-DeepSeek silent-stop mitigation:** Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances. Reworked both combos: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort. See [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]].
## Related
- [[model-providers]]
- [[model-routing]]