BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic. Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)". A scan of all 117 pages found exactly one such line; it is now properly quoted. Quartz now builds cleanly: 690 files emitted and deployed. RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page). The page claimed the recall-ranking fix "was not present in live config" and that conflict 152 was "the only open conflict". Both claims were false: - The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5, fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 — not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the 2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks. - The original regression test now passes: "mnemosyne recall 'Are you up to date'" returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale Uber Eats episode no longer outranks anything. - Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise. All three open items closed; page moved to status: resolved. LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01): - infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping times out); m93p peer record reads false for the 4th consecutive time while the host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages refreshed; no IP changes, no missing peers. - infrastructure/docker-services.md: CT201's Tailscale IP was wrong (100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale worldmonitor-seeder-fast entry removed from the CT200 list, which is now regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to thirsty_payne. - infrastructure/proxmox.md: storage percentages refreshed on both hosts from live pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an elevated load average recorded. - systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md: re-probed and verified_on bumped. - systems/skills-index.md: regenerated — 301 unique skills in 68 categories, 46 archived. (301 is the unique (category, skill_name) count; find returns 302 files due to the known nested obsidian-git-sync alias duplicate.) Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
182 lines
17 KiB
Markdown
182 lines
17 KiB
Markdown
---
|
||
title: OmniRoute
|
||
type: system
|
||
status: active
|
||
created: 2026-07-22
|
||
updated: 2026-11-01
|
||
verified_on: 2026-11-01
|
||
confidence: high
|
||
tags: [routing, provider, model, inference]
|
||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||
---
|
||
|
||
# OmniRoute
|
||
|
||
## Recent Changes (2026-10-21)
|
||
|
||
- **Third failure mode on the same node, and it is not the 503 string.** `dca8482e4f76` (Daily Marketing Job
|
||
Radar) failed on 2026-10-21 at 07:40:48 after 40 m 36 s with
|
||
`RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after
|
||
19m 20s`. **Zero `HTTP 503 … resource pressure` rows exist for that day** (last: 2026-10-19 09:02) and the
|
||
sibling 07:00 job `85edf6772bca` completed normally, so the node was serving work at that hour.
|
||
→ [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]
|
||
- **One pin, three signatures in eight days:** connection-refused storm (10-14/15), resource-pressure 503s
|
||
(10-16/17 and 10-19), pre-dispatch filter skip (10-21) — all on `custom:omni-101`, all with
|
||
`model: smart-route`. No job in the fleet has a second provider pin.
|
||
- **Router-side telemetry is still missing for every one of them** — `omni-101` host logs have not been
|
||
captured during a failure, which is why none of the three has a confirmed root cause.
|
||
|
||
## Recent Changes (2026-10-20)
|
||
|
||
- **Recurring `503 … resource pressure` load-shedding on `omniroute-1` — second burst in four days.** On
|
||
**2026-10-19** the node failed **9 model calls across 8 jobs** (06:01, then 07:01 → 12:01, plus 4 derivative
|
||
Mnemosyne Watchdog reds) with
|
||
`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.`, while
|
||
`/api/health` answered `200 {"status":"ok"}` and `/v1/models` answered `401` throughout. Burst 1 ran
|
||
**2026-10-16 08:00 → 10-17 03:00** with the identical error string and the identical 9-call blast radius.
|
||
Both bursts **self-cleared**; nothing was changed on CT600 or `omniroute-1` in between.
|
||
**Root cause unknown — no host telemetry was captured during either burst.**
|
||
→ [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||
- **Every failing job is pinned to the same alias on the same node.** All eight read
|
||
`provider: custom:omni-101`, `model: smart-route` in `~/.hermes/cron/jobs.json` (verified 2026-10-20).
|
||
With no second provider pin and no router-side failover between `omniroute-1` and CT600 `omniroute`, one
|
||
node's load-shedding blanks the fleet's digest slots.
|
||
- **Probe properly.** A green `/api/health` proves nothing about the model path — it stayed `200 ok` through a
|
||
~19 h burst and a second 7 h burst. To catch the next one, run a **real completion** against
|
||
`custom:omni-101` while jobs are still failing, and capture the node's CPU / RAM / queue depth *during* the
|
||
burst, not afterwards.
|
||
|
||
## Purpose
|
||
Provider/router for Hermes — ⚠ **updated 2026-10-26: the `omniroute-1` node is the DEFAULT chat path again.**
|
||
Hermes's default provider is `custom:omni-101` → `http://100.78.63.20:20128` with model `smart-route`
|
||
(reverted 2026-10-24 19:06; the 2026-10-20 → 10-24 window pointed the default at `nous`).
|
||
The CT600 provider key `custom:omni` (`http://omniroute:20128/v1`) remains auxiliary-only.
|
||
→ [[systems/current-profile]]
|
||
|
||
## Current Configuration
|
||
- **Hermes default provider:** `custom:omni-101` → base URL `http://100.78.63.20:20128` (CT101, `m93p`), model default **`smart-route`** — **re-verified live 2026-10-31** from `~/.hermes/config.yaml` (`model.default: smart-route`, `model.provider: custom:omni-101`, `model.base_url: http://100.78.63.20:20128`)
|
||
- **Secondary provider entry:** `custom:Omniroute:20128` → base URL `http://omniroute:20128/v1` (CT600, `pve`), referenced by the `moa` reference-model set (`vertex/gemini-3.6-flash`, `kilo-gateway/kilo-auto/free`, `nvidia/deepseek-ai/deepseek-v4-pro`, `freeapi/nemotron-3-ultra-550b`) — auxiliary/MoA use, **not** the main chat default
|
||
- **Host reachability:** bare name `omniroute` resolves to `100.88.81.19`; direct HTTP on `:20128` answers `401` (auth-required, service live)
|
||
- **Runs on:** two live nodes — CT600 (`omniroute`, Tailscale `100.88.81.19`, on `pve`) and CT101 (`omniroute`, Tailscale `100.78.63.20`, on `m93p`) — **both verified running and listening on `0.0.0.0:20128` on 2026-10-31**
|
||
- **DB path:** `/opt/omniroute/storage.sqlite` (was `/var/lib/omniroute/storage.sqlite`)
|
||
- **Backup:** pre-migration config saved at `~/.hermes/config.yaml.bak.omniroute-removal`
|
||
- **Correction (2026-10-26):** the earlier *"Hermes default provider path does not use OmniRoute for main chat"* note is **stale** — the default is OmniRoute again (`custom:omni-101`, model `smart-route`). The **CT600** key `custom:Omniroute:20128` is still auxiliary-only (MoA reference models).
|
||
- **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp` — all on `custom:omni-101` with model `no-think/freeapi/auto` (live-verified 2026-10-31)
|
||
- **Key prefix visible in config:** `sk-5c67fbdc0...` — full value not stored in wiki
|
||
|
||
## CT600 / CT601 / omniroute-1 — actual state (re-verified 2026-10-31)
|
||
|
||
The wiki previously recorded CT600 as retired and CT601 as the live successor, then CT601 as stopped.
|
||
**All three framings are now superseded.** Live `pct list` on **both** hypervisors and live HTTP probes return:
|
||
|
||
| Node | Status (live 2026-10-31) | Tailscale IP | Notes |
|
||
|---|---|---|---|
|
||
| CT600 `omniroute` | **running** | `100.88.81.19` | On `pve`. OmniRoute listening on `0.0.0.0:20128`; `/v1/models` and `/api/combos` both return `401` (alive, auth required) |
|
||
| CT101 `omniroute-1` | **running** | `100.78.63.20` | On `m93p`. Live `pct config 101` → `hostname: omniroute`; `omniroute` listening on `0.0.0.0:20128`; `/v1/models` and `/api/combos` both return `401`. Registered in Hermes config as `custom:omni-101` — **the live default provider** |
|
||
| CT601 `omni` | **does not exist** | — | `/etc/pve/lxc/601.conf` absent on **both** `pve` and `m93p`. The `omni` peer (`100.93.204.4`) has **expired out of the tailnet entirely** — live `tailscale status --json` has no peer named `omni` and no peer at that IP |
|
||
|
||
- Hermes config resolving to the omniroute hostname (`custom:Omniroute:20128` → `omniroute:20128`) therefore points at **CT600**; the live default `custom:omni-101` points at **CT101**.
|
||
- **CT601 was deleted, not merely stopped** — the earlier *"stopped, onboot=0"* entry was wrong.
|
||
- **The `omniroute-1` identity is now confirmed** — it is **CT101 on `m93p`**, not "a VM or off-PVE host". The 2026-10-11 note that it was "not visible in `pct list`" was an artifact of probing only `pve`.
|
||
- **Two live OmniRoute nodes remain** (CT600, CT101), both serving `:20128` with separate credentials — a pin's usability must be probed on the node that actually serves it.
|
||
|
||
|
||
## Catalog and Routing Facts
|
||
- Catalog reachable: yes, 47 combos returned at `/api/combos`
|
||
- Combo schema observed: `name`, `strategy`, `models`, `id`; some combos have `config`, `isHidden`, `uuid`
|
||
- Combo-level context can override individual models
|
||
- `auto`/`lkgp` strategies may silently change chosen models
|
||
- Model mappings can substitute models silently
|
||
- Session affinity caches last successful provider in `session_model_history`
|
||
|
||
## Catalog Evidence (2026-07-22)
|
||
- Advertised context for `gemini/gemini-3.6-flash`: `1048576`
|
||
- Related models in catalog: `kc/google/gemini-3.6-flash`, `kilocode/google/gemini-3.6-flash`, `nous/google/gemini-3.6-flash`, `opencode-zen/gemini-3.6-flash`, `vertex/gemini-3.6-flash`
|
||
- No live combo in the returned 47 combos referenced `gemini-3.6-flash`
|
||
|
||
## Known Issues
|
||
- opencode-zen proxy pitfall documented in omniroute-ops skill
|
||
- Web search proxy failures documented in omniroute-ops skill
|
||
- Combo discovery can differ from resolution;
|
||
|
||
## AutoRoute variants (2026-08-12)
|
||
- `auto/best-free` (Auto Cheap) and `auto/coding-free` are **virtual** AutoRoute variants — they do NOT appear in the persisted combo list (`/api/combos` returns 54 combos, none named these).
|
||
- The LKGP engine resolves them dynamically: it scores the operator's **live connected provider accounts** (e.g. `antigravity`, `opencode-zen`, `nvidia`, `openrouter`) rather than a hardcoded candidate list.
|
||
- Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
|
||
- Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
|
||
|
||
## Recent Changes (2026-10-18) — `omniroute-1` 503 resource pressure (2026-10-16 -> 2026-10-17)
|
||
|
||
- **A second, different failure mode from the same node.** Three days after the 34-hour TCP blackout,
|
||
`omniroute-1` answered model calls with **`HTTP 503: Service temporarily unavailable due to resource
|
||
pressure. Retry shortly.`** — **9 model calls across 7 jobs**, between **10-16 08:00:37** and
|
||
**10-17 03:00:28** (~19 h). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||
- **The control plane looked healthy throughout.** `/api/health` returned `200 {"status":"ok"}` and
|
||
`/v1/models` returned `401` (auth-required) before, during and after the window — re-probed 2026-10-18
|
||
00:33. A green health endpoint does **not** mean the model path is serving: 503 here is load-shedding
|
||
under resource pressure, not a dead process, so process checks miss it.
|
||
- **It recovered on its own; nothing was changed on the node.** Last 503 **10-17 03:00:28**; `10-17`
|
||
closed **162 completed / 5 failed** (all five before 03:01) and `10-18` is green so far. Every affected
|
||
job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01, `08eb86f02d37` 10-17 08:31,
|
||
`ecc1fc13655a` and `ea79a2107e36` both 10-17 12:02, auto-closing their own incidents.
|
||
|
||
## Recent Changes (2026-10-16) — `omniroute-1` connection-refused storm
|
||
|
||
- **`omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) was TCP-unreachable from 2026-10-14
|
||
08:30 to 2026-10-15 ~18:07 AWST** — `httpcore.ConnectError: [Errno 111] Connection refused` →
|
||
`openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` Result: **24 genuine
|
||
cron model-call failures across 12 jobs** plus 12 derivative watchdog rows, and **every scheduled digest
|
||
for ~34 hours was silently suppressed** (including the 10-14 23:59 Wiki Audit).
|
||
- **This node is a single point of failure for the whole scheduled estate.** All **18** enabled
|
||
`agent`-mode cron jobs are pinned to `custom:omni-101` (`smart-route` ×10, `free-stack` ×6,
|
||
`deepseek-paid-smart` ×2). The only other model-capable cron pin — **`custom:omni` (CT600,
|
||
`omniroute:20128`)** — is attached to a **`no_agent`** job (`e530c89f2be0`, Evening Stand-Down), so it is
|
||
**never exercised by a model call**; `d40085da631c`, the only other `custom:omni` pin, is **disabled**.
|
||
The cron fleet therefore has **no model-call failover between the two nodes**, and no job that can even
|
||
tell you whether the second node is up.
|
||
- **Recovered — verified 2026-10-16 00:1x:** `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in
|
||
10 ms** (up, auth required); same on `omniroute:20128` / `100.88.81.19`. Four jobs have completed on the
|
||
node since 2026-10-15 22:01.
|
||
- Full write-up: [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
|
||
|
||
## Recent Changes (2026-08-14)
|
||
- **OmniRoute factory reset on CT601.** Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (`auto/best-fast`, `auto/best-coding`, etc.). Old state backed up at `/opt/omniroute/factory-reset-backup-20260814-225524`.
|
||
- **Providers re-added from Notion API vault:** Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry.
|
||
- **Hermes config default model changed:** `auto/best-free` → `resilient-free` (via `~/.hermes/config.yaml`). `auto/best-free` was misconfigured — had no combo definition and routed to `gpt-5.6-sol` (paid) instead of free models. Switched to `resilient-free` which is a working free combo.
|
||
- **Multiple free combos broken by upstream model ID changes:**
|
||
- OpenRouter: `poolside/laguna-s-2-1:free` (dash) → `poolside/laguna-s-2.1:free` (dot) — model ID format changed
|
||
- Groq: `meta-llama/llama-4-scout-17b-16e-instruct` — **retired**, only llama-3.3/3.1 remain
|
||
- Affected combos: `resilient-free`, `free-IA`, `free-stack`, `good-free`, `deepseek-flash-free`, `mimo-free`
|
||
- All route through OpenRouter → fail with 400/404 when trying these model IDs
|
||
- **Working free combos as of 2026-08-14:** `free-lkgp` (DeepSeek V4 Flash), `mega-free` (StepFun 3.7 Flash), `smart-route` / `smart-route-ia` (nested combos)
|
||
- **LiteLLM routing issue identified:** Hermes was previously pointing at `http://litellm:4000/v1` with `free-auto` model, causing 6–26s per call. OmniRoute's `auto/best-free` tested at 265ms. Config now points at OmniRoute.
|
||
- **GPT-5.6 Luna research:** Community consensus — Luna is cleaner/more polished; DeepSeek V4 Flash is more persistent/cheaper. Luna better for visual/UI work and final review; DeepSeek better for automation and high-volume executor work. Provider route matters more than model name.
|
||
|
||
## Recent Changes (2026-08-13)
|
||
- **OmniRoute migrated CT600 → CT601.** Provider renamed `custom:omniroute` → `custom:omni`; base URL `http://omniroute:20128/v1` → `http://omni:20128/v1`. Host `omni.kangaroo-eel.ts.net` (100.93.204.4, active) replaces `omniroute.kangaroo-eel.ts.net` (100.88.81.19, offline). CT600 stopped, CT601 running. Database path `/var/lib/omniroute/storage.sqlite` → `/opt/omniroute/storage.sqlite`. API key unchanged. All 14 config + 22 cron-job `custom:omniroute` references updated to `custom:omni`. Pre-migration backup at `~/.hermes/config.yaml.bak.omniroute-removal`. Verified live 2026-08-13 (`pct list`: 600 stopped, 601 running; `grep`: 0 remaining `custom:omniroute` refs).
|
||
|
||
## Recent Changes (2026-07-25)
|
||
- **free-stack combo updated:** Nemotron 3 Ultra 550B model at index 4 now configured with NVIDIA-specific parameters:
|
||
- `max_tokens`: 8192 (was default 65536, exceeding NVIDIA NIM limit of 16384)
|
||
- `reasoning_effort`: medium (was high, reducing long internal monologues)
|
||
- `chat_template_kwargs.force_nonempty_content`: true (prevents empty responses when tool calling + reasoning)
|
||
- **Combo ID:** `23d25f8f-5eb0-4b6f-b966-e260ca3905b3`
|
||
- **Hermes config.yaml updated:** `agent.reasoning_effort: medium`, `agent.max_tokens: 8192`, `agent.chat_template_kwargs.force_nonempty_content: true`
|
||
- **Rationale:** NVIDIA NIM API documents max output of 16384 tokens; medium reasoning uses more efficient chat-template mode; force_nonempty_content resolves empty responses with tool calls combo in picker may still route
|
||
|
||
## Recent Changes (2026-08-07)
|
||
- **Three-layer root cause for slowness identified and fixed:**
|
||
1. **npm package .env override:** OmniRoute loads three .env files (`/var/lib/omniroute/.env`, `/root/.omniroute/.env`, `/usr/lib/node_modules/omniroute/.env`). The npm package's `.env` ships with `ENABLE_SOCKS5_PROXY=true` and `npm update` re-installs it with that default, silently overriding the `/var/lib` config. Fix: flip all three to `false`, then restart. After ANY upgrade, check all three.
|
||
2. **Dead provider connection:** Connection `58debd59` (prefix `nanobot`) points at `clawtest:8900/v1` — nanobot on clawtest is DOWN. Every model sync + credential health check retried it, burning event-loop time. Disabled it.
|
||
3. **PVE host load leaking into CT600:** CT600 has `nesting=1`, so `uptime`/`top` inside the container mirror PVE's host load exactly (verified: identical 10.30/9.23/10.37 on both). Same trap as CT241. Judge CT600 CPU from `pct exec 600 -- ps -o %cpu` or host-side `ps`, never from in-container load average.
|
||
- **First-request-after-cache-expiry stall:** `/v1/models` hangs 10-15s+ on the FIRST hit after the model catalog cache expires (rebuild enumerates all connections, including slow dead ones), then answers in ~0.02s from cache. Don't judge health on one curl — run a burst of 5 and check if later tries are sub-second.
|
||
- **`is_active=0` does NOT stop CredentialHealth probes:** Disabling a dead connection stops it from ModelSync but the health checker still probes it and logs `[ProxyFetch] ECONNREFUSED` noise. That noise is benign once the endpoint is unreachable.
|
||
- **Lessons banked in `omniroute-ops` skill:** `references/omniroute-upgrade-procedure.md` updated with all three pitfalls.
|
||
|
||
## Recent Changes (2026-08-04)
|
||
- **OmniRoute free-DeepSeek silent-stop mitigation:** Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances. Reworked both combos: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort. See [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]].
|
||
|
||
## Related
|
||
- [[model-providers]]
|
||
- [[model-routing]]
|