Files
agent-estate-wiki/systems/omniroute.md
T
Hermes Rhino 95fd7d33f1 audit(2026-10-20): omni-101 503 burst 2 — incident back to active; scheduled-tasks rebuilt from jobs.json
- burst 2 (2026-10-19): 9 model calls across 8 jobs, 159/13 day
- burst-2 table names/schedules/aliases corrected from live jobs.json
- non-clean last_status 7 -> 12; unclosed incident rows 13 (query stated)
- 2026-10-17 tail corrected 162/5 -> 163/5
2026-10-20 00:12:39 +08:00

14 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
OmniRoute system active 2026-07-22 2026-10-20 2026-10-20 high
routing
provider
model
inference
raw/configs/hermes-config-sanitized.txt

OmniRoute

Recent Changes (2026-10-20)

  • Recurring 503 … resource pressure load-shedding on omniroute-1 — second burst in four days. On 2026-10-19 the node failed 9 model calls across 8 jobs (06:01, then 07:01 → 12:01, plus 4 derivative Mnemosyne Watchdog reds) with RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly., while /api/health answered 200 {"status":"ok"} and /v1/models answered 401 throughout. Burst 1 ran 2026-10-16 08:00 → 10-17 03:00 with the identical error string and the identical 9-call blast radius. Both bursts self-cleared; nothing was changed on CT600 or omniroute-1 in between. Root cause unknown — no host telemetry was captured during either burst. → incidents/2026-10-16-omniroute-503-resource-pressure
  • Every failing job is pinned to the same alias on the same node. All eight read provider: custom:omni-101, model: smart-route in ~/.hermes/cron/jobs.json (verified 2026-10-20). With no second provider pin and no router-side failover between omniroute-1 and CT600 omniroute, one node's load-shedding blanks the fleet's digest slots.
  • Probe properly. A green /api/health proves nothing about the model path — it stayed 200 ok through a ~19 h burst and a second 7 h burst. To catch the next one, run a real completion against custom:omni-101 while jobs are still failing, and capture the node's CPU / RAM / queue depth during the burst, not afterwards.

Purpose

Secondary provider/router for Hermes auxiliary roles and some agent routing.

Current Configuration

  • Hermes provider name: custom:omni (maps to provider omniroute:20128)
  • Base URL: http://omniroute:20128/v1
  • Host reachability: bare name omniroute resolves to 100.88.81.19; direct HTTP on :20128 answers 401 (auth-required, service live)
  • Runs on: Proxmox CT600 (omniroute, Tailscale 100.88.81.19, onboot=1) — verified running 2026-10-11
  • DB path: /opt/omniroute/storage.sqlite (was /var/lib/omniroute/storage.sqlite)
  • Backup: pre-migration config saved at ~/.hermes/config.yaml.bak.omniroute-removal
  • Miss: Hermes default provider path does not use OmniRoute for main chat; used for auxiliary roles instead
  • Auxiliary roles in Hermes config: skills_hub, approval, mcp
  • Key prefix visible in config: sk-5c67fbdc0... — full value not stored in wiki

CT600 / CT601 / omniroute-1 — actual state (2026-10-11)

The wiki previously recorded CT600 as retired and CT601 as the live successor. That is no longer true. Live PVE and Tailscale checks return:

Node Status (live 2026-10-11) Tailscale IP Notes
CT600 omniroute running, onboot=1 100.88.81.19 OmniRoute listening on 0.0.0.0:20128; /v1/models returns 401 (alive, auth required)
CT601 omni stopped, onboot=0 100.93.204.4 Tailscale peer last seen 54d ago
omniroute-1 active (not in PVE pct list) 100.78.63.20 Third OmniRoute node; /v1/models returns 401. Registered in Hermes config as custom provider omni-101, which carries skills_hub (model free-stack, 5384 models, default smart-route)
  • Hermes config resolving to the omniroute hostname (custom:omni → omniroute:20128) therefore points at CT600, not CT601.
  • CT600's combo API is reachable with its local key and returns combos (e.g. go-pro), confirming it is the serving instance.
  • Do not treat CT601 as the production OmniRoute node. It is stopped with onboot=0.
  • Treat omniroute-1 (100.78.63.20) as a second live instance used by the omni-101 provider. It is not visible in pct list, so it is likely a VM or off-PVE host; its identity was not confirmed.

Catalog and Routing Facts

  • Catalog reachable: yes, 47 combos returned at /api/combos
  • Combo schema observed: name, strategy, models, id; some combos have config, isHidden, uuid
  • Combo-level context can override individual models
  • auto/lkgp strategies may silently change chosen models
  • Model mappings can substitute models silently
  • Session affinity caches last successful provider in session_model_history

Catalog Evidence (2026-07-22)

  • Advertised context for gemini/gemini-3.6-flash: 1048576
  • Related models in catalog: kc/google/gemini-3.6-flash, kilocode/google/gemini-3.6-flash, nous/google/gemini-3.6-flash, opencode-zen/gemini-3.6-flash, vertex/gemini-3.6-flash
  • No live combo in the returned 47 combos referenced gemini-3.6-flash

Known Issues

  • opencode-zen proxy pitfall documented in omniroute-ops skill
  • Web search proxy failures documented in omniroute-ops skill
  • Combo discovery can differ from resolution;

AutoRoute variants (2026-08-12)

  • auto/best-free (Auto Cheap) and auto/coding-free are virtual AutoRoute variants — they do NOT appear in the persisted combo list (/api/combos returns 54 combos, none named these).
  • The LKGP engine resolves them dynamically: it scores the operator's live connected provider accounts (e.g. antigravity, opencode-zen, nvidia, openrouter) rather than a hardcoded candidate list.
  • Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
  • Example observed: an auto/best-free request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.

Recent Changes (2026-10-18) — omniroute-1 503 resource pressure (2026-10-16 -> 2026-10-17)

  • A second, different failure mode from the same node. Three days after the 34-hour TCP blackout, omniroute-1 answered model calls with HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly. — 9 model calls across 7 jobs, between 10-16 08:00:37 and 10-17 03:00:28 (~19 h). → incidents/2026-10-16-omniroute-503-resource-pressure
  • The control plane looked healthy throughout. /api/health returned 200 {"status":"ok"} and /v1/models returned 401 (auth-required) before, during and after the window — re-probed 2026-10-18 00:33. A green health endpoint does not mean the model path is serving: 503 here is load-shedding under resource pressure, not a dead process, so process checks miss it.
  • It recovered on its own; nothing was changed on the node. Last 503 10-17 03:00:28; 10-17 closed 162 completed / 5 failed (all five before 03:01) and 10-18 is green so far. Every affected job re-ran clean on its next slot — b63ed5817212 10-17 06:01, 08eb86f02d37 10-17 08:31, ecc1fc13655a and ea79a2107e36 both 10-17 12:02, auto-closing their own incidents.

Recent Changes (2026-10-16) — omniroute-1 connection-refused storm

  • omniroute-1 (100.78.63.20:20128, provider custom:omni-101) was TCP-unreachable from 2026-10-14 08:30 to 2026-10-15 ~18:07 AWST — httpcore.ConnectError: [Errno 111] Connection refused → openai.APIConnectionError: Connection error. → RuntimeError: Connection error. Result: 24 genuine cron model-call failures across 12 jobs plus 12 derivative watchdog rows, and every scheduled digest for ~34 hours was silently suppressed (including the 10-14 23:59 Wiki Audit).
  • This node is a single point of failure for the whole scheduled estate. All 18 enabled agent-mode cron jobs are pinned to custom:omni-101 (smart-route ×10, free-stack ×6, deepseek-paid-smart ×2). The only other model-capable cron pin — custom:omni (CT600, omniroute:20128) — is attached to a no_agent job (e530c89f2be0, Evening Stand-Down), so it is never exercised by a model call; d40085da631c, the only other custom:omni pin, is disabled. The cron fleet therefore has no model-call failover between the two nodes, and no job that can even tell you whether the second node is up.
  • Recovered — verified 2026-10-16 00:1x: GET /v1/models on 100.78.63.20:20128 → HTTP 401 in 10 ms (up, auth required); same on omniroute:20128 / 100.88.81.19. Four jobs have completed on the node since 2026-10-15 22:01.
  • Full write-up: incidents/2026-10-14-omniroute-node-connection-refused-storm

Recent Changes (2026-08-14)

  • OmniRoute factory reset on CT601. Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (auto/best-fast, auto/best-coding, etc.). Old state backed up at /opt/omniroute/factory-reset-backup-20260814-225524.
  • Providers re-added from Notion API vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry.
  • Hermes config default model changed: auto/best-free → resilient-free (via ~/.hermes/config.yaml). auto/best-free was misconfigured — had no combo definition and routed to gpt-5.6-sol (paid) instead of free models. Switched to resilient-free which is a working free combo.
  • Multiple free combos broken by upstream model ID changes:
    • OpenRouter: poolside/laguna-s-2-1:free (dash) → poolside/laguna-s-2.1:free (dot) — model ID format changed
    • Groq: meta-llama/llama-4-scout-17b-16e-instruct — retired, only llama-3.3/3.1 remain
    • Affected combos: resilient-free, free-IA, free-stack, good-free, deepseek-flash-free, mimo-free
    • All route through OpenRouter → fail with 400/404 when trying these model IDs
  • Working free combos as of 2026-08-14: free-lkgp (DeepSeek V4 Flash), mega-free (StepFun 3.7 Flash), smart-route / smart-route-ia (nested combos)
  • LiteLLM routing issue identified: Hermes was previously pointing at http://litellm:4000/v1 with free-auto model, causing 6–26s per call. OmniRoute's auto/best-free tested at 265ms. Config now points at OmniRoute.
  • GPT-5.6 Luna research: Community consensus — Luna is cleaner/more polished; DeepSeek V4 Flash is more persistent/cheaper. Luna better for visual/UI work and final review; DeepSeek better for automation and high-volume executor work. Provider route matters more than model name.

Recent Changes (2026-08-13)

  • OmniRoute migrated CT600 → CT601. Provider renamed custom:omniroute → custom:omni; base URL http://omniroute:20128/v1 → http://omni:20128/v1. Host omni.kangaroo-eel.ts.net (100.93.204.4, active) replaces omniroute.kangaroo-eel.ts.net (100.88.81.19, offline). CT600 stopped, CT601 running. Database path /var/lib/omniroute/storage.sqlite → /opt/omniroute/storage.sqlite. API key unchanged. All 14 config + 22 cron-job custom:omniroute references updated to custom:omni. Pre-migration backup at ~/.hermes/config.yaml.bak.omniroute-removal. Verified live 2026-08-13 (pct list: 600 stopped, 601 running; grep: 0 remaining custom:omniroute refs).

Recent Changes (2026-07-25)

  • free-stack combo updated: Nemotron 3 Ultra 550B model at index 4 now configured with NVIDIA-specific parameters:
    • max_tokens: 8192 (was default 65536, exceeding NVIDIA NIM limit of 16384)
    • reasoning_effort: medium (was high, reducing long internal monologues)
    • chat_template_kwargs.force_nonempty_content: true (prevents empty responses when tool calling + reasoning)
  • Combo ID: 23d25f8f-5eb0-4b6f-b966-e260ca3905b3
  • Hermes config.yaml updated: agent.reasoning_effort: medium, agent.max_tokens: 8192, agent.chat_template_kwargs.force_nonempty_content: true
  • Rationale: NVIDIA NIM API documents max output of 16384 tokens; medium reasoning uses more efficient chat-template mode; force_nonempty_content resolves empty responses with tool calls combo in picker may still route

Recent Changes (2026-08-07)

  • Three-layer root cause for slowness identified and fixed:
    1. npm package .env override: OmniRoute loads three .env files (/var/lib/omniroute/.env, /root/.omniroute/.env, /usr/lib/node_modules/omniroute/.env). The npm package's .env ships with ENABLE_SOCKS5_PROXY=true and npm update re-installs it with that default, silently overriding the /var/lib config. Fix: flip all three to false, then restart. After ANY upgrade, check all three.
    2. Dead provider connection: Connection 58debd59 (prefix nanobot) points at clawtest:8900/v1 — nanobot on clawtest is DOWN. Every model sync + credential health check retried it, burning event-loop time. Disabled it.
    3. PVE host load leaking into CT600: CT600 has nesting=1, so uptime/top inside the container mirror PVE's host load exactly (verified: identical 10.30/9.23/10.37 on both). Same trap as CT241. Judge CT600 CPU from pct exec 600 -- ps -o %cpu or host-side ps, never from in-container load average.
  • First-request-after-cache-expiry stall: /v1/models hangs 10-15s+ on the FIRST hit after the model catalog cache expires (rebuild enumerates all connections, including slow dead ones), then answers in ~0.02s from cache. Don't judge health on one curl — run a burst of 5 and check if later tries are sub-second.
  • is_active=0 does NOT stop CredentialHealth probes: Disabling a dead connection stops it from ModelSync but the health checker still probes it and logs [ProxyFetch] ECONNREFUSED noise. That noise is benign once the endpoint is unreachable.
  • Lessons banked in omniroute-ops skill: references/omniroute-upgrade-procedure.md updated with all three pitfalls.

Recent Changes (2026-08-04)

  • OmniRoute free-DeepSeek silent-stop mitigation: Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances. Reworked both combos: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort. See incidents/2026-08-04-omniroute-free-deepseek-silent-stops.