Files
agent-estate-wiki/systems/omniroute.md
T
hermes d981e30128 stocktake(2026-10-11): skills-index 346 skills/70 cats; freshness 62 stale/24 fresh/7 no-date; 5 live-drift corrections
Drift found and corrected against live sources (not guessed):
- OmniRoute: CT600 omniroute is RUNNING and serving :20128 (combo API live);
  CT601 omni is STOPPED with onboot=0 (peer offline 54d). Wiki had these inverted.
  Hermes custom:omni resolves to omniroute -> CT600, not CT601.
- Third OmniRoute node discovered: omniroute-1 at 100.78.63.20:20128, live (401),
  registered as custom provider omni-101 (5384 models, default smart-route,
  carries skills_hub/free-stack). Not in pct list; identity unconfirmed.
- Ignis: TWO live instances. CT460 ignis (:8080, vault wiki/agent-estate-wiki) and
  CT200 ignis-ignis-1 (image ignis-server:latest, host 8092, vault /opt/ignis/vaults).
  Both return HTTP 200. Page now distinguishes them.
- Tailscale: /agent-estate targets bare 127.0.0.1:8769 (was .../8769/agent-estate-role-dashboard.html);
  undocumented tailnet-only :9443 -> 127.0.0.1:5173 added; CT201 routes corrected
  (8085 targets + /steel 5173 + /steel-api 3000, replacing the wrong :8443 -> :8082 row).
- Proxmox: rebuilt from live pct list + qm list. Added CT475 agentzero and VM110 omarchy
  (both previously missing); 282 ubuntu-server-xfce corrected to stopped; dead rows removed.
  VM403 open-claw and VM401 hermes-old absent from qm list - flagged unconfirmed, not deleted.
- Nanobot: gateway nanobot-gateway (PID 2259048) alive on 8765/8766. It is NOT systemd
  nanobot.service/gateway.service (both return inactive - wrong unit names). Port 8900
  no longer listening; "Serve API" claim flagged unverified.
- Docker: old "runtime not available" was a CT460 socket artifact (root:docker srw-rw----,
  hermes not in group docker, no passwordless sudo in cron). Documented CT200 (~50 containers),
  CT333 (9 containers), CT460 via sudo.
- Mattermost CT255 confirmed stopped; matches Tailscale peer offline 21d.

Flagged, not investigated (no root cause):
- mycompose-db-1 on CT200 is in Restarting (1) - crash-looping.

62 pages remain >21d stale and were NOT rewritten this pass (concepts, decisions,
comparisons, most runbooks) - not probed against a live source, so reported not faked.
Quartz rebuild step still inert (no quartz.config.ts) - public/ not refreshed.

Artifacts: systems/skills-index.md regenerated, raw/stocktake/freshness-20261011.txt.
2026-10-11 03:11:07 +08:00

10 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
OmniRoute system active 2026-07-22 2026-10-11 2026-10-11 high
routing
provider
model
inference
raw/configs/hermes-config-sanitized.txt

OmniRoute

Purpose

Secondary provider/router for Hermes auxiliary roles and some agent routing.

Current Configuration

  • Hermes provider name: custom:omni (maps to provider omniroute:20128)
  • Base URL: http://omniroute:20128/v1
  • Host reachability: bare name omniroute resolves to 100.88.81.19; direct HTTP on :20128 answers 401 (auth-required, service live)
  • Runs on: Proxmox CT600 (omniroute, Tailscale 100.88.81.19, onboot=1) — verified running 2026-10-11
  • DB path: /opt/omniroute/storage.sqlite (was /var/lib/omniroute/storage.sqlite)
  • Backup: pre-migration config saved at ~/.hermes/config.yaml.bak.omniroute-removal
  • Miss: Hermes default provider path does not use OmniRoute for main chat; used for auxiliary roles instead
  • Auxiliary roles in Hermes config: skills_hub, approval, mcp
  • Key prefix visible in config: sk-5c67fbdc0... — full value not stored in wiki

CT600 / CT601 / omniroute-1 — actual state (2026-10-11)

The wiki previously recorded CT600 as retired and CT601 as the live successor. That is no longer true. Live PVE and Tailscale checks return:

Node Status (live 2026-10-11) Tailscale IP Notes
CT600 omniroute running, onboot=1 100.88.81.19 OmniRoute listening on 0.0.0.0:20128; /v1/models returns 401 (alive, auth required)
CT601 omni stopped, onboot=0 100.93.204.4 Tailscale peer last seen 54d ago
omniroute-1 active (not in PVE pct list) 100.78.63.20 Third OmniRoute node; /v1/models returns 401. Registered in Hermes config as custom provider omni-101, which carries skills_hub (model free-stack, 5384 models, default smart-route)
  • Hermes config resolving to the omniroute hostname (custom:omni → omniroute:20128) therefore points at CT600, not CT601.
  • CT600's combo API is reachable with its local key and returns combos (e.g. go-pro), confirming it is the serving instance.
  • Do not treat CT601 as the production OmniRoute node. It is stopped with onboot=0.
  • Treat omniroute-1 (100.78.63.20) as a second live instance used by the omni-101 provider. It is not visible in pct list, so it is likely a VM or off-PVE host; its identity was not confirmed.

Catalog and Routing Facts

  • Catalog reachable: yes, 47 combos returned at /api/combos
  • Combo schema observed: name, strategy, models, id; some combos have config, isHidden, uuid
  • Combo-level context can override individual models
  • auto/lkgp strategies may silently change chosen models
  • Model mappings can substitute models silently
  • Session affinity caches last successful provider in session_model_history

Catalog Evidence (2026-07-22)

  • Advertised context for gemini/gemini-3.6-flash: 1048576
  • Related models in catalog: kc/google/gemini-3.6-flash, kilocode/google/gemini-3.6-flash, nous/google/gemini-3.6-flash, opencode-zen/gemini-3.6-flash, vertex/gemini-3.6-flash
  • No live combo in the returned 47 combos referenced gemini-3.6-flash

Known Issues

  • opencode-zen proxy pitfall documented in omniroute-ops skill
  • Web search proxy failures documented in omniroute-ops skill
  • Combo discovery can differ from resolution;

AutoRoute variants (2026-08-12)

  • auto/best-free (Auto Cheap) and auto/coding-free are virtual AutoRoute variants — they do NOT appear in the persisted combo list (/api/combos returns 54 combos, none named these).
  • The LKGP engine resolves them dynamically: it scores the operator's live connected provider accounts (e.g. antigravity, opencode-zen, nvidia, openrouter) rather than a hardcoded candidate list.
  • Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
  • Example observed: an auto/best-free request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.

Recent Changes (2026-08-14)

  • OmniRoute factory reset on CT601. Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (auto/best-fast, auto/best-coding, etc.). Old state backed up at /opt/omniroute/factory-reset-backup-20260814-225524.
  • Providers re-added from Notion API vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry.
  • Hermes config default model changed: auto/best-free → resilient-free (via ~/.hermes/config.yaml). auto/best-free was misconfigured — had no combo definition and routed to gpt-5.6-sol (paid) instead of free models. Switched to resilient-free which is a working free combo.
  • Multiple free combos broken by upstream model ID changes:
    • OpenRouter: poolside/laguna-s-2-1:free (dash) → poolside/laguna-s-2.1:free (dot) — model ID format changed
    • Groq: meta-llama/llama-4-scout-17b-16e-instruct — retired, only llama-3.3/3.1 remain
    • Affected combos: resilient-free, free-IA, free-stack, good-free, deepseek-flash-free, mimo-free
    • All route through OpenRouter → fail with 400/404 when trying these model IDs
  • Working free combos as of 2026-08-14: free-lkgp (DeepSeek V4 Flash), mega-free (StepFun 3.7 Flash), smart-route / smart-route-ia (nested combos)
  • LiteLLM routing issue identified: Hermes was previously pointing at http://litellm:4000/v1 with free-auto model, causing 6–26s per call. OmniRoute's auto/best-free tested at 265ms. Config now points at OmniRoute.
  • GPT-5.6 Luna research: Community consensus — Luna is cleaner/more polished; DeepSeek V4 Flash is more persistent/cheaper. Luna better for visual/UI work and final review; DeepSeek better for automation and high-volume executor work. Provider route matters more than model name.

Recent Changes (2026-08-13)

  • OmniRoute migrated CT600 → CT601. Provider renamed custom:omniroute → custom:omni; base URL http://omniroute:20128/v1 → http://omni:20128/v1. Host omni.kangaroo-eel.ts.net (100.93.204.4, active) replaces omniroute.kangaroo-eel.ts.net (100.88.81.19, offline). CT600 stopped, CT601 running. Database path /var/lib/omniroute/storage.sqlite → /opt/omniroute/storage.sqlite. API key unchanged. All 14 config + 22 cron-job custom:omniroute references updated to custom:omni. Pre-migration backup at ~/.hermes/config.yaml.bak.omniroute-removal. Verified live 2026-08-13 (pct list: 600 stopped, 601 running; grep: 0 remaining custom:omniroute refs).

Recent Changes (2026-07-25)

  • free-stack combo updated: Nemotron 3 Ultra 550B model at index 4 now configured with NVIDIA-specific parameters:
    • max_tokens: 8192 (was default 65536, exceeding NVIDIA NIM limit of 16384)
    • reasoning_effort: medium (was high, reducing long internal monologues)
    • chat_template_kwargs.force_nonempty_content: true (prevents empty responses when tool calling + reasoning)
  • Combo ID: 23d25f8f-5eb0-4b6f-b966-e260ca3905b3
  • Hermes config.yaml updated: agent.reasoning_effort: medium, agent.max_tokens: 8192, agent.chat_template_kwargs.force_nonempty_content: true
  • Rationale: NVIDIA NIM API documents max output of 16384 tokens; medium reasoning uses more efficient chat-template mode; force_nonempty_content resolves empty responses with tool calls combo in picker may still route

Recent Changes (2026-08-07)

  • Three-layer root cause for slowness identified and fixed:
    1. npm package .env override: OmniRoute loads three .env files (/var/lib/omniroute/.env, /root/.omniroute/.env, /usr/lib/node_modules/omniroute/.env). The npm package's .env ships with ENABLE_SOCKS5_PROXY=true and npm update re-installs it with that default, silently overriding the /var/lib config. Fix: flip all three to false, then restart. After ANY upgrade, check all three.
    2. Dead provider connection: Connection 58debd59 (prefix nanobot) points at clawtest:8900/v1 — nanobot on clawtest is DOWN. Every model sync + credential health check retried it, burning event-loop time. Disabled it.
    3. PVE host load leaking into CT600: CT600 has nesting=1, so uptime/top inside the container mirror PVE's host load exactly (verified: identical 10.30/9.23/10.37 on both). Same trap as CT241. Judge CT600 CPU from pct exec 600 -- ps -o %cpu or host-side ps, never from in-container load average.
  • First-request-after-cache-expiry stall: /v1/models hangs 10-15s+ on the FIRST hit after the model catalog cache expires (rebuild enumerates all connections, including slow dead ones), then answers in ~0.02s from cache. Don't judge health on one curl — run a burst of 5 and check if later tries are sub-second.
  • is_active=0 does NOT stop CredentialHealth probes: Disabling a dead connection stops it from ModelSync but the health checker still probes it and logs [ProxyFetch] ECONNREFUSED noise. That noise is benign once the endpoint is unreachable.
  • Lessons banked in omniroute-ops skill: references/omniroute-upgrade-procedure.md updated with all three pitfalls.

Recent Changes (2026-08-04)

  • OmniRoute free-DeepSeek silent-stop mitigation: Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances. Reworked both combos: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort. See incidents/2026-08-04-omniroute-free-deepseek-silent-stops.