Drift found and corrected against live sources (not guessed):
- OmniRoute: CT600 omniroute is RUNNING and serving :20128 (combo API live);
CT601 omni is STOPPED with onboot=0 (peer offline 54d). Wiki had these inverted.
Hermes custom:omni resolves to omniroute -> CT600, not CT601.
- Third OmniRoute node discovered: omniroute-1 at 100.78.63.20:20128, live (401),
registered as custom provider omni-101 (5384 models, default smart-route,
carries skills_hub/free-stack). Not in pct list; identity unconfirmed.
- Ignis: TWO live instances. CT460 ignis (:8080, vault wiki/agent-estate-wiki) and
CT200 ignis-ignis-1 (image ignis-server:latest, host 8092, vault /opt/ignis/vaults).
Both return HTTP 200. Page now distinguishes them.
- Tailscale: /agent-estate targets bare 127.0.0.1:8769 (was .../8769/agent-estate-role-dashboard.html);
undocumented tailnet-only :9443 -> 127.0.0.1:5173 added; CT201 routes corrected
(8085 targets + /steel 5173 + /steel-api 3000, replacing the wrong :8443 -> :8082 row).
- Proxmox: rebuilt from live pct list + qm list. Added CT475 agentzero and VM110 omarchy
(both previously missing); 282 ubuntu-server-xfce corrected to stopped; dead rows removed.
VM403 open-claw and VM401 hermes-old absent from qm list - flagged unconfirmed, not deleted.
- Nanobot: gateway nanobot-gateway (PID 2259048) alive on 8765/8766. It is NOT systemd
nanobot.service/gateway.service (both return inactive - wrong unit names). Port 8900
no longer listening; "Serve API" claim flagged unverified.
- Docker: old "runtime not available" was a CT460 socket artifact (root:docker srw-rw----,
hermes not in group docker, no passwordless sudo in cron). Documented CT200 (~50 containers),
CT333 (9 containers), CT460 via sudo.
- Mattermost CT255 confirmed stopped; matches Tailscale peer offline 21d.
Flagged, not investigated (no root cause):
- mycompose-db-1 on CT200 is in Restarting (1) - crash-looping.
62 pages remain >21d stale and were NOT rewritten this pass (concepts, decisions,
comparisons, most runbooks) - not probed against a live source, so reported not faked.
Quartz rebuild step still inert (no quartz.config.ts) - public/ not refreshed.
Artifacts: systems/skills-index.md regenerated, raw/stocktake/freshness-20261011.txt.
DB path:/opt/omniroute/storage.sqlite (was /var/lib/omniroute/storage.sqlite)
Backup: pre-migration config saved at ~/.hermes/config.yaml.bak.omniroute-removal
Miss: Hermes default provider path does not use OmniRoute for main chat; used for auxiliary roles instead
Auxiliary roles in Hermes config:skills_hub, approval, mcp
Key prefix visible in config:sk-5c67fbdc0... — full value not stored in wiki
CT600 / CT601 / omniroute-1 — actual state (2026-10-11)
The wiki previously recorded CT600 as retired and CT601 as the live successor. That is no longer true. Live PVE and Tailscale checks return:
Node
Status (live 2026-10-11)
Tailscale IP
Notes
CT600 omniroute
running, onboot=1
100.88.81.19
OmniRoute listening on 0.0.0.0:20128; /v1/models returns 401 (alive, auth required)
CT601 omni
stopped, onboot=0
100.93.204.4
Tailscale peer last seen 54d ago
omniroute-1
active (not in PVE pct list)
100.78.63.20
Third OmniRoute node; /v1/models returns 401. Registered in Hermes config as custom provider omni-101, which carries skills_hub (model free-stack, 5384 models, default smart-route)
Hermes config resolving to the omniroute hostname (custom:omni → omniroute:20128) therefore points at CT600, not CT601.
CT600's combo API is reachable with its local key and returns combos (e.g. go-pro), confirming it is the serving instance.
Do not treat CT601 as the production OmniRoute node. It is stopped with onboot=0.
Treat omniroute-1 (100.78.63.20) as a second live instance used by the omni-101 provider. It is not visible in pct list, so it is likely a VM or off-PVE host; its identity was not confirmed.
Catalog and Routing Facts
Catalog reachable: yes, 47 combos returned at /api/combos
Combo schema observed: name, strategy, models, id; some combos have config, isHidden, uuid
Combo-level context can override individual models
auto/lkgp strategies may silently change chosen models
Model mappings can substitute models silently
Session affinity caches last successful provider in session_model_history
Catalog Evidence (2026-07-22)
Advertised context for gemini/gemini-3.6-flash: 1048576
Related models in catalog: kc/google/gemini-3.6-flash, kilocode/google/gemini-3.6-flash, nous/google/gemini-3.6-flash, opencode-zen/gemini-3.6-flash, vertex/gemini-3.6-flash
No live combo in the returned 47 combos referenced gemini-3.6-flash
Known Issues
opencode-zen proxy pitfall documented in omniroute-ops skill
Web search proxy failures documented in omniroute-ops skill
Combo discovery can differ from resolution;
AutoRoute variants (2026-08-12)
auto/best-free (Auto Cheap) and auto/coding-free are virtual AutoRoute variants — they do NOT appear in the persisted combo list (/api/combos returns 54 combos, none named these).
The LKGP engine resolves them dynamically: it scores the operator's live connected provider accounts (e.g. antigravity, opencode-zen, nvidia, openrouter) rather than a hardcoded candidate list.
Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
Example observed: an auto/best-free request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
Recent Changes (2026-08-14)
OmniRoute factory reset on CT601. Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (auto/best-fast, auto/best-coding, etc.). Old state backed up at /opt/omniroute/factory-reset-backup-20260814-225524.
Providers re-added from Notion API vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry.
Hermes config default model changed:auto/best-free → resilient-free (via ~/.hermes/config.yaml). auto/best-free was misconfigured — had no combo definition and routed to gpt-5.6-sol (paid) instead of free models. Switched to resilient-free which is a working free combo.
Multiple free combos broken by upstream model ID changes:
OpenRouter: poolside/laguna-s-2-1:free (dash) → poolside/laguna-s-2.1:free (dot) — model ID format changed
Groq: meta-llama/llama-4-scout-17b-16e-instruct — retired, only llama-3.3/3.1 remain
All route through OpenRouter → fail with 400/404 when trying these model IDs
Working free combos as of 2026-08-14:free-lkgp (DeepSeek V4 Flash), mega-free (StepFun 3.7 Flash), smart-route / smart-route-ia (nested combos)
LiteLLM routing issue identified: Hermes was previously pointing at http://litellm:4000/v1 with free-auto model, causing 6–26s per call. OmniRoute's auto/best-free tested at 265ms. Config now points at OmniRoute.
GPT-5.6 Luna research: Community consensus — Luna is cleaner/more polished; DeepSeek V4 Flash is more persistent/cheaper. Luna better for visual/UI work and final review; DeepSeek better for automation and high-volume executor work. Provider route matters more than model name.
Rationale: NVIDIA NIM API documents max output of 16384 tokens; medium reasoning uses more efficient chat-template mode; force_nonempty_content resolves empty responses with tool calls combo in picker may still route
Recent Changes (2026-08-07)
Three-layer root cause for slowness identified and fixed:
npm package .env override: OmniRoute loads three .env files (/var/lib/omniroute/.env, /root/.omniroute/.env, /usr/lib/node_modules/omniroute/.env). The npm package's .env ships with ENABLE_SOCKS5_PROXY=true and npm update re-installs it with that default, silently overriding the /var/lib config. Fix: flip all three to false, then restart. After ANY upgrade, check all three.
Dead provider connection: Connection 58debd59 (prefix nanobot) points at clawtest:8900/v1 — nanobot on clawtest is DOWN. Every model sync + credential health check retried it, burning event-loop time. Disabled it.
PVE host load leaking into CT600: CT600 has nesting=1, so uptime/top inside the container mirror PVE's host load exactly (verified: identical 10.30/9.23/10.37 on both). Same trap as CT241. Judge CT600 CPU from pct exec 600 -- ps -o %cpu or host-side ps, never from in-container load average.
First-request-after-cache-expiry stall:/v1/models hangs 10-15s+ on the FIRST hit after the model catalog cache expires (rebuild enumerates all connections, including slow dead ones), then answers in ~0.02s from cache. Don't judge health on one curl — run a burst of 5 and check if later tries are sub-second.
is_active=0 does NOT stop CredentialHealth probes: Disabling a dead connection stops it from ModelSync but the health checker still probes it and logs [ProxyFetch] ECONNREFUSED noise. That noise is benign once the endpoint is unreachable.
Lessons banked in omniroute-ops skill:references/omniroute-upgrade-procedure.md updated with all three pitfalls.
Recent Changes (2026-08-04)
OmniRoute free-DeepSeek silent-stop mitigation: Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances. Reworked both combos: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort. See incidents/2026-08-04-omniroute-free-deepseek-silent-stops.