Third failure mode on the same node, and it is not the 503 string.dca8482e4f76 (Daily Marketing Job
Radar) failed on 2026-10-21 at 07:40:48 after 40 m 36 s with
RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s. Zero HTTP 503 … resource pressure rows exist for that day (last: 2026-10-19 09:02) and the
sibling 07:00 job 85edf6772bca completed normally, so the node was serving work at that hour.
→ incidents/2026-10-21-omniroute-pre-dispatch-filter-skip
One pin, three signatures in eight days: connection-refused storm (10-14/15), resource-pressure 503s
(10-16/17 and 10-19), pre-dispatch filter skip (10-21) — all on custom:omni-101, all with
model: smart-route. No job in the fleet has a second provider pin.
Router-side telemetry is still missing for every one of them — omni-101 host logs have not been
captured during a failure, which is why none of the three has a confirmed root cause.
Recent Changes (2026-10-20)
Recurring 503 … resource pressure load-shedding on omniroute-1 — second burst in four days. On
2026-10-19 the node failed 9 model calls across 8 jobs (06:01, then 07:01 → 12:01, plus 4 derivative
Mnemosyne Watchdog reds) with
RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly., while
/api/health answered 200 {"status":"ok"} and /v1/models answered 401 throughout. Burst 1 ran
2026-10-16 08:00 → 10-17 03:00 with the identical error string and the identical 9-call blast radius.
Both bursts self-cleared; nothing was changed on CT600 or omniroute-1 in between.
Root cause unknown — no host telemetry was captured during either burst.
→ incidents/2026-10-16-omniroute-503-resource-pressure
Every failing job is pinned to the same alias on the same node. All eight read
provider: custom:omni-101, model: smart-route in ~/.hermes/cron/jobs.json (verified 2026-10-20).
With no second provider pin and no router-side failover between omniroute-1 and CT600 omniroute, one
node's load-shedding blanks the fleet's digest slots.
Probe properly. A green /api/health proves nothing about the model path — it stayed 200 ok through a
~19 h burst and a second 7 h burst. To catch the next one, run a real completion against
custom:omni-101 while jobs are still failing, and capture the node's CPU / RAM / queue depth during the
burst, not afterwards.
Purpose
Secondary provider/router for Hermes auxiliary roles and some agent routing.
Current Configuration
Hermes provider name:custom:omni (maps to provider omniroute:20128)
Base URL:http://omniroute:20128/v1
Host reachability: bare name omniroute resolves to 100.88.81.19; direct HTTP on :20128 answers 401 (auth-required, service live)
DB path:/opt/omniroute/storage.sqlite (was /var/lib/omniroute/storage.sqlite)
Backup: pre-migration config saved at ~/.hermes/config.yaml.bak.omniroute-removal
Miss: Hermes default provider path does not use OmniRoute for main chat; used for auxiliary roles instead
Auxiliary roles in Hermes config:skills_hub, approval, mcp
Key prefix visible in config:sk-5c67fbdc0... — full value not stored in wiki
CT600 / CT601 / omniroute-1 — actual state (2026-10-11)
The wiki previously recorded CT600 as retired and CT601 as the live successor. That is no longer true. Live PVE and Tailscale checks return:
Node
Status (live 2026-10-11)
Tailscale IP
Notes
CT600 omniroute
running, onboot=1
100.88.81.19
OmniRoute listening on 0.0.0.0:20128; /v1/models returns 401 (alive, auth required)
CT601 omni
stopped, onboot=0
100.93.204.4
Tailscale peer last seen 54d ago
omniroute-1
active (not in PVE pct list)
100.78.63.20
Third OmniRoute node; /v1/models returns 401. Registered in Hermes config as custom provider omni-101, which carries skills_hub (model free-stack, 5384 models, default smart-route)
Hermes config resolving to the omniroute hostname (custom:omni → omniroute:20128) therefore points at CT600, not CT601.
CT600's combo API is reachable with its local key and returns combos (e.g. go-pro), confirming it is the serving instance.
Do not treat CT601 as the production OmniRoute node. It is stopped with onboot=0.
Treat omniroute-1 (100.78.63.20) as a second live instance used by the omni-101 provider. It is not visible in pct list, so it is likely a VM or off-PVE host; its identity was not confirmed.
Catalog and Routing Facts
Catalog reachable: yes, 47 combos returned at /api/combos
Combo schema observed: name, strategy, models, id; some combos have config, isHidden, uuid
Combo-level context can override individual models
auto/lkgp strategies may silently change chosen models
Model mappings can substitute models silently
Session affinity caches last successful provider in session_model_history
Catalog Evidence (2026-07-22)
Advertised context for gemini/gemini-3.6-flash: 1048576
Related models in catalog: kc/google/gemini-3.6-flash, kilocode/google/gemini-3.6-flash, nous/google/gemini-3.6-flash, opencode-zen/gemini-3.6-flash, vertex/gemini-3.6-flash
No live combo in the returned 47 combos referenced gemini-3.6-flash
Known Issues
opencode-zen proxy pitfall documented in omniroute-ops skill
Web search proxy failures documented in omniroute-ops skill
Combo discovery can differ from resolution;
AutoRoute variants (2026-08-12)
auto/best-free (Auto Cheap) and auto/coding-free are virtual AutoRoute variants — they do NOT appear in the persisted combo list (/api/combos returns 54 combos, none named these).
The LKGP engine resolves them dynamically: it scores the operator's live connected provider accounts (e.g. antigravity, opencode-zen, nvidia, openrouter) rather than a hardcoded candidate list.
Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
Example observed: an auto/best-free request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
A second, different failure mode from the same node. Three days after the 34-hour TCP blackout,
omniroute-1 answered model calls with HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly. — 9 model calls across 7 jobs, between 10-16 08:00:37 and
10-17 03:00:28 (~19 h). → incidents/2026-10-16-omniroute-503-resource-pressure
The control plane looked healthy throughout./api/health returned 200 {"status":"ok"} and
/v1/models returned 401 (auth-required) before, during and after the window — re-probed 2026-10-18
00:33. A green health endpoint does not mean the model path is serving: 503 here is load-shedding
under resource pressure, not a dead process, so process checks miss it.
It recovered on its own; nothing was changed on the node. Last 503 10-17 03:00:28; 10-17
closed 162 completed / 5 failed (all five before 03:01) and 10-18 is green so far. Every affected
job re-ran clean on its next slot — b63ed5817212 10-17 06:01, 08eb86f02d37 10-17 08:31,
ecc1fc13655a and ea79a2107e36 both 10-17 12:02, auto-closing their own incidents.
omniroute-1 (100.78.63.20:20128, provider custom:omni-101) was TCP-unreachable from 2026-10-14
08:30 to 2026-10-15 ~18:07 AWST — httpcore.ConnectError: [Errno 111] Connection refused →
openai.APIConnectionError: Connection error. → RuntimeError: Connection error. Result: 24 genuine
cron model-call failures across 12 jobs plus 12 derivative watchdog rows, and every scheduled digest
for ~34 hours was silently suppressed (including the 10-14 23:59 Wiki Audit).
This node is a single point of failure for the whole scheduled estate. All 18 enabled
agent-mode cron jobs are pinned to custom:omni-101 (smart-route×10, free-stack×6,
deepseek-paid-smart×2). The only other model-capable cron pin — custom:omni (CT600,
omniroute:20128) — is attached to a no_agent job (e530c89f2be0, Evening Stand-Down), so it is
never exercised by a model call; d40085da631c, the only other custom:omni pin, is disabled.
The cron fleet therefore has no model-call failover between the two nodes, and no job that can even
tell you whether the second node is up.
Recovered — verified 2026-10-16 00:1x:GET /v1/models on 100.78.63.20:20128 → HTTP 401 in
10 ms (up, auth required); same on omniroute:20128 / 100.88.81.19. Four jobs have completed on the
node since 2026-10-15 22:01.
OmniRoute factory reset on CT601. Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (auto/best-fast, auto/best-coding, etc.). Old state backed up at /opt/omniroute/factory-reset-backup-20260814-225524.
Providers re-added from Notion API vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry.
Hermes config default model changed:auto/best-free → resilient-free (via ~/.hermes/config.yaml). auto/best-free was misconfigured — had no combo definition and routed to gpt-5.6-sol (paid) instead of free models. Switched to resilient-free which is a working free combo.
Multiple free combos broken by upstream model ID changes:
OpenRouter: poolside/laguna-s-2-1:free (dash) → poolside/laguna-s-2.1:free (dot) — model ID format changed
Groq: meta-llama/llama-4-scout-17b-16e-instruct — retired, only llama-3.3/3.1 remain
All route through OpenRouter → fail with 400/404 when trying these model IDs
Working free combos as of 2026-08-14:free-lkgp (DeepSeek V4 Flash), mega-free (StepFun 3.7 Flash), smart-route / smart-route-ia (nested combos)
LiteLLM routing issue identified: Hermes was previously pointing at http://litellm:4000/v1 with free-auto model, causing 6–26s per call. OmniRoute's auto/best-free tested at 265ms. Config now points at OmniRoute.
GPT-5.6 Luna research: Community consensus — Luna is cleaner/more polished; DeepSeek V4 Flash is more persistent/cheaper. Luna better for visual/UI work and final review; DeepSeek better for automation and high-volume executor work. Provider route matters more than model name.
Rationale: NVIDIA NIM API documents max output of 16384 tokens; medium reasoning uses more efficient chat-template mode; force_nonempty_content resolves empty responses with tool calls combo in picker may still route
Recent Changes (2026-08-07)
Three-layer root cause for slowness identified and fixed:
npm package .env override: OmniRoute loads three .env files (/var/lib/omniroute/.env, /root/.omniroute/.env, /usr/lib/node_modules/omniroute/.env). The npm package's .env ships with ENABLE_SOCKS5_PROXY=true and npm update re-installs it with that default, silently overriding the /var/lib config. Fix: flip all three to false, then restart. After ANY upgrade, check all three.
Dead provider connection: Connection 58debd59 (prefix nanobot) points at clawtest:8900/v1 — nanobot on clawtest is DOWN. Every model sync + credential health check retried it, burning event-loop time. Disabled it.
PVE host load leaking into CT600: CT600 has nesting=1, so uptime/top inside the container mirror PVE's host load exactly (verified: identical 10.30/9.23/10.37 on both). Same trap as CT241. Judge CT600 CPU from pct exec 600 -- ps -o %cpu or host-side ps, never from in-container load average.
First-request-after-cache-expiry stall:/v1/models hangs 10-15s+ on the FIRST hit after the model catalog cache expires (rebuild enumerates all connections, including slow dead ones), then answers in ~0.02s from cache. Don't judge health on one curl — run a burst of 5 and check if later tries are sub-second.
is_active=0 does NOT stop CredentialHealth probes: Disabling a dead connection stops it from ModelSync but the health checker still probes it and logs [ProxyFetch] ECONNREFUSED noise. That noise is benign once the endpoint is unreachable.
Lessons banked in omniroute-ops skill:references/omniroute-upgrade-procedure.md updated with all three pitfalls.
Recent Changes (2026-08-04)
OmniRoute free-DeepSeek silent-stop mitigation: Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances. Reworked both combos: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort. See incidents/2026-08-04-omniroute-free-deepseek-silent-stops.