BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic.
Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an
unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a
mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)".
A scan of all 117 pages found exactly one such line; it is now properly quoted.
Quartz now builds cleanly: 690 files emitted and deployed.
RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page).
The page claimed the recall-ranking fix "was not present in live config" and that
conflict 152 was "the only open conflict". Both claims were false:
- The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5,
fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 —
not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the
2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks.
- The original regression test now passes: "mnemosyne recall 'Are you up to date'"
returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale
Uber Eats episode no longer outranks anything.
- Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise.
All three open items closed; page moved to status: resolved.
LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01):
- infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping
times out); m93p peer record reads false for the 4th consecutive time while the
host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages
refreshed; no IP changes, no missing peers.
- infrastructure/docker-services.md: CT201's Tailscale IP was wrong
(100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale
worldmonitor-seeder-fast entry removed from the CT200 list, which is now
regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to
thirsty_payne.
- infrastructure/proxmox.md: storage percentages refreshed on both hosts from live
pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an
elevated load average recorded.
- systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md:
re-probed and verified_on bumped.
- systems/skills-index.md: regenerated — 301 unique skills in 68 categories,
46 archived. (301 is the unique (category, skill_name) count; find returns 302
files due to the known nested obsidian-git-sync alias duplicate.)
Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
Third failure mode on the same node, and it is not the 503 string.dca8482e4f76 (Daily Marketing Job
Radar) failed on 2026-10-21 at 07:40:48 after 40 m 36 s with
RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s. Zero HTTP 503 … resource pressure rows exist for that day (last: 2026-10-19 09:02) and the
sibling 07:00 job 85edf6772bca completed normally, so the node was serving work at that hour.
→ incidents/2026-10-21-omniroute-pre-dispatch-filter-skip
One pin, three signatures in eight days: connection-refused storm (10-14/15), resource-pressure 503s
(10-16/17 and 10-19), pre-dispatch filter skip (10-21) — all on custom:omni-101, all with
model: smart-route. No job in the fleet has a second provider pin.
Router-side telemetry is still missing for every one of them — omni-101 host logs have not been
captured during a failure, which is why none of the three has a confirmed root cause.
Recent Changes (2026-10-20)
Recurring 503 … resource pressure load-shedding on omniroute-1 — second burst in four days. On
2026-10-19 the node failed 9 model calls across 8 jobs (06:01, then 07:01 → 12:01, plus 4 derivative
Mnemosyne Watchdog reds) with
RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly., while
/api/health answered 200 {"status":"ok"} and /v1/models answered 401 throughout. Burst 1 ran
2026-10-16 08:00 → 10-17 03:00 with the identical error string and the identical 9-call blast radius.
Both bursts self-cleared; nothing was changed on CT600 or omniroute-1 in between.
Root cause unknown — no host telemetry was captured during either burst.
→ incidents/2026-10-16-omniroute-503-resource-pressure
Every failing job is pinned to the same alias on the same node. All eight read
provider: custom:omni-101, model: smart-route in ~/.hermes/cron/jobs.json (verified 2026-10-20).
With no second provider pin and no router-side failover between omniroute-1 and CT600 omniroute, one
node's load-shedding blanks the fleet's digest slots.
Probe properly. A green /api/health proves nothing about the model path — it stayed 200 ok through a
~19 h burst and a second 7 h burst. To catch the next one, run a real completion against
custom:omni-101 while jobs are still failing, and capture the node's CPU / RAM / queue depth during the
burst, not afterwards.
Purpose
Provider/router for Hermes — ⚠ updated 2026-10-26: the omniroute-1 node is the DEFAULT chat path again.
Hermes's default provider is custom:omni-101 → http://100.78.63.20:20128 with model smart-route
(reverted 2026-10-24 19:06; the 2026-10-20 → 10-24 window pointed the default at nous).
The CT600 provider key custom:omni (http://omniroute:20128/v1) remains auxiliary-only.
→ systems/current-profile
Current Configuration
Hermes default provider:custom:omni-101 → base URL http://100.78.63.20:20128 (CT101, m93p), model default smart-route — re-verified live 2026-10-31 from ~/.hermes/config.yaml (model.default: smart-route, model.provider: custom:omni-101, model.base_url: http://100.78.63.20:20128)
Secondary provider entry:custom:Omniroute:20128 → base URL http://omniroute:20128/v1 (CT600, pve), referenced by the moa reference-model set (vertex/gemini-3.6-flash, kilo-gateway/kilo-auto/free, nvidia/deepseek-ai/deepseek-v4-pro, freeapi/nemotron-3-ultra-550b) — auxiliary/MoA use, not the main chat default
Host reachability: bare name omniroute resolves to 100.88.81.19; direct HTTP on :20128 answers 401 (auth-required, service live)
Runs on: two live nodes — CT600 (omniroute, Tailscale 100.88.81.19, on pve) and CT101 (omniroute, Tailscale 100.78.63.20, on m93p) — both verified running and listening on 0.0.0.0:20128 on 2026-10-31
DB path:/opt/omniroute/storage.sqlite (was /var/lib/omniroute/storage.sqlite)
Backup: pre-migration config saved at ~/.hermes/config.yaml.bak.omniroute-removal
Correction (2026-10-26): the earlier "Hermes default provider path does not use OmniRoute for main chat" note is stale — the default is OmniRoute again (custom:omni-101, model smart-route). The CT600 key custom:Omniroute:20128 is still auxiliary-only (MoA reference models).
Auxiliary roles in Hermes config:skills_hub, approval, mcp — all on custom:omni-101 with model no-think/freeapi/auto (live-verified 2026-10-31)
Key prefix visible in config:sk-5c67fbdc0... — full value not stored in wiki
CT600 / CT601 / omniroute-1 — actual state (re-verified 2026-10-31)
The wiki previously recorded CT600 as retired and CT601 as the live successor, then CT601 as stopped.
All three framings are now superseded. Live pct list on both hypervisors and live HTTP probes return:
Node
Status (live 2026-10-31)
Tailscale IP
Notes
CT600 omniroute
running
100.88.81.19
On pve. OmniRoute listening on 0.0.0.0:20128; /v1/models and /api/combos both return 401 (alive, auth required)
CT101 omniroute-1
running
100.78.63.20
On m93p. Live pct config 101 → hostname: omniroute; omniroute listening on 0.0.0.0:20128; /v1/models and /api/combos both return 401. Registered in Hermes config as custom:omni-101 — the live default provider
CT601 omni
does not exist
—
/etc/pve/lxc/601.conf absent on bothpve and m93p. The omni peer (100.93.204.4) has expired out of the tailnet entirely — live tailscale status --json has no peer named omni and no peer at that IP
Hermes config resolving to the omniroute hostname (custom:Omniroute:20128 → omniroute:20128) therefore points at CT600; the live default custom:omni-101 points at CT101.
CT601 was deleted, not merely stopped — the earlier "stopped, onboot=0" entry was wrong.
The omniroute-1 identity is now confirmed — it is CT101 on m93p, not "a VM or off-PVE host". The 2026-10-11 note that it was "not visible in pct list" was an artifact of probing only pve.
Two live OmniRoute nodes remain (CT600, CT101), both serving :20128 with separate credentials — a pin's usability must be probed on the node that actually serves it.
Catalog and Routing Facts
Catalog reachable: yes, 47 combos returned at /api/combos
Combo schema observed: name, strategy, models, id; some combos have config, isHidden, uuid
Combo-level context can override individual models
auto/lkgp strategies may silently change chosen models
Model mappings can substitute models silently
Session affinity caches last successful provider in session_model_history
Catalog Evidence (2026-07-22)
Advertised context for gemini/gemini-3.6-flash: 1048576
Related models in catalog: kc/google/gemini-3.6-flash, kilocode/google/gemini-3.6-flash, nous/google/gemini-3.6-flash, opencode-zen/gemini-3.6-flash, vertex/gemini-3.6-flash
No live combo in the returned 47 combos referenced gemini-3.6-flash
Known Issues
opencode-zen proxy pitfall documented in omniroute-ops skill
Web search proxy failures documented in omniroute-ops skill
Combo discovery can differ from resolution;
AutoRoute variants (2026-08-12)
auto/best-free (Auto Cheap) and auto/coding-free are virtual AutoRoute variants — they do NOT appear in the persisted combo list (/api/combos returns 54 combos, none named these).
The LKGP engine resolves them dynamically: it scores the operator's live connected provider accounts (e.g. antigravity, opencode-zen, nvidia, openrouter) rather than a hardcoded candidate list.
Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
Example observed: an auto/best-free request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
A second, different failure mode from the same node. Three days after the 34-hour TCP blackout,
omniroute-1 answered model calls with HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly. — 9 model calls across 7 jobs, between 10-16 08:00:37 and
10-17 03:00:28 (~19 h). → incidents/2026-10-16-omniroute-503-resource-pressure
The control plane looked healthy throughout./api/health returned 200 {"status":"ok"} and
/v1/models returned 401 (auth-required) before, during and after the window — re-probed 2026-10-18
00:33. A green health endpoint does not mean the model path is serving: 503 here is load-shedding
under resource pressure, not a dead process, so process checks miss it.
It recovered on its own; nothing was changed on the node. Last 503 10-17 03:00:28; 10-17
closed 162 completed / 5 failed (all five before 03:01) and 10-18 is green so far. Every affected
job re-ran clean on its next slot — b63ed5817212 10-17 06:01, 08eb86f02d37 10-17 08:31,
ecc1fc13655a and ea79a2107e36 both 10-17 12:02, auto-closing their own incidents.
omniroute-1 (100.78.63.20:20128, provider custom:omni-101) was TCP-unreachable from 2026-10-14
08:30 to 2026-10-15 ~18:07 AWST — httpcore.ConnectError: [Errno 111] Connection refused →
openai.APIConnectionError: Connection error. → RuntimeError: Connection error. Result: 24 genuine
cron model-call failures across 12 jobs plus 12 derivative watchdog rows, and every scheduled digest
for ~34 hours was silently suppressed (including the 10-14 23:59 Wiki Audit).
This node is a single point of failure for the whole scheduled estate. All 18 enabled
agent-mode cron jobs are pinned to custom:omni-101 (smart-route×10, free-stack×6,
deepseek-paid-smart×2). The only other model-capable cron pin — custom:omni (CT600,
omniroute:20128) — is attached to a no_agent job (e530c89f2be0, Evening Stand-Down), so it is
never exercised by a model call; d40085da631c, the only other custom:omni pin, is disabled.
The cron fleet therefore has no model-call failover between the two nodes, and no job that can even
tell you whether the second node is up.
Recovered — verified 2026-10-16 00:1x:GET /v1/models on 100.78.63.20:20128 → HTTP 401 in
10 ms (up, auth required); same on omniroute:20128 / 100.88.81.19. Four jobs have completed on the
node since 2026-10-15 22:01.
OmniRoute factory reset on CT601. Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (auto/best-fast, auto/best-coding, etc.). Old state backed up at /opt/omniroute/factory-reset-backup-20260814-225524.
Providers re-added from Notion API vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry.
Hermes config default model changed:auto/best-free → resilient-free (via ~/.hermes/config.yaml). auto/best-free was misconfigured — had no combo definition and routed to gpt-5.6-sol (paid) instead of free models. Switched to resilient-free which is a working free combo.
Multiple free combos broken by upstream model ID changes:
OpenRouter: poolside/laguna-s-2-1:free (dash) → poolside/laguna-s-2.1:free (dot) — model ID format changed
Groq: meta-llama/llama-4-scout-17b-16e-instruct — retired, only llama-3.3/3.1 remain
All route through OpenRouter → fail with 400/404 when trying these model IDs
Working free combos as of 2026-08-14:free-lkgp (DeepSeek V4 Flash), mega-free (StepFun 3.7 Flash), smart-route / smart-route-ia (nested combos)
LiteLLM routing issue identified: Hermes was previously pointing at http://litellm:4000/v1 with free-auto model, causing 6–26s per call. OmniRoute's auto/best-free tested at 265ms. Config now points at OmniRoute.
GPT-5.6 Luna research: Community consensus — Luna is cleaner/more polished; DeepSeek V4 Flash is more persistent/cheaper. Luna better for visual/UI work and final review; DeepSeek better for automation and high-volume executor work. Provider route matters more than model name.
Rationale: NVIDIA NIM API documents max output of 16384 tokens; medium reasoning uses more efficient chat-template mode; force_nonempty_content resolves empty responses with tool calls combo in picker may still route
Recent Changes (2026-08-07)
Three-layer root cause for slowness identified and fixed:
npm package .env override: OmniRoute loads three .env files (/var/lib/omniroute/.env, /root/.omniroute/.env, /usr/lib/node_modules/omniroute/.env). The npm package's .env ships with ENABLE_SOCKS5_PROXY=true and npm update re-installs it with that default, silently overriding the /var/lib config. Fix: flip all three to false, then restart. After ANY upgrade, check all three.
Dead provider connection: Connection 58debd59 (prefix nanobot) points at clawtest:8900/v1 — nanobot on clawtest is DOWN. Every model sync + credential health check retried it, burning event-loop time. Disabled it.
PVE host load leaking into CT600: CT600 has nesting=1, so uptime/top inside the container mirror PVE's host load exactly (verified: identical 10.30/9.23/10.37 on both). Same trap as CT241. Judge CT600 CPU from pct exec 600 -- ps -o %cpu or host-side ps, never from in-container load average.
First-request-after-cache-expiry stall:/v1/models hangs 10-15s+ on the FIRST hit after the model catalog cache expires (rebuild enumerates all connections, including slow dead ones), then answers in ~0.02s from cache. Don't judge health on one curl — run a burst of 5 and check if later tries are sub-second.
is_active=0 does NOT stop CredentialHealth probes: Disabling a dead connection stops it from ModelSync but the health checker still probes it and logs [ProxyFetch] ECONNREFUSED noise. That noise is benign once the endpoint is unreachable.
Lessons banked in omniroute-ops skill:references/omniroute-upgrade-procedure.md updated with all three pitfalls.
Recent Changes (2026-08-04)
OmniRoute free-DeepSeek silent-stop mitigation: Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances. Reworked both combos: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort. See incidents/2026-08-04-omniroute-free-deepseek-silent-stops.