wiki-maintain.sh ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted, service restarted). The freshness report listed 8 stale pages; each was diffed against live state and corrected only where reality could be verified. A second script run rewrote the same-day report with 0 stale / 103 fresh / 4 no-date, so the committed report matches the tree it ships with. - infrastructure/hosts.md: CT600 is LIVE (not retired); CT601 does not exist on either hypervisor; CT333 verified running; both hypervisors verified; config 33->51; skills 66->302; cron 25->28 (25 enabled / 3 disabled); Xvfb :102->:99; providers split into Omniroute:20128 (CT600) and omni-101 (CT101, default). - agents/hermes-production.md: Tailscale IP 100.96.100.82 -> 100.118.5.51 (the old value is the PVE hypervisor); v0.21.5 -> v0.22.3; dead 8082 claim removed; fallback chain rewritten from live config (first entry is the dead nous credential); all 8 auxiliary slots now no-think/freeapi/auto; cron 30->28. - agents/hermes-chaos.md: v0.22.3 (same build as CT460); "3 commits behind" -> 0 behind origin/master; default model smart-route on custom:omni-101; cron "11 paused" -> 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460. - infrastructure/tailscale.md: node table rebuilt from status --json; octoagent and omarchymac-1 were missing entirely (52 -> 54 non-funnel nodes); m93p now online; every offline age refreshed; omni row corrected to "CT601 removed from both hypervisors"; Serve tables re-probed per host. - systems/memory-backend.md: re-verified against live config; coverage 1899/2350 (80.8%), working 96.8%, episodic 99.4% — above the 95% reindex threshold. - runbooks/provider-health.md: the step-3 OmniRoute snippet raised IndexError (provider renamed omniroute -> Omniroute:20128; key moved to key_env) — fixed; new step 3b documents the omni-101 /v1/models false-failure quirk. - runbooks/update-hermes-safely.md: added managed-runtime dependency re-assert step; clarified --branch vs stable channel (v0.22.5 available). - incidents/2026-10-03-ct450-hermes-update.md + incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md: Current-state sections recording current versions and embedding coverage. All 9 corrected pages verified 200 at http://127.0.0.1:9120/<path>. Vault mirror at /home/hermes/wiki synced for all touched files (22 copies, sha256-verified) and committed locally; the vault is local-only and is not pushed. No secrets written — IDs, URLs, key env var names, counts and timestamps only.
4.8 KiB
title, type, status, created, updated, verified_on, last_tested, confidence, tags, sources
| title | type | status | created | updated | verified_on | last_tested | confidence | tags | sources | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Provider Health Check | runbook | active | 2026-07-22 | 2026-10-26 | 2026-10-26 | 2026-10-26 | high |
|
|
Provider Health Check
Purpose
Verify Hermes can reach LiteLLM, OmniRoute, OpenRouter-style fallbacks, and the WebUI, and report which paths are healthy or degraded.
Symptoms that match this runbook
- Models list is slow or blank in Hermes
- Fallback model retries happen immediately
- Chat returns 5xx or gateway errors
Prerequisites
- Network access from this host to
litellm:4000,omniroute:20128,opencode.ai,api.telegram.org, Telegram fallback IPs - Hermes config located at
~/.hermes/config.yaml
Procedure
1. Check local Hermes services
hermes gateway status
hermes dashboard status
2. Check main provider reachability
python3 -c "import urllib.request; urllib.request.urlopen('http://litellm:4000/v1/models', timeout=4)"
Current install behavior: LiteLLM responds but requires auth, so HTTP 401 is expected from an unauthenticated check.
3. Check OmniRoute reachability
python3 -c "
import urllib.request, json, yaml
with open('/home/hermes/.hermes/config.yaml') as f:
cfg = yaml.safe_load(f)
import os
# NOTE: the provider was renamed 'omniroute' -> 'Omniroute:20128' and the key moved
# from an inline 'api_key' to a 'key_env' indirection. The old snippet raised IndexError.
p = [x for x in cfg.get('custom_providers',[]) if x.get('name')=='Omniroute:20128'][0]
key = os.environ[p['key_env']]
req = urllib.request.Request(p['base_url'] + '/models', headers={'Authorization': f'Bearer {key}'})
with urllib.request.urlopen(req, timeout=5) as r:
data = json.loads(r.read())
print('OmniRoute OK, models:', len(data.get('data', [])))
"
Current verified result (2026-10-26): reachable and returns a non-empty model catalog — 6,480 models on omniroute:20128 (CT600, 100.88.81.19).
3b. Check the default omni-101 provider (CT101, 100.78.63.20)
The default provider is custom:omni-101 → http://100.78.63.20:20128, keyed via
HERMES_CUSTOM_100_78_63_20_20128_API_KEY. Probe it with its own key — the CT600 key is rejected here:
curl -s --max-time 8 -H "Authorization: Bearer $HERMES_CUSTOM_100_78_63_20_20128_API_KEY" \
http://100.78.63.20:20128/v1/models | head -c 200
⚠ Known quirk (verified 2026-10-26): /v1/models on this node returns
Invalid API key even with the correct key, while /v1/chat/completions works
(a smart-route call returned minimaxai/minimax-m3.1-flash). So a /v1/models
check on omni-101 will read as a false failure — smoke-test the chat endpoint instead.
4. Check fallback provider endpoints
curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://opencode.ai/zen/v1/models
curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://inference-api.nousresearch.com/v1/models
curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://ollama.com/v1/models
Any non-200 here means that fallback path is currently offline.
5. Check WebUI health
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8787/health
Expected: 200
6. Check gateway platform reachability
Tail the gateway journal for errors:
journalctl --user -u hermes-gateway.service -n 100 --no-pager
Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or Email fetch.
Verification
- Gateway status shows
active (running). - LiteLLM 401 from unauthenticated check is expected.
- OmniRoute returns model count > 0.
- Fallback URLs return 200 or expected auth codes.
- WebUI
/healthreturns200. - Gateway journal shows no red alarm pattern.
Current install references
- Hermes: v0.22.3 (2026.9.24), local
5b147ab036d6, git install:/home/hermes/.hermes/hermes-agent(hermes update --checkreports stable v0.22.5 available) - LiteLLM base URL:
http://litellm:4000/v1 - OmniRoute base URL:
http://omniroute:20128/v1(provider nameOmniroute:20128, key envHERMES_CUSTOM_OMNIROUTE_20128_API_KEY) - OmniRoute direct IP (CT600):
100.88.81.19:20128 - Default provider
omni-101(CT101):http://100.78.63.20:20128(key envHERMES_CUSTOM_100_78_63_20_20128_API_KEY) — ⚠/v1/modelsreturns a falseInvalid API key; chat works - WebUI:
http://0.0.0.0:8787/health
Rollback
- This read-only runbook has no rollback side effects
- If you change fallback providers during remediation, record and revert via
hermes fallback list+ previous wiki state
Last tested
2026-10-26 — LiteLLM 401 unauthed (expected), OmniRoute/CT600 6,480 models, WebUI /health 200, opencode/nous/ollama fallbacks all 200