--- title: Provider Health Check type: runbook status: active created: 2026-07-22 updated: 2026-11-18 verified_on: 2026-11-18 last_tested: 2026-10-26 confidence: high tags: [runbook, providers, health, routing] sources: [raw/configs/hermes-config-sanitized.txt] --- # Provider Health Check ## Purpose Verify Hermes can reach LiteLLM, OmniRoute, OpenRouter-style fallbacks, and the WebUI, and report which paths are healthy or degraded. ## Symptoms that match this runbook - Models list is slow or blank in Hermes - Fallback model retries happen immediately - Chat returns 5xx or gateway errors ## Prerequisites - Network access from this host to `litellm:4000`, `omniroute:20128`, `opencode.ai`, `api.telegram.org`, Telegram fallback IPs - Hermes config located at `~/.hermes/config.yaml` ## Procedure ### 1. Check local Hermes services ```bash hermes gateway status hermes dashboard status ``` ### 2. Check main provider reachability ```bash python3 -c "import urllib.request; urllib.request.urlopen('http://litellm:4000/v1/models', timeout=4)" ``` Current install behavior: LiteLLM responds but requires auth, so HTTP 401 is expected from an unauthenticated check. ### 3. Check OmniRoute reachability ```bash python3 -c " import urllib.request, json, yaml with open('/home/hermes/.hermes/config.yaml') as f: cfg = yaml.safe_load(f) import os # NOTE: the provider was renamed 'omniroute' -> 'Omniroute:20128' and the key moved # from an inline 'api_key' to a 'key_env' indirection. The old snippet raised IndexError. p = [x for x in cfg.get('custom_providers',[]) if x.get('name')=='Omniroute:20128'][0] key = os.environ[p['key_env']] req = urllib.request.Request(p['base_url'] + '/models', headers={'Authorization': f'Bearer {key}'}) with urllib.request.urlopen(req, timeout=5) as r: data = json.loads(r.read()) print('OmniRoute OK, models:', len(data.get('data', []))) " ``` Current verified result (2026-10-26): reachable and returns a non-empty model catalog — **6,480 models** on `omniroute:20128` (CT600, 100.88.81.19). > **Re-executed 2026-11-18:** step-3 snippet OK — CT600 `/models` → 200, **6,501 models** (3.6 s; one earlier attempt at the default 5 s timeout timed out, so give this probe headroom). CT600 `/api/health` → 200; `omni-101` `/api/health` → 200; fallback endpoints `opencode.ai` / `inference-api.nousresearch.com` / `ollama.com` all → 200. ### 3b. Check the default `omni-101` provider (CT101, 100.78.63.20) The **default provider** is `custom:omni-101` → `http://100.78.63.20:20128`, keyed via `HERMES_CUSTOM_100_78_63_20_20128_API_KEY`. Probe it with its **own** key — the CT600 key is rejected here: ```bash curl -s --max-time 8 -H "Authorization: Bearer $HERMES_CUSTOM_100_78_63_20_20128_API_KEY" \ http://100.78.63.20:20128/v1/models | head -c 200 ``` ⚠ **Known quirk (verified 2026-10-26):** `/v1/models` on this node returns `Invalid API key` even with the correct key, while `/v1/chat/completions` **works** (a `smart-route` call returned `minimaxai/minimax-m3.1-flash`). So a `/v1/models` check on `omni-101` will read as a false failure — **smoke-test the chat endpoint instead**. ### 4. Check fallback provider endpoints ```bash curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://opencode.ai/zen/v1/models curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://inference-api.nousresearch.com/v1/models curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://ollama.com/v1/models ``` Any non-200 here means that fallback path is currently offline. ### 5. Check WebUI health ```bash curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8787/health ``` Expected: `200` ### 6. Check gateway platform reachability Tail the gateway journal for errors: ```bash journalctl --user -u hermes-gateway.service -n 100 --no-pager ``` Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or Email fetch. ## Verification - Gateway status shows `active (running)`. - LiteLLM 401 from unauthenticated check is expected. - OmniRoute returns model count > 0. - Fallback URLs return 200 or expected auth codes. - WebUI `/health` returns `200`. - Gateway journal shows no red alarm pattern. ## Current install references - Hermes: **v0.22.3** (2026.9.24), local `5b147ab036d6`, git install: `/home/hermes/.hermes/hermes-agent` (`hermes update --check` reports stable **v0.22.5** available) - LiteLLM base URL: `http://litellm:4000/v1` - OmniRoute base URL: `http://omniroute:20128/v1` (provider name `Omniroute:20128`, key env `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) - OmniRoute direct IP (CT600): `100.88.81.19:20128` - Default provider `omni-101` (CT101): `http://100.78.63.20:20128` (key env `HERMES_CUSTOM_100_78_63_20_20128_API_KEY`) — ⚠ `/v1/models` returns a false `Invalid API key`; chat works - WebUI: `http://0.0.0.0:8787/health` ## Rollback - This read-only runbook has no rollback side effects - If you change fallback providers during remediation, record and revert via `hermes fallback list` + previous wiki state ## Last tested 2026-10-26 — LiteLLM 401 unauthed (expected), OmniRoute/CT600 6,480 models, WebUI `/health` 200, opencode/nous/ollama fallbacks all 200 ## Related - [[model-providers]] - [[messaging-integrations]] - [[restart-hermes]]