Files
agent-estate-wiki/runbooks/provider-health.md
T

5.2 KiB

title, type, status, created, updated, verified_on, last_tested, confidence, tags, sources
title type status created updated verified_on last_tested confidence tags sources
Provider Health Check runbook active 2026-07-22 2026-11-18 2026-11-18 2026-10-26 high
runbook
providers
health
routing
raw/configs/hermes-config-sanitized.txt

Provider Health Check

Purpose

Verify Hermes can reach LiteLLM, OmniRoute, OpenRouter-style fallbacks, and the WebUI, and report which paths are healthy or degraded.

Symptoms that match this runbook

  • Models list is slow or blank in Hermes
  • Fallback model retries happen immediately
  • Chat returns 5xx or gateway errors

Prerequisites

  • Network access from this host to litellm:4000, omniroute:20128, opencode.ai, api.telegram.org, Telegram fallback IPs
  • Hermes config located at ~/.hermes/config.yaml

Procedure

1. Check local Hermes services

hermes gateway status
hermes dashboard status

2. Check main provider reachability

python3 -c "import urllib.request; urllib.request.urlopen('http://litellm:4000/v1/models', timeout=4)"

Current install behavior: LiteLLM responds but requires auth, so HTTP 401 is expected from an unauthenticated check.

3. Check OmniRoute reachability

python3 -c "
import urllib.request, json, yaml
with open('/home/hermes/.hermes/config.yaml') as f:
    cfg = yaml.safe_load(f)
import os
# NOTE: the provider was renamed 'omniroute' -> 'Omniroute:20128' and the key moved
# from an inline 'api_key' to a 'key_env' indirection. The old snippet raised IndexError.
p = [x for x in cfg.get('custom_providers',[]) if x.get('name')=='Omniroute:20128'][0]
key = os.environ[p['key_env']]
req = urllib.request.Request(p['base_url'] + '/models', headers={'Authorization': f'Bearer {key}'})
with urllib.request.urlopen(req, timeout=5) as r:
    data = json.loads(r.read())
    print('OmniRoute OK, models:', len(data.get('data', [])))
"

Current verified result (2026-10-26): reachable and returns a non-empty model catalog — 6,480 models on omniroute:20128 (CT600, 100.88.81.19).

Re-executed 2026-11-18: step-3 snippet OK — CT600 /models → 200, 6,501 models (3.6 s; one earlier attempt at the default 5 s timeout timed out, so give this probe headroom). CT600 /api/health → 200; omni-101 /api/health → 200; fallback endpoints opencode.ai / inference-api.nousresearch.com / ollama.com all → 200.

3b. Check the default omni-101 provider (CT101, 100.78.63.20)

The default provider is custom:omni-101 → http://100.78.63.20:20128, keyed via HERMES_CUSTOM_100_78_63_20_20128_API_KEY. Probe it with its own key — the CT600 key is rejected here:

curl -s --max-time 8 -H "Authorization: Bearer $HERMES_CUSTOM_100_78_63_20_20128_API_KEY" \
  http://100.78.63.20:20128/v1/models | head -c 200

⚠ Known quirk (verified 2026-10-26): /v1/models on this node returns Invalid API key even with the correct key, while /v1/chat/completions works (a smart-route call returned minimaxai/minimax-m3.1-flash). So a /v1/models check on omni-101 will read as a false failure — smoke-test the chat endpoint instead.

4. Check fallback provider endpoints

curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://opencode.ai/zen/v1/models
curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://inference-api.nousresearch.com/v1/models
curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://ollama.com/v1/models

Any non-200 here means that fallback path is currently offline.

5. Check WebUI health

curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8787/health

Expected: 200

6. Check gateway platform reachability

Tail the gateway journal for errors:

journalctl --user -u hermes-gateway.service -n 100 --no-pager

Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or Email fetch.

Verification

  • Gateway status shows active (running).
  • LiteLLM 401 from unauthenticated check is expected.
  • OmniRoute returns model count > 0.
  • Fallback URLs return 200 or expected auth codes.
  • WebUI /health returns 200.
  • Gateway journal shows no red alarm pattern.

Current install references

  • Hermes: v0.22.3 (2026.9.24), local 5b147ab036d6, git install: /home/hermes/.hermes/hermes-agent (hermes update --check reports stable v0.22.5 available)
  • LiteLLM base URL: http://litellm:4000/v1
  • OmniRoute base URL: http://omniroute:20128/v1 (provider name Omniroute:20128, key env HERMES_CUSTOM_OMNIROUTE_20128_API_KEY)
  • OmniRoute direct IP (CT600): 100.88.81.19:20128
  • Default provider omni-101 (CT101): http://100.78.63.20:20128 (key env HERMES_CUSTOM_100_78_63_20_20128_API_KEY) — ⚠ /v1/models returns a false Invalid API key; chat works
  • WebUI: http://0.0.0.0:8787/health

Rollback

  • This read-only runbook has no rollback side effects
  • If you change fallback providers during remediation, record and revert via hermes fallback list + previous wiki state

Last tested

2026-10-26 — LiteLLM 401 unauthed (expected), OmniRoute/CT600 6,480 models, WebUI /health 200, opencode/nous/ollama fallbacks all 200