Files
agent-estate-wiki/runbooks/provider-health.md
T
Tony0410 4126093644 stocktake(2026-10-26): correct 8 stale pages against live state — 0 stale remaining
wiki-maintain.sh ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted,
service restarted). The freshness report listed 8 stale pages; each was diffed
against live state and corrected only where reality could be verified. A second
script run rewrote the same-day report with 0 stale / 103 fresh / 4 no-date,
so the committed report matches the tree it ships with.

- infrastructure/hosts.md: CT600 is LIVE (not retired); CT601 does not exist on
  either hypervisor; CT333 verified running; both hypervisors verified; config
  33->51; skills 66->302; cron 25->28 (25 enabled / 3 disabled); Xvfb :102->:99;
  providers split into Omniroute:20128 (CT600) and omni-101 (CT101, default).
- agents/hermes-production.md: Tailscale IP 100.96.100.82 -> 100.118.5.51 (the
  old value is the PVE hypervisor); v0.21.5 -> v0.22.3; dead 8082 claim removed;
  fallback chain rewritten from live config (first entry is the dead nous
  credential); all 8 auxiliary slots now no-think/freeapi/auto; cron 30->28.
- agents/hermes-chaos.md: v0.22.3 (same build as CT460); "3 commits behind" ->
  0 behind origin/master; default model smart-route on custom:omni-101; cron
  "11 paused" -> 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460.
- infrastructure/tailscale.md: node table rebuilt from status --json; octoagent
  and omarchymac-1 were missing entirely (52 -> 54 non-funnel nodes); m93p now
  online; every offline age refreshed; omni row corrected to "CT601 removed
  from both hypervisors"; Serve tables re-probed per host.
- systems/memory-backend.md: re-verified against live config; coverage 1899/2350
  (80.8%), working 96.8%, episodic 99.4% — above the 95% reindex threshold.
- runbooks/provider-health.md: the step-3 OmniRoute snippet raised IndexError
  (provider renamed omniroute -> Omniroute:20128; key moved to key_env) — fixed;
  new step 3b documents the omni-101 /v1/models false-failure quirk.
- runbooks/update-hermes-safely.md: added managed-runtime dependency re-assert
  step; clarified --branch vs stable channel (v0.22.5 available).
- incidents/2026-10-03-ct450-hermes-update.md + incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md:
  Current-state sections recording current versions and embedding coverage.

All 9 corrected pages verified 200 at http://127.0.0.1:9120/<path>.
Vault mirror at /home/hermes/wiki synced for all touched files (22 copies,
sha256-verified) and committed locally; the vault is local-only and is not pushed.
No secrets written — IDs, URLs, key env var names, counts and timestamps only.
2026-10-26 04:27:49 +08:00

4.8 KiB

title, type, status, created, updated, verified_on, last_tested, confidence, tags, sources
title type status created updated verified_on last_tested confidence tags sources
Provider Health Check runbook active 2026-07-22 2026-10-26 2026-10-26 2026-10-26 high
runbook
providers
health
routing
raw/configs/hermes-config-sanitized.txt

Provider Health Check

Purpose

Verify Hermes can reach LiteLLM, OmniRoute, OpenRouter-style fallbacks, and the WebUI, and report which paths are healthy or degraded.

Symptoms that match this runbook

  • Models list is slow or blank in Hermes
  • Fallback model retries happen immediately
  • Chat returns 5xx or gateway errors

Prerequisites

  • Network access from this host to litellm:4000, omniroute:20128, opencode.ai, api.telegram.org, Telegram fallback IPs
  • Hermes config located at ~/.hermes/config.yaml

Procedure

1. Check local Hermes services

hermes gateway status
hermes dashboard status

2. Check main provider reachability

python3 -c "import urllib.request; urllib.request.urlopen('http://litellm:4000/v1/models', timeout=4)"

Current install behavior: LiteLLM responds but requires auth, so HTTP 401 is expected from an unauthenticated check.

3. Check OmniRoute reachability

python3 -c "
import urllib.request, json, yaml
with open('/home/hermes/.hermes/config.yaml') as f:
    cfg = yaml.safe_load(f)
import os
# NOTE: the provider was renamed 'omniroute' -> 'Omniroute:20128' and the key moved
# from an inline 'api_key' to a 'key_env' indirection. The old snippet raised IndexError.
p = [x for x in cfg.get('custom_providers',[]) if x.get('name')=='Omniroute:20128'][0]
key = os.environ[p['key_env']]
req = urllib.request.Request(p['base_url'] + '/models', headers={'Authorization': f'Bearer {key}'})
with urllib.request.urlopen(req, timeout=5) as r:
    data = json.loads(r.read())
    print('OmniRoute OK, models:', len(data.get('data', [])))
"

Current verified result (2026-10-26): reachable and returns a non-empty model catalog — 6,480 models on omniroute:20128 (CT600, 100.88.81.19).

3b. Check the default omni-101 provider (CT101, 100.78.63.20)

The default provider is custom:omni-101 → http://100.78.63.20:20128, keyed via HERMES_CUSTOM_100_78_63_20_20128_API_KEY. Probe it with its own key — the CT600 key is rejected here:

curl -s --max-time 8 -H "Authorization: Bearer $HERMES_CUSTOM_100_78_63_20_20128_API_KEY" \
  http://100.78.63.20:20128/v1/models | head -c 200

⚠ Known quirk (verified 2026-10-26): /v1/models on this node returns Invalid API key even with the correct key, while /v1/chat/completions works (a smart-route call returned minimaxai/minimax-m3.1-flash). So a /v1/models check on omni-101 will read as a false failure — smoke-test the chat endpoint instead.

4. Check fallback provider endpoints

curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://opencode.ai/zen/v1/models
curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://inference-api.nousresearch.com/v1/models
curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://ollama.com/v1/models

Any non-200 here means that fallback path is currently offline.

5. Check WebUI health

curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8787/health

Expected: 200

6. Check gateway platform reachability

Tail the gateway journal for errors:

journalctl --user -u hermes-gateway.service -n 100 --no-pager

Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or Email fetch.

Verification

  • Gateway status shows active (running).
  • LiteLLM 401 from unauthenticated check is expected.
  • OmniRoute returns model count > 0.
  • Fallback URLs return 200 or expected auth codes.
  • WebUI /health returns 200.
  • Gateway journal shows no red alarm pattern.

Current install references

  • Hermes: v0.22.3 (2026.9.24), local 5b147ab036d6, git install: /home/hermes/.hermes/hermes-agent (hermes update --check reports stable v0.22.5 available)
  • LiteLLM base URL: http://litellm:4000/v1
  • OmniRoute base URL: http://omniroute:20128/v1 (provider name Omniroute:20128, key env HERMES_CUSTOM_OMNIROUTE_20128_API_KEY)
  • OmniRoute direct IP (CT600): 100.88.81.19:20128
  • Default provider omni-101 (CT101): http://100.78.63.20:20128 (key env HERMES_CUSTOM_100_78_63_20_20128_API_KEY) — ⚠ /v1/models returns a false Invalid API key; chat works
  • WebUI: http://0.0.0.0:8787/health

Rollback

  • This read-only runbook has no rollback side effects
  • If you change fallback providers during remediation, record and revert via hermes fallback list + previous wiki state

Last tested

2026-10-26 — LiteLLM 401 unauthed (expected), OmniRoute/CT600 6,480 models, WebUI /health 200, opencode/nous/ollama fallbacks all 200