104 lines
3.3 KiB
Markdown
104 lines
3.3 KiB
Markdown
---
|
|
title: Provider Health Check
|
|
type: runbook
|
|
status: active
|
|
created: 2026-07-22
|
|
updated: 2026-07-22
|
|
verified_on: 2026-07-22
|
|
last_tested: 2026-07-22
|
|
confidence: high
|
|
tags: [runbook, providers, health, routing]
|
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
|
---
|
|
|
|
# Provider Health Check
|
|
|
|
## Purpose
|
|
Verify Hermes can reach LiteLLM, OmniRoute, OpenRouter-style fallbacks, and the WebUI, and report which paths are healthy or degraded.
|
|
|
|
## Symptoms that match this runbook
|
|
- Models list is slow or blank in Hermes
|
|
- Fallback model retries happen immediately
|
|
- Chat returns 5xx or gateway errors
|
|
|
|
## Prerequisites
|
|
- Network access from this host to `litellm:4000`, `omniroute:20128`, `opencode.ai`, `api.telegram.org`, Telegram fallback IPs
|
|
- Hermes config slocated at `~/.hermes/config.yaml`
|
|
|
|
## Procedure
|
|
|
|
### 1. Check local Hermes services
|
|
```bash
|
|
hermes gateway status
|
|
hermes dashboard status
|
|
```
|
|
|
|
### 2. Check main provider reachability
|
|
```bash
|
|
python3 -c "import urllib.request; urllib.request.urlopen('http://litellm:4000/v1/models', timeout=4)"
|
|
```
|
|
Current install behavior: LiteLLM responds but requires auth, so HTTP 401 is expected from an unauthenticated check.
|
|
|
|
### 3. Check OmniRoute reachability
|
|
```bash
|
|
python3 -c "
|
|
import urllib.request, json, yaml
|
|
with open('/home/hermes/.hermes/config.yaml') as f:
|
|
cfg = yaml.safe_load(f)
|
|
key = [p['api_key'] for p in cfg.get('custom_providers',[]) if p.get('name')=='omniroute'][0]
|
|
req = urllib.request.Request('http://omniroute:20128/v1/models', headers={'Authorization': f'Bearer {key}'})
|
|
with urllib.request.urlopen(req, timeout=5) as r:
|
|
data = json.loads(r.read())
|
|
print('OmniRoute OK, models:', len(data.get('data', [])))
|
|
"
|
|
```
|
|
Current verified result: reachable and returns a non-empty model catalog.
|
|
|
|
### 4. Check fallback provider endpoints
|
|
```bash
|
|
curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://opencode.ai/zen/v1/models
|
|
curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://inference-api.nousresearch.com/v1/models
|
|
curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://ollama.com/v1/models
|
|
```
|
|
Any non-200 here means that fallback path is currently offline.
|
|
|
|
### 5. Check WebUI health
|
|
```bash
|
|
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8787/health
|
|
```
|
|
Expected: `200`
|
|
|
|
### 6. Check gateway platform reachability
|
|
Tail the gateway journal for errors:
|
|
```bash
|
|
journalctl --user -u hermes-gateway.service -n 100 --no-pager
|
|
```
|
|
Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or Email fetch.
|
|
|
|
## Verification
|
|
- Gateway status shows `active (running)`.
|
|
- LiteLLM 401 from unauthenticated check is expected.
|
|
- OmniRoute returns model count > 0.
|
|
- Fallback URLs return 200 or expected auth codes.
|
|
- WebUI `/health` returns `200`.
|
|
- Gateway journal shows no red alarm pattern.
|
|
|
|
## Current install references
|
|
- Hermes: v0.19.0, git install: `/home/hermes/.hermes/hermes-agent`
|
|
- LiteLLM base URL: `http://litellm:4000/v1`
|
|
- OmniRoute base URL: `http://omniroute:20128/v1`
|
|
- OmniRoute direct IP: `100.88.81.19:20128`
|
|
- WebUI: `http://0.0.0.0:8787/health`
|
|
|
|
## Rollback
|
|
- This read-only runbook has no rollback side effects
|
|
- If you change fallback providers during remediation, record and revert via `hermes fallback list` + previous wiki state
|
|
|
|
## Last tested
|
|
2026-07-22
|
|
|
|
## Related
|
|
- [[model-providers]]
|
|
- [[messaging-integrations]]
|
|
- [[restart-hermes]]
|