stocktake(2026-10-26): correct 8 stale pages against live state — 0 stale remaining
wiki-maintain.sh ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted, service restarted). The freshness report listed 8 stale pages; each was diffed against live state and corrected only where reality could be verified. A second script run rewrote the same-day report with 0 stale / 103 fresh / 4 no-date, so the committed report matches the tree it ships with. - infrastructure/hosts.md: CT600 is LIVE (not retired); CT601 does not exist on either hypervisor; CT333 verified running; both hypervisors verified; config 33->51; skills 66->302; cron 25->28 (25 enabled / 3 disabled); Xvfb :102->:99; providers split into Omniroute:20128 (CT600) and omni-101 (CT101, default). - agents/hermes-production.md: Tailscale IP 100.96.100.82 -> 100.118.5.51 (the old value is the PVE hypervisor); v0.21.5 -> v0.22.3; dead 8082 claim removed; fallback chain rewritten from live config (first entry is the dead nous credential); all 8 auxiliary slots now no-think/freeapi/auto; cron 30->28. - agents/hermes-chaos.md: v0.22.3 (same build as CT460); "3 commits behind" -> 0 behind origin/master; default model smart-route on custom:omni-101; cron "11 paused" -> 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460. - infrastructure/tailscale.md: node table rebuilt from status --json; octoagent and omarchymac-1 were missing entirely (52 -> 54 non-funnel nodes); m93p now online; every offline age refreshed; omni row corrected to "CT601 removed from both hypervisors"; Serve tables re-probed per host. - systems/memory-backend.md: re-verified against live config; coverage 1899/2350 (80.8%), working 96.8%, episodic 99.4% — above the 95% reindex threshold. - runbooks/provider-health.md: the step-3 OmniRoute snippet raised IndexError (provider renamed omniroute -> Omniroute:20128; key moved to key_env) — fixed; new step 3b documents the omni-101 /v1/models false-failure quirk. - runbooks/update-hermes-safely.md: added managed-runtime dependency re-assert step; clarified --branch vs stable channel (v0.22.5 available). - incidents/2026-10-03-ct450-hermes-update.md + incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md: Current-state sections recording current versions and embedding coverage. All 9 corrected pages verified 200 at http://127.0.0.1:9120/<path>. Vault mirror at /home/hermes/wiki synced for all touched files (22 copies, sha256-verified) and committed locally; the vault is local-only and is not pushed. No secrets written — IDs, URLs, key env var names, counts and timestamps only.
This commit is contained in:
+30
-11
@@ -3,9 +3,9 @@ title: Provider Health Check
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-07-22
|
||||
last_tested: 2026-07-22
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
last_tested: 2026-10-26
|
||||
confidence: high
|
||||
tags: [runbook, providers, health, routing]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -23,7 +23,7 @@ Verify Hermes can reach LiteLLM, OmniRoute, OpenRouter-style fallbacks, and the
|
||||
|
||||
## Prerequisites
|
||||
- Network access from this host to `litellm:4000`, `omniroute:20128`, `opencode.ai`, `api.telegram.org`, Telegram fallback IPs
|
||||
- Hermes config slocated at `~/.hermes/config.yaml`
|
||||
- Hermes config located at `~/.hermes/config.yaml`
|
||||
|
||||
## Procedure
|
||||
|
||||
@@ -45,14 +45,32 @@ python3 -c "
|
||||
import urllib.request, json, yaml
|
||||
with open('/home/hermes/.hermes/config.yaml') as f:
|
||||
cfg = yaml.safe_load(f)
|
||||
key = [p['api_key'] for p in cfg.get('custom_providers',[]) if p.get('name')=='omniroute'][0]
|
||||
req = urllib.request.Request('http://omniroute:20128/v1/models', headers={'Authorization': f'Bearer {key}'})
|
||||
import os
|
||||
# NOTE: the provider was renamed 'omniroute' -> 'Omniroute:20128' and the key moved
|
||||
# from an inline 'api_key' to a 'key_env' indirection. The old snippet raised IndexError.
|
||||
p = [x for x in cfg.get('custom_providers',[]) if x.get('name')=='Omniroute:20128'][0]
|
||||
key = os.environ[p['key_env']]
|
||||
req = urllib.request.Request(p['base_url'] + '/models', headers={'Authorization': f'Bearer {key}'})
|
||||
with urllib.request.urlopen(req, timeout=5) as r:
|
||||
data = json.loads(r.read())
|
||||
print('OmniRoute OK, models:', len(data.get('data', [])))
|
||||
"
|
||||
```
|
||||
Current verified result: reachable and returns a non-empty model catalog.
|
||||
Current verified result (2026-10-26): reachable and returns a non-empty model catalog — **6,480 models** on `omniroute:20128` (CT600, 100.88.81.19).
|
||||
|
||||
### 3b. Check the default `omni-101` provider (CT101, 100.78.63.20)
|
||||
The **default provider** is `custom:omni-101` → `http://100.78.63.20:20128`, keyed via
|
||||
`HERMES_CUSTOM_100_78_63_20_20128_API_KEY`. Probe it with its **own** key — the CT600 key is rejected here:
|
||||
|
||||
```bash
|
||||
curl -s --max-time 8 -H "Authorization: Bearer $HERMES_CUSTOM_100_78_63_20_20128_API_KEY" \
|
||||
http://100.78.63.20:20128/v1/models | head -c 200
|
||||
```
|
||||
|
||||
⚠ **Known quirk (verified 2026-10-26):** `/v1/models` on this node returns
|
||||
`Invalid API key` even with the correct key, while `/v1/chat/completions` **works**
|
||||
(a `smart-route` call returned `minimaxai/minimax-m3.1-flash`). So a `/v1/models`
|
||||
check on `omni-101` will read as a false failure — **smoke-test the chat endpoint instead**.
|
||||
|
||||
### 4. Check fallback provider endpoints
|
||||
```bash
|
||||
@@ -84,10 +102,11 @@ Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or
|
||||
- Gateway journal shows no red alarm pattern.
|
||||
|
||||
## Current install references
|
||||
- Hermes: v0.21.5+6169.gab86a20 (2026.9.24), git install: `/home/hermes/.hermes/hermes-agent`
|
||||
- Hermes: **v0.22.3** (2026.9.24), local `5b147ab036d6`, git install: `/home/hermes/.hermes/hermes-agent` (`hermes update --check` reports stable **v0.22.5** available)
|
||||
- LiteLLM base URL: `http://litellm:4000/v1`
|
||||
- OmniRoute base URL: `http://omniroute:20128/v1`
|
||||
- OmniRoute direct IP: `100.88.81.19:20128`
|
||||
- OmniRoute base URL: `http://omniroute:20128/v1` (provider name `Omniroute:20128`, key env `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`)
|
||||
- OmniRoute direct IP (CT600): `100.88.81.19:20128`
|
||||
- Default provider `omni-101` (CT101): `http://100.78.63.20:20128` (key env `HERMES_CUSTOM_100_78_63_20_20128_API_KEY`) — ⚠ `/v1/models` returns a false `Invalid API key`; chat works
|
||||
- WebUI: `http://0.0.0.0:8787/health`
|
||||
|
||||
## Rollback
|
||||
@@ -95,7 +114,7 @@ Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or
|
||||
- If you change fallback providers during remediation, record and revert via `hermes fallback list` + previous wiki state
|
||||
|
||||
## Last tested
|
||||
2026-07-22
|
||||
2026-10-26 — LiteLLM 401 unauthed (expected), OmniRoute/CT600 6,480 models, WebUI `/health` 200, opencode/nous/ollama fallbacks all 200
|
||||
|
||||
## Related
|
||||
- [[model-providers]]
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Update Hermes safely
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-07-22
|
||||
last_tested: 2026-07-22
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
last_tested: 2026-10-26
|
||||
confidence: high
|
||||
tags: [runbook, hermes, update]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -24,6 +24,7 @@ Update Hermes to a newer release without losing configuration, skills, or sessio
|
||||
## Prerequisites
|
||||
- Hermes install method: git-based install at `/home/hermes/.hermes/hermes-agent`
|
||||
- Working Python virtualenv at `/home/hermes/.hermes/hermes-agent/venv`
|
||||
- **Managed runtimes live under `~/.hermes/installs/<install>/environments/<hash>/venv`** — these are what the gateway actually imports from; a missing dep there breaks cron jobs with no obvious trace (see [[incidents/2026-09-28-hermes-managed-runtime-deps]])
|
||||
- Available disk space in Hermes install and backup paths
|
||||
- No active Hermes chat/webui shell issues before starting
|
||||
|
||||
@@ -53,7 +54,17 @@ ls -la ~/.hermes/config.yaml.bak*
|
||||
```bash
|
||||
cd /home/hermes/.hermes/hermes-agent
|
||||
git status --short
|
||||
hermes update --yes --branch main
|
||||
hermes update --yes
|
||||
```
|
||||
Use `--branch main` only when deliberately tracking a branch other than the configured
|
||||
channel. The **stable** channel is currently the default and offers **v0.22.5**
|
||||
(`hermes update --check`, verified 2026-10-26; local install is v0.22.3 / `5b147ab036d6`).
|
||||
`--force-venv` rebuilds the virtualenv — avoid unless the venv itself is broken.
|
||||
|
||||
**After any update, re-assert the managed-runtime dependencies** (the updater can
|
||||
leave a runtime without `fastembed`/`dotenv`/`mnemosyne`):
|
||||
```bash
|
||||
~/.hermes/scripts/hermes-runtime-deps-guard.sh
|
||||
```
|
||||
|
||||
### 5. Verify the new version
|
||||
@@ -93,7 +104,7 @@ hermes dashboard restart
|
||||
- If update warns about config migration, revert the backup and re-run manually
|
||||
|
||||
## Notes
|
||||
- Hermes version on this install: v0.21.5+6169.gab86a20 (2026.9.24); install tree rebuilt 2026-10-03 10:11 AWST
|
||||
- Hermes version on this install: **v0.22.3** (2026.9.24), local `5b147ab036d6`; install tree rebuilt 2026-10-21 22:47→22:57 AWST. Stable **v0.22.5** available as of 2026-10-26
|
||||
- Install directory: `/home/hermes/.hermes/hermes-agent`
|
||||
- Update preserves venv by default unless `--force-venv` is used
|
||||
|
||||
@@ -118,7 +129,7 @@ pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"
|
||||
**After:** `pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"`, confirm CT460 production is untouched (separate config/skills/cron/memory). Full record: [[incidents/2026-10-03-ct450-hermes-update]].
|
||||
|
||||
## Last tested
|
||||
2026-07-22 on debian 13, git-based Hermes install
|
||||
2026-10-26 on debian 13, git-based Hermes install — `hermes update --check` (stable v0.22.5 available), `hermes backup --help` and `hermes update --help` flags confirmed
|
||||
|
||||
## Related
|
||||
- [[restart-hermes]]
|
||||
|
||||
Reference in New Issue
Block a user