stocktake(2026-10-26): correct 8 stale pages against live state — 0 stale remaining
wiki-maintain.sh ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted, service restarted). The freshness report listed 8 stale pages; each was diffed against live state and corrected only where reality could be verified. A second script run rewrote the same-day report with 0 stale / 103 fresh / 4 no-date, so the committed report matches the tree it ships with. - infrastructure/hosts.md: CT600 is LIVE (not retired); CT601 does not exist on either hypervisor; CT333 verified running; both hypervisors verified; config 33->51; skills 66->302; cron 25->28 (25 enabled / 3 disabled); Xvfb :102->:99; providers split into Omniroute:20128 (CT600) and omni-101 (CT101, default). - agents/hermes-production.md: Tailscale IP 100.96.100.82 -> 100.118.5.51 (the old value is the PVE hypervisor); v0.21.5 -> v0.22.3; dead 8082 claim removed; fallback chain rewritten from live config (first entry is the dead nous credential); all 8 auxiliary slots now no-think/freeapi/auto; cron 30->28. - agents/hermes-chaos.md: v0.22.3 (same build as CT460); "3 commits behind" -> 0 behind origin/master; default model smart-route on custom:omni-101; cron "11 paused" -> 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460. - infrastructure/tailscale.md: node table rebuilt from status --json; octoagent and omarchymac-1 were missing entirely (52 -> 54 non-funnel nodes); m93p now online; every offline age refreshed; omni row corrected to "CT601 removed from both hypervisors"; Serve tables re-probed per host. - systems/memory-backend.md: re-verified against live config; coverage 1899/2350 (80.8%), working 96.8%, episodic 99.4% — above the 95% reindex threshold. - runbooks/provider-health.md: the step-3 OmniRoute snippet raised IndexError (provider renamed omniroute -> Omniroute:20128; key moved to key_env) — fixed; new step 3b documents the omni-101 /v1/models false-failure quirk. - runbooks/update-hermes-safely.md: added managed-runtime dependency re-assert step; clarified --branch vs stable channel (v0.22.5 available). - incidents/2026-10-03-ct450-hermes-update.md + incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md: Current-state sections recording current versions and embedding coverage. All 9 corrected pages verified 200 at http://127.0.0.1:9120/<path>. Vault mirror at /home/hermes/wiki synced for all touched files (22 copies, sha256-verified) and committed locally; the vault is local-only and is not pushed. No secrets written — IDs, URLs, key env var names, counts and timestamps only.
This commit is contained in:
@@ -3,8 +3,8 @@ title: Hermes Chaos
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-08-15
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
confidence: high
|
||||
tags: [hermes, chaos, ct450, experimental, agent]
|
||||
sources: [live-probe, config-inspection]
|
||||
@@ -28,15 +28,15 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod
|
||||
## Current State
|
||||
|
||||
- **Status:** Active (experimental)
|
||||
- **Hermes version:** `v0.21.5+6170.ga510d6c` (2026.9.24), upstream `a510d6c6` — **updated from v0.19.0 on 2026-10-03**, first update since July; 3 commits behind upstream. Python 3.14.7. See [[incidents/2026-10-03-ct450-hermes-update]].
|
||||
- **Hermes version:** `v0.22.3` (2026.9.24), local `5b147ab036d6` — **the same build as CT460** (updated 2026-10-21). **0 commits behind `origin/main`** (verified 2026-10-26; the old "3 commits behind" claim is stale), though `hermes update --check` reports stable **v0.22.5** available. Python 3.14.7. See [[incidents/2026-10-03-ct450-hermes-update]].
|
||||
- **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]].
|
||||
- **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01)
|
||||
- **Duplicate cron jobs:** 11 paused (no post-pause execution records since 2026-08-01)
|
||||
- **Production counterparts:** CT460 has 11 enabled jobs with successful latest runs
|
||||
- **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` 2026-10-26). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460)
|
||||
- **Production counterparts:** CT460 has **28 jobs / 25 enabled** — the two fleets no longer share a job count, and the CT450 duplicates are paused
|
||||
|
||||
## Providers & Routing
|
||||
|
||||
- **Default model:** `deepseek-flash-mix` via `custom:omniroute` (`http://100.88.81.19:20128/v1`, key via `$HERMES_CUSTOM_100_88_81_19_20128_API_KEY`)
|
||||
- **Default model:** **`smart-route`** via `custom` → `http://100.78.63.20:20128` (the `omni-101` node) — verified live in CT450's `config.yaml` 2026-10-26. ⚠ The old claim of `deepseek-flash-mix` on `custom:omniroute` (`100.88.81.19`) no longer matches the config
|
||||
- **Fallback chain** (fixed 2026-08-15 — OmniRoute fallback entries removed, they were non-functional):
|
||||
1. `kilo-stepfun-step-3_7-flash-free` via `litellm_4000`
|
||||
2. `nvidia/nemotron-3-ultra:free` via `nous`
|
||||
|
||||
+11
-11
@@ -3,8 +3,8 @@ title: Hermes Production
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-08-15
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
confidence: high
|
||||
tags: [hermes, production, ct460, agent]
|
||||
sources: [live-probe, config-inspection]
|
||||
@@ -19,15 +19,15 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
||||
## Location
|
||||
|
||||
- **Container:** LXC CT460 (`hermes`)
|
||||
- **Tailscale IP:** `100.96.100.82`
|
||||
- **Tailscale IP:** `100.118.5.51` (⚠ `100.96.100.82` is the **PVE hypervisor** `pve`, not CT460 — corrected 2026-10-26)
|
||||
- **Hostname:** `hermes`
|
||||
- **Port:** 8082 (HTTP), 9119 (gateway)
|
||||
- **Ports:** 9119 (gateway/API), 8085 (webhook) — ⚠ the old **8082** claim was dead: no 8082 listener on CT460 as of 2026-10-26
|
||||
- **Memory:** 14GB (upgraded from 5096MB on 2026-08-07)
|
||||
|
||||
## Current State
|
||||
|
||||
- **Status:** Active
|
||||
- **Hermes version:** `v0.21.5+6169.gab86a20` (2026.9.24), upstream `ab86a20a`; install tree rebuilt 2026-10-03 10:11 AWST — verified live 2026-10-04
|
||||
- **Hermes version:** `v0.22.3` (2026.9.24), local `5b147ab036d6` — updated 2026-10-21 22:47→22:57 AWST; **verified live 2026-10-26**. (`hermes update --check` reports stable **v0.22.5** available)
|
||||
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
|
||||
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
|
||||
- **Config:** `~/.hermes/config.yaml`
|
||||
@@ -43,10 +43,10 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
||||
|
||||
## Providers & Routing
|
||||
|
||||
- **Default provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — as attested 2026-09-30; the `custom:omni` provider key no longer exists
|
||||
- **Default provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, CT101/m93p) — verified live 2026-10-26. The `custom:omni` provider key does not exist in `custom_providers`, but **2 cron jobs still pin it** (see [[systems/scheduled-tasks]])
|
||||
- **Default model:** `smart-route` (previously `freellmapi-auto`)
|
||||
- **Fallback providers:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`, `custom / litellm:4000`
|
||||
- **Auxiliary curator model:** `openai-codex / gpt-5.6-terra`
|
||||
- **Fallback providers** (live chain, re-verified 2026-10-26): `nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6_1t`. ⚠ The **first** entry is the dead `nous` credential — a fallover tries it first
|
||||
- **Auxiliary models:** all eight `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) are **`no-think/freeapi/auto` on `custom:omni-101`** with a `freeapi/deepseek-v4-flash` → `kg/kilo-auto/free` fallback chain (moved 2026-10-25 19:13)
|
||||
- See [[systems/model-providers]] for full breakdown.
|
||||
|
||||
## Memory
|
||||
@@ -58,7 +58,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
||||
|
||||
## Scheduled Tasks
|
||||
|
||||
- **Cron jobs:** 30 (27 enabled / 3 disabled; **6 failing** as of 2026-10-04)
|
||||
- **Cron jobs:** **28 — 25 enabled / 3 disabled** (verified live `jobs.json` 2026-10-26). **5 non-clean `last_status`**: `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`), `08eb86f02d37` (`delivery_queued`, empty `last_error`), `c0a9fa314542` (`error`, streak 1)
|
||||
- Delivery targets: Telegram (primary), Discord, origin, local
|
||||
- Notable: Birthday checker, job radar, newsletter digests, mnemosyne health/watchdog/backup
|
||||
- See [[systems/scheduled-tasks]] for full table.
|
||||
@@ -66,7 +66,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
||||
## Messaging
|
||||
|
||||
- Telegram, Discord, Slack, Email, Mattermost, ntfy
|
||||
- **Mattermost is currently unreachable** — the gateway logs `mattermost connect timed out after 30s, next retry in 300s` (live, 2026-10-04) and the Mattermost host does not answer. Treat Mattermost delivery as down until probed.
|
||||
- **Mattermost is currently unreachable** — the gateway logs `mattermost connect timed out after 30s, next retry in 300s` (re-probed live 2026-10-26) and the Mattermost host does not answer: CT255 is `stopped` on `pve` and its Tailscale peer has been offline **36d**. Treat Mattermost delivery as down until probed.
|
||||
- See [[systems/messaging-integrations]] for details.
|
||||
|
||||
## Key Differences from Chaos
|
||||
@@ -74,7 +74,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
||||
| Aspect | Production (CT460) | Chaos (CT450) |
|
||||
|--------|-------------------|---------------|
|
||||
| Purpose | Stable daily operations | Experimental/testing |
|
||||
| Cron jobs | 30 (27 enabled / 3 disabled) | 11 paused duplicates |
|
||||
| Cron jobs | 28 (25 enabled / 3 disabled) | 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460 |
|
||||
| Config changes | Requires verification | Safe to experiment |
|
||||
| Memory | Production Mnemosyne | Separate state |
|
||||
|
||||
|
||||
@@ -3,8 +3,8 @@ title: CT450 Hermes Update to v0.21.5
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-04
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
confidence: high
|
||||
tags: [hermes, ct450, chaos, update, incident]
|
||||
sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
|
||||
@@ -48,6 +48,9 @@ Two gotchas that cost time:
|
||||
- `clang` present; no stale locks present.
|
||||
- CT460 production untouched — separate config, skills, cron and memory state.
|
||||
|
||||
## Current state (re-verified 2026-10-26)
|
||||
Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.
|
||||
|
||||
## Follow-ups
|
||||
- CT450 reports `Update available: 3 commits behind`. Harmless; re-run `hermes update` if the chaos box should track upstream exactly.
|
||||
- CT450's Mattermost connection is failing (`mattermost connect timed out after 30s`, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See [[agents/hermes-chaos]].
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Mnemosyne Vector Recall Ran FTS-Only
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-04
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
|
||||
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
|
||||
@@ -49,6 +49,22 @@ Embeddings generated (24h): 13 (working: 7, episodic: 6)
|
||||
## Benign noise
|
||||
`[E:onnxruntime:Default, env.cc:228 ThreadMain] pthread_setaffinity_np failed for thread: … mask: {0,}` — a container CPU-affinity warning emitted on every model load. Not an error; do not chase it.
|
||||
|
||||
## Current state (re-verified 2026-10-26)
|
||||
The 2026-10-03 follow-up is **satisfied**: `fastembed` 0.8.1 is importable in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`) as well as 0.8.0 in the legacy venv, and `FASTEMBED_CACHE_PATH` is pinned in `~/.hermes/.env`.
|
||||
|
||||
Current `check_embedding_coverage.py` output:
|
||||
```
|
||||
Memories with embeddings: 1899 / 2350 (80.8%)
|
||||
Working memory: 815 / 842 (96.8%)
|
||||
Episodic memory: 1084 / 1090 (99.4%)
|
||||
vec_episodes_rowids: 1086
|
||||
vec_working_rowids: 828
|
||||
Embeddings generated (24h): 15 (working: 9, episodic: 6)
|
||||
```
|
||||
Both tiers are **above the 95 % working-coverage reindex threshold**, so the 6-hourly
|
||||
monitor job `b63ed5817212` (`last_status: ok`) has nothing to fire on. The 2026-10-04
|
||||
figures below are superseded.
|
||||
|
||||
## Follow-ups
|
||||
- Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.
|
||||
- Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Anthony's Systems Wiki
|
||||
> Operational knowledge for Anthony's agents, infrastructure,
|
||||
> model routing and automation. Last updated: 2026-10-26
|
||||
> model routing and automation. Last updated: 2026-10-26 (stocktake — 8 stale pages corrected)
|
||||
>
|
||||
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
|
||||
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
|
||||
|
||||
+21
-14
@@ -3,8 +3,8 @@ title: Known Hosts and Paths
|
||||
type: infrastructure
|
||||
status: partial
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-07-22
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
confidence: medium
|
||||
tags: [infrastructure, hosts, containers, paths]
|
||||
sources: []
|
||||
@@ -16,12 +16,15 @@ sources: []
|
||||
|
||||
| Host | Role | Address | Verified |
|
||||
|---|---|---|---|
|
||||
| localhost (this host) | Primary Hermes runtime, Xvfb, file storage | `:102` display | Yes |
|
||||
| localhost (this host) | Primary Hermes runtime, Xvfb, file storage | `:99` display (`Xvfb :99 -screen 0 1600x1200x24`) | Yes (live probe 2026-10-26) |
|
||||
| litellm | LiteLLM proxy | `http://litellm:4000/v1` | Yes (from config) |
|
||||
| omni | OmniRoute proxy | `http://omni:20128/v1` | Yes (from config, 2026-08-13) |
|
||||
| `custom:Omniroute:20128` | OmniRoute proxy (CT600 node `omniroute`) | `http://omniroute:20128/v1` → `100.88.81.19:20128` | Yes (live probe 2026-10-26: 6,480 models) |
|
||||
| `custom:omni-101` | OmniRoute proxy (CT101 node `omniroute-1`) — **the live default provider** | `http://100.78.63.20:20128` | Yes (live probe 2026-10-26) |
|
||||
| CT201 | Steel Browser, Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.166` (LAN) | Yes (live probe) |
|
||||
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net` | Yes |
|
||||
| CT450 (hermesagent) | Hermes Agent **v0.21.5+6170.ga510d6c** (updated 2026-10-03 from v0.19.0), hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes |
|
||||
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net`, `100.118.5.51` (Tailscale) | Yes |
|
||||
| PVE `pve` | Proxmox VE hypervisor hosting CT460 and the other `pve` guests | `100.96.100.82` (Tailscale), `pve.kangaroo-eel.ts.net` | Yes (live `pct list` 2026-10-26) |
|
||||
| PVE `m93p` | Second, unclustered Proxmox VE 9.2.10 hypervisor | `100.104.51.98` (Tailscale) | Yes (live `pct list` 2026-10-26) |
|
||||
| CT450 (hermesagent) | Hermes Agent **v0.22.3** (2026.9.24), local `5b147ab036d6` — same build as CT460, hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes |
|
||||
|
||||
## Project Paths
|
||||
|
||||
@@ -30,15 +33,16 @@ sources: []
|
||||
| `/home/hermes` | Hermes home | `~/.hermes/` contains config, skills, cache |
|
||||
| `/home/hermes/workspace` | Workspace | default terminal cwd |
|
||||
| `/home/hermes/.local/bin/hermes` | Hermes binary | |
|
||||
| `/home/hermes/.hermes/config.yaml` | Config | version 33 |
|
||||
| `/home/hermes/.hermes/skills/` | Skills dir | 66 skills installed |
|
||||
| `/home/hermes/.hermes/cron/` | Cron jobs | 25 active jobs |
|
||||
| `/home/hermes/.hermes/config.yaml` | Config | `_config_version: 51` |
|
||||
| `/home/hermes/.hermes/skills/` | Skills dir | 302 skills across 69 categories (`.archive/` excluded) |
|
||||
| `/home/hermes/.hermes/cron/` | Cron jobs | 28 jobs — 25 enabled / 3 disabled |
|
||||
| `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ |
|
||||
| `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 |
|
||||
| `~/.camofox/` | Camofox profiles | managed persistence |
|
||||
| `~/.camoufox/` | Camoufox profiles | multiple profiles present |
|
||||
| `/home/hermes/.hermes/browser_screenshots/` | Browser captures | |
|
||||
| `/home/hermes/wiki/` | Wiki | This knowledge base |
|
||||
| `/home/hermes/workspace/agent-estate-wiki/` | Wiki — **authoring tree** (source of truth, pushed to Gitea) |
|
||||
| `/home/hermes/wiki/agent-estate-wiki/` | Wiki — **Obsidian/Quartz vault mirror** (local-only git; synced per audit, never pushed) |
|
||||
| `/home/hermes/obsidian-vault/` | Obsidian vault | personal notes (50 - Research etc.) |
|
||||
| `/home/hermes/hermes-webui/` | Hermes WebUI | web UI source |
|
||||
| `/home/hermes/hermes-ui/` | Hermes Desktop browser (Vite dev) | present on CT460 and CT450 port 5174 |
|
||||
@@ -55,12 +59,15 @@ sources: []
|
||||
- `/home/hermes/workspace/readlater/docker-compose.yml`
|
||||
|
||||
## Homelab References
|
||||
- **PVE host:** `pve.kangaroo-eel.ts.net` (100.96.100.82) per session notes — Unverified on this host
|
||||
- **CT600:** OmniRoute server (retired → CT601 `omni`, 2026-08-13) — Verified via `pct list`
|
||||
- **PVE host `pve`:** `pve.kangaroo-eel.ts.net` (100.96.100.82) — **verified** via live `pct list` 2026-10-26 (17 LXCs)
|
||||
- **PVE host `m93p`:** 100.104.51.98, Proxmox VE 9.2.10 — **verified** via live `pct list` 2026-10-26 (19 LXCs). Guest IDs collide between the two hosts
|
||||
- **CT600 (`omniroute`):** OmniRoute server, `100.88.81.19` — **running and live**; `omniroute` is the hostname that resolves to it
|
||||
- **CT601 (`omni`):** the old CT601 OmniRoute guest **no longer exists** on either hypervisor (`pct config 601` → no such config). The `omni` Tailscale peer (100.93.204.4) is its orphaned node, offline 69d
|
||||
- **CT101 (`omniroute`, m93p):** second OmniRoute node, `100.78.63.20` — running
|
||||
|
||||
## Unverified
|
||||
- CT600, CT333, and other remote Proxmox containers/VMs referenced in session notes but not confirmed from this host
|
||||
- PVE host identity not confirmed from this host without explicit confirmation
|
||||
- Individual LXCs are verified only when probed directly; the two `pct list` snapshots above are the current authority for what is running
|
||||
- Guests not present in those `pct list` outputs are **not** running on either hypervisor — do not assume existence from older session notes
|
||||
|
||||
## Related
|
||||
- [[docker-services]]
|
||||
|
||||
+27
-23
@@ -3,8 +3,8 @@ title: Tailscale
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-26
|
||||
updated: 2026-10-25
|
||||
verified_on: 2026-10-25
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
confidence: high
|
||||
tags: [tailscale, vpn, networking, infrastructure]
|
||||
---
|
||||
@@ -20,7 +20,7 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access
|
||||
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
|
||||
|
||||
## Tailscale Node Inventory (verified 2026-10-25)
|
||||
Full node list from `tailscale status` — **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
|
||||
Full node list from `tailscale status` — **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
|
||||
Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong).
|
||||
Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`.
|
||||
Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**.
|
||||
@@ -29,43 +29,45 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
||||
|
||||
| Node | IP | State |
|
||||
|---|---|---|
|
||||
| hermes | 100.118.5.51 | **self** |
|
||||
| hermes | 100.118.5.51 | **self** — CT460 (pve) |
|
||||
| agentzero | 100.106.54.94 | online — CT475 (pve) |
|
||||
| anthonys-imac | 100.112.123.124 | idle/offline 31d |
|
||||
| anthonys-imac | 100.112.123.124 | idle/offline 32d |
|
||||
| anthonys-ipad | 100.111.198.120 | online |
|
||||
| anthonys-iphone | 100.100.116.67 | online |
|
||||
| anthonys-mac-mini | 100.74.17.15 | online |
|
||||
| cachyos | 100.124.208.15 | idle/offline 51d |
|
||||
| cachyos | 100.124.208.15 | idle/offline 52d |
|
||||
| clawtest | 100.122.189.81 | online — CT333 (pve) |
|
||||
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
|
||||
| cloudreve | 100.111.187.7 | idle/offline 92d — CT209 stopped |
|
||||
| conduit | 100.81.79.87 | idle/offline 48d |
|
||||
| cloudreve | 100.111.187.7 | idle/offline 93d — CT209 stopped |
|
||||
| conduit | 100.81.79.87 | idle/offline 49d |
|
||||
| docker | 100.96.150.69 | online — CT200 (pve) |
|
||||
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
|
||||
| elementsynapse | 100.75.250.37 | idle/offline 95d — CT555 stopped |
|
||||
| elementsynapse | 100.75.250.37 | idle/offline 96d — CT555 stopped |
|
||||
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
|
||||
| freshrss | 100.83.60.93 | online — CT1260 (m93p) |
|
||||
| freshrss-new | 100.110.180.115 | idle/offline 68d |
|
||||
| freshrss-new | 100.110.180.115 | idle/offline 69d |
|
||||
| gitea | 100.82.100.128 | online — CT221 (m93p) |
|
||||
| guest-vps | 100.117.14.84 | online |
|
||||
| hermesagent | 100.98.205.112 | online — CT450 (pve) |
|
||||
| homeassistant | 100.68.139.5 | online |
|
||||
| litellm | 100.79.29.64 | online — CT500 (m93p) |
|
||||
| m93p | 100.104.51.98 | idle/offline 13m ago |
|
||||
| m93p | 100.104.51.98 | online |
|
||||
| mailflow | 100.115.60.92 | online — CT1262 (m93p) |
|
||||
| mattermost | 100.98.116.14 | idle/offline 35d — CT255 stopped |
|
||||
| n8n | 100.84.124.30 | online — CT215 (pve) |
|
||||
| mattermost | 100.98.116.14 | idle/offline 36d — CT255 stopped |
|
||||
| n8n | 100.84.124.30 | online |
|
||||
| nextcloud | 100.90.102.19 | online — CT270 (pve) |
|
||||
| ntfy | 100.67.153.116 | online — CT1241 (m93p) |
|
||||
| octoagent | 100.117.10.68 | online |
|
||||
| octop | 100.92.163.122 | online |
|
||||
| ollama | 100.108.44.119 | online — CT205 (pve) |
|
||||
| omarchy | 100.73.227.103 | online |
|
||||
| omarchybook | 100.107.9.100 | idle/offline 3d |
|
||||
| omarchymac | 100.84.180.69 | idle/offline 7d |
|
||||
| omni | 100.93.204.4 | idle/offline 68d — CT601 stopped |
|
||||
| omarchybook | 100.107.9.100 | idle/offline 4d |
|
||||
| omarchymac | 100.84.180.69 | idle/offline 8d |
|
||||
| omarchymac-1 | 100.126.173.115 | online |
|
||||
| omni | 100.93.204.4 | idle/offline 69d — CT601 removed from both hypervisors |
|
||||
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
|
||||
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
|
||||
| openmausbot | 100.91.214.91 | idle/offline 3d |
|
||||
| openmausbot | 100.91.214.91 | idle/offline 4d |
|
||||
| openmausbot-vm | 100.77.107.30 | online |
|
||||
| openweb | 100.69.230.13 | online — CT245 (m93p) |
|
||||
| paseo | 100.72.188.93 | online — CT1261 (m93p) |
|
||||
@@ -78,7 +80,7 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
||||
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
|
||||
| springroll | 100.100.153.125 | online |
|
||||
| springroll-browser | 100.109.221.91 | online |
|
||||
| ubuntu-server-xfce | 100.101.26.104 | idle/offline 64d |
|
||||
| ubuntu-server-xfce | 100.101.26.104 | idle/offline 66d |
|
||||
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
|
||||
| webserver | 100.82.40.23 | online — CT207 (pve) |
|
||||
|
||||
@@ -102,7 +104,7 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
||||
| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) |
|
||||
| `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid |
|
||||
|
||||
Verified against `tailscale serve status` on 2026-10-25 — the CT460 route table above matched live exactly (12 paths, no drift).
|
||||
Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 route table above matched live exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; no drift).
|
||||
|
||||
## CT460 Funnel and 9443 Routes
|
||||
| Port | Target | Notes |
|
||||
@@ -110,16 +112,18 @@ Verified against `tailscale serve status` on 2026-10-25 — the CT460 route tabl
|
||||
| `:8443/webhook` | `http://127.0.0.1:8085/webhook` | Public webhook Funnel (`# Funnel on:` confirmed) |
|
||||
| `:9443` (tailnet only) `/` | `http://127.0.0.1:5173` | Vite dev server on port 5173 exposed via a **tailnet-only** Serve on port 9443 — was missing from this page, added 2026-10-11 |
|
||||
|
||||
Re-verified 2026-10-26 (stocktake): the node table was rebuilt again from `tailscale status --json` — **`octoagent` (100.117.10.68) and `omarchymac-1` (100.126.173.115) were both missing entirely** and have been added; `m93p` (100.104.51.98) now reads **online** (it was "idle/offline 13m ago" — a stale Tailscale peer record, and the host was plainly reachable throughout); every offline age was refreshed (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**, `ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**, `openmausbot` 3d → **4d**). The `omni` row now reads **"CT601 removed from both hypervisors"** rather than "CT601 stopped" — live `pct list` on both `pve` and `m93p` shows **no CT601 at all**, so the old guest is gone, not merely halted. Container identity re-confirmed from **inside** each LXC via `pct exec <vmid> -- tailscale ip -4` on both hypervisors; every peer whose LXC has no Tailscale IP was checked against `pct list` (`CT105`/`CT1264` run but do not connect — `conduit` and `openmausbot` offline; `CT209`/`CT255`/`CT555` are `stopped`). Serve tables re-probed per host with `pct exec 201 -- tailscale serve status`.
|
||||
|
||||
## CT201 Serve Routes
|
||||
Verified live on 2026-10-25 via `pct exec 201 -- tailscale serve status` on `pve` (100.96.100.82). The previous version of this table was wrong in two ways and has been corrected.
|
||||
Verified live on 2026-10-26 via `pct exec 201 -- tailscale serve status` on `pve` (100.96.100.82). The previous version of this table was wrong in two ways and has been corrected.
|
||||
|
||||
| Host / Port | Path | Target | Notes |
|
||||
|---|---|---|---|
|
||||
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Homepage asset proxy front door — **currently 502** (upstream 8082 has no listener) |
|
||||
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Homepage asset proxy front door — **502** (re-probed 2026-10-26; upstream 8082 has no listener) |
|
||||
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` container — **200** |
|
||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Same asset proxy — **currently 502** |
|
||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Same asset proxy — **502** (re-probed 2026-10-26) |
|
||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` — **200** |
|
||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/steel` | `http://127.0.0.1:5173` | Steel Browser UI — **200** on :443. ⚠ **`/steel` is NOT served on :8443** — it 502s there |
|
||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/steel` | `http://127.0.0.1:5173` | Steel Browser UI — **200** on :443 (re-probed 2026-10-26). ⚠ **`/steel` is NOT served on :8443** — it 502s there |
|
||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/steel-api` | `http://127.0.0.1:3000` | Steel Browser API (path prefix not stripped — use direct IP for API calls) |
|
||||
|
||||
Two corrections to the pre-2026-10-25 table:
|
||||
|
||||
@@ -1310,3 +1310,75 @@ No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe st
|
||||
- **No secrets written.** Job IDs, incident IDs, base URLs, counts, paths and timestamps only.
|
||||
- **Not changed:** `current-state.md` (its dated sections are historical records), and the dead `custom:omni`
|
||||
provider entry (still named by 2 jobs — Anthony's call, not the wiki's).
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-26 — Stocktake: 8 stale pages corrected against live state (0 stale remaining)
|
||||
|
||||
`wiki-maintain.sh` ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted, `hermes-wiki-serve.service`
|
||||
restarted). The first freshness report listed **8 stale pages**; each was then **diffed against live state** and
|
||||
corrected where reality could be verified. A second script run then rewrote the same-day report with
|
||||
**0 stale / 103 fresh / 4 no-date**, so the committed report matches the tree it ships with.
|
||||
|
||||
- **`infrastructure/hosts.md`** — **CT600 is live, not retired.** Live `pct list` on both hypervisors shows
|
||||
**CT600 running** on `pve` (100.88.81.19) and **CT101 running** on `m93p` (100.78.63.20); **CT601 does not
|
||||
exist on either host** (`pct config 601` → no such config), so the old "retired → CT601" note was wrong on both
|
||||
counts. **CT333** (`clawtest`) moved from "unverified" to verified running. Both hypervisors now carry a
|
||||
verified row (`pve` 100.96.100.82 / 17 LXCs; `m93p` 100.104.51.98 / 19 LXCs). Config version **33 → 51**;
|
||||
skills **66 → 302**; cron **"25 active" → 28 (25 enabled / 3 disabled)**; Xvfb display **:102 → :99**
|
||||
(`Xvfb :99 -screen 0 1600x1200x24`). The `omni` provider entry was replaced by the two real ones —
|
||||
`custom:Omniroute:20128` → `http://omniroute:20128/v1` (6,480 models) and **`custom:omni-101`** →
|
||||
`http://100.78.63.20:20128` (**the live default provider**). The authoring tree vs the vault mirror are now
|
||||
distinguished by path.
|
||||
- **`agents/hermes-production.md`** — **Tailscale IP corrected: `100.96.100.82` → `100.118.5.51`.** The old value
|
||||
is the **PVE hypervisor**, not CT460. Version **v0.21.5 → v0.22.3** (`5b147ab036d6`); stable **v0.22.5** is
|
||||
available (`hermes update --check`). The **dead 8082** port claim was removed (no listener; live ports are
|
||||
9119 gateway + 8085 webhook). Default provider is **`custom:omni-101`**, and the **fallback chain was rewritten
|
||||
from live config** — its **first entry is the dead `nous` credential**, so a fallover tries a broken provider
|
||||
first. All **eight `auxiliary:` slots** are `no-think/freeapi/auto` on `custom:omni-101` (moved 2026-10-25
|
||||
19:13). Cron **30 (27/3) → 28 (25/3)** with the 5 non-clean `last_status` named. Mattermost still unreachable —
|
||||
CT255 `stopped`, peer offline **36 d**.
|
||||
- **`agents/hermes-chaos.md`** — version **v0.22.3**, and **both CT450 and CT460 now run the same build**
|
||||
(`5b147ab036d6`); the "**3 commits behind**" claim is superseded (**0 behind `origin/main`**). Default model
|
||||
is **`smart-route`** on `custom` → `http://100.78.63.20:20128`, not the old `deepseek-flash-mix` on
|
||||
`custom:omniroute`. Cron **"11 paused" → 25 jobs (13 enabled / 12 disabled)**, with **9 names shared with
|
||||
CT460** (8 disabled here, `Important-mail monitor` the reverse).
|
||||
- **`infrastructure/tailscale.md`** — node table rebuilt from `tailscale status --json` (keyed on `DNSName`,
|
||||
IPv4-filtered) and container identity re-confirmed from **inside** each LXC via
|
||||
`pct exec <vmid> -- tailscale ip -4` on both hypervisors. **`octoagent` (100.117.10.68) and
|
||||
`omarchymac-1` (100.126.173.115) were missing entirely** and are now listed — **52 → 54 non-funnel nodes**
|
||||
(53 live IPv4 peers + `hermes`). `m93p` now reads **online** (it was a stale peer record). **Every offline age
|
||||
was refreshed** (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**,
|
||||
`ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**,
|
||||
`openmausbot` 3d → **4d**). The `omni` row now reads "**CT601 removed from both hypervisors**" rather than
|
||||
"CT601 stopped". Serve tables re-probed per host: **CT460 matched live exactly** (12 paths on :443 plus the
|
||||
:8443 Funnel and :9443 route); **CT201 still 502s on `/`** (asset proxy → dead 8082) while `/mgr` and `/steel`
|
||||
return **200** — the 502 claim was re-confirmed, not merely carried over.
|
||||
- **`systems/memory-backend.md`** — re-verified against live `config.yaml`: added `reflect.disabled_for_cron`,
|
||||
`sleep_threshold: 50`, `nudge_interval: 10`, and live totals (**2,350 memories — 842 working / 1,090 episodic /
|
||||
418 `memoria_facts`**). Embedding coverage re-measured: **1,899 / 2,350 (80.8 %)** overall, **working 815 / 842
|
||||
(96.8 %)** and **episodic 1,084 / 1,090 (99.4 %)** — both tiers **above the 95 % reindex threshold**, so no
|
||||
reindex was triggered.
|
||||
- **`runbooks/provider-health.md`** — the step-3 OmniRoute snippet was **broken and raised `IndexError`**: the
|
||||
provider was renamed `omniroute` → **`Omniroute:20128`** and the key moved from an inline `api_key` to a
|
||||
**`key_env`** indirection. The snippet now resolves the key from `key_env`. A **new step 3b** documents the
|
||||
**`omni-101` quirk** — its `/v1/models` returns `Invalid API key` even with the correct key while
|
||||
`/v1/chat/completions` **works** (a `smart-route` call returned `minimaxai/minimax-m3.1-flash`), so a models
|
||||
check on that node is a **false failure**. Also corrected the typo "config slocated" → "located".
|
||||
Re-tested 2026-10-26: LiteLLM 401 unauthed (expected), CT600 OmniRoute 6,480 models, WebUI `/health` 200,
|
||||
opencode/nous/ollama fallbacks all 200.
|
||||
- **`runbooks/update-hermes-safely.md`** — added the **managed-runtime dependency re-assert step**
|
||||
(`hermes-runtime-deps-guard.sh`) after an update, since the updater can leave a runtime without
|
||||
`fastembed`/`dotenv`/`mnemosyne`. Clarified `--branch main` vs the **stable** channel (currently **v0.22.5**,
|
||||
local **v0.22.3**). Version and `last_tested` brought to 2026-10-26.
|
||||
- **`incidents/2026-10-03-ct450-hermes-update.md`** — added a **Current state** section: both instances now run
|
||||
**v0.22.3** (`5b147ab036d6`), CT450 is **0 behind `origin/main`**, and the CT450 Mattermost timeout is still an
|
||||
infrastructure condition (CT255 `stopped`), not an update regression.
|
||||
- **`incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md`** — added a **Current state** section: the
|
||||
follow-up is **satisfied** — `fastembed` 0.8.1 imports in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`,
|
||||
`d41b16c8b6674454a4a819bd4acced1f`) and 0.8.0 in the legacy venv, `FASTEMBED_CACHE_PATH` pinned. Current
|
||||
coverage figures recorded; both tiers above the 95 % threshold, so monitor job `b63ed5817212`
|
||||
(`last_status: ok`) has nothing to fire on.
|
||||
- **All 9 corrected pages verified at `http://127.0.0.1:9120/<path>` — every one returns 200**, and the served
|
||||
`infrastructure/tailscale` contains `octoagent`, `omarchymac-1` and `54 non-funnel`.
|
||||
- **No secrets written.** Job IDs, incident IDs, base URLs, key *env var names*, counts, paths and timestamps only.
|
||||
|
||||
@@ -1,20 +1,14 @@
|
||||
# Wiki freshness audit — 2026-10-26
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (8)
|
||||
## STALE (0)
|
||||
|
||||
- [ 22d] agents/hermes-chaos.md (updated 2026-10-04)
|
||||
- [ 22d] agents/hermes-production.md (updated 2026-10-04)
|
||||
- [ 22d] incidents/2026-10-03-ct450-hermes-update.md (updated 2026-10-04)
|
||||
- [ 22d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md (updated 2026-10-04)
|
||||
- [ 22d] infrastructure/hosts.md (updated 2026-10-04)
|
||||
- [ 22d] runbooks/provider-health.md (updated 2026-10-04)
|
||||
- [ 22d] runbooks/update-hermes-safely.md (updated 2026-10-04)
|
||||
- [ 22d] systems/memory-backend.md (updated 2026-10-04)
|
||||
|
||||
## FRESH (95)
|
||||
## FRESH (103)
|
||||
|
||||
- [ 6d] CONTRIBUTING.md
|
||||
- [ 0d] agents/hermes-chaos.md
|
||||
- [ 0d] agents/hermes-production.md
|
||||
- [ 6d] agents/index.md
|
||||
- [ 15d] agents/nanobot.md
|
||||
- [ 6d] agents/open-claw.md
|
||||
@@ -47,6 +41,8 @@ Threshold: >21 days since `updated`
|
||||
- [ 5d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 15d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 20d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 0d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 0d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 20d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 2d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 17d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
@@ -63,9 +59,10 @@ Threshold: >21 days since `updated`
|
||||
- [ 6d] infrastructure/credentials-map.md
|
||||
- [ 15d] infrastructure/docker-services.md
|
||||
- [ 6d] infrastructure/domains-and-tunnels.md
|
||||
- [ 0d] infrastructure/hosts.md
|
||||
- [ 12d] infrastructure/network-map.md
|
||||
- [ 12d] infrastructure/proxmox.md
|
||||
- [ 1d] infrastructure/tailscale.md
|
||||
- [ 0d] infrastructure/tailscale.md
|
||||
- [ 14d] queries/retained-research-answers.md
|
||||
- [ 1d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 5d] reports/uber-eats/2026-07.md
|
||||
@@ -77,12 +74,14 @@ Threshold: >21 days since `updated`
|
||||
- [ 6d] runbooks/gateway-resource-resilience.md
|
||||
- [ 13d] runbooks/job-radar.md
|
||||
- [ 6d] runbooks/multiplexer-setup.md
|
||||
- [ 0d] runbooks/provider-health.md
|
||||
- [ 6d] runbooks/recover-docker-service.md
|
||||
- [ 6d] runbooks/restart-browser.md
|
||||
- [ 6d] runbooks/restart-hermes.md
|
||||
- [ 6d] runbooks/restore-openrouter.md
|
||||
- [ 6d] runbooks/test-openrouter.md
|
||||
- [ 6d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 0d] runbooks/update-hermes-safely.md
|
||||
- [ 13d] systems/browser-backend.md
|
||||
- [ 6d] systems/cloudflare-tunnel.md
|
||||
- [ 0d] systems/conduit.md
|
||||
@@ -94,6 +93,7 @@ Threshold: >21 days since `updated`
|
||||
- [ 14d] systems/home-assistant.md
|
||||
- [ 1d] systems/homepage-dashboard.md
|
||||
- [ 15d] systems/ignis.md
|
||||
- [ 0d] systems/memory-backend.md
|
||||
- [ 0d] systems/messaging-integrations.md
|
||||
- [ 2d] systems/mnemosyne.md
|
||||
- [ 0d] systems/model-providers.md
|
||||
@@ -117,4 +117,4 @@ Threshold: >21 days since `updated`
|
||||
- index.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 8 stale, 95 fresh, 4 no-date of 107 pages
|
||||
TOTAL: 0 stale, 103 fresh, 4 no-date of 107 pages
|
||||
+30
-11
@@ -3,9 +3,9 @@ title: Provider Health Check
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-07-22
|
||||
last_tested: 2026-07-22
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
last_tested: 2026-10-26
|
||||
confidence: high
|
||||
tags: [runbook, providers, health, routing]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -23,7 +23,7 @@ Verify Hermes can reach LiteLLM, OmniRoute, OpenRouter-style fallbacks, and the
|
||||
|
||||
## Prerequisites
|
||||
- Network access from this host to `litellm:4000`, `omniroute:20128`, `opencode.ai`, `api.telegram.org`, Telegram fallback IPs
|
||||
- Hermes config slocated at `~/.hermes/config.yaml`
|
||||
- Hermes config located at `~/.hermes/config.yaml`
|
||||
|
||||
## Procedure
|
||||
|
||||
@@ -45,14 +45,32 @@ python3 -c "
|
||||
import urllib.request, json, yaml
|
||||
with open('/home/hermes/.hermes/config.yaml') as f:
|
||||
cfg = yaml.safe_load(f)
|
||||
key = [p['api_key'] for p in cfg.get('custom_providers',[]) if p.get('name')=='omniroute'][0]
|
||||
req = urllib.request.Request('http://omniroute:20128/v1/models', headers={'Authorization': f'Bearer {key}'})
|
||||
import os
|
||||
# NOTE: the provider was renamed 'omniroute' -> 'Omniroute:20128' and the key moved
|
||||
# from an inline 'api_key' to a 'key_env' indirection. The old snippet raised IndexError.
|
||||
p = [x for x in cfg.get('custom_providers',[]) if x.get('name')=='Omniroute:20128'][0]
|
||||
key = os.environ[p['key_env']]
|
||||
req = urllib.request.Request(p['base_url'] + '/models', headers={'Authorization': f'Bearer {key}'})
|
||||
with urllib.request.urlopen(req, timeout=5) as r:
|
||||
data = json.loads(r.read())
|
||||
print('OmniRoute OK, models:', len(data.get('data', [])))
|
||||
"
|
||||
```
|
||||
Current verified result: reachable and returns a non-empty model catalog.
|
||||
Current verified result (2026-10-26): reachable and returns a non-empty model catalog — **6,480 models** on `omniroute:20128` (CT600, 100.88.81.19).
|
||||
|
||||
### 3b. Check the default `omni-101` provider (CT101, 100.78.63.20)
|
||||
The **default provider** is `custom:omni-101` → `http://100.78.63.20:20128`, keyed via
|
||||
`HERMES_CUSTOM_100_78_63_20_20128_API_KEY`. Probe it with its **own** key — the CT600 key is rejected here:
|
||||
|
||||
```bash
|
||||
curl -s --max-time 8 -H "Authorization: Bearer $HERMES_CUSTOM_100_78_63_20_20128_API_KEY" \
|
||||
http://100.78.63.20:20128/v1/models | head -c 200
|
||||
```
|
||||
|
||||
⚠ **Known quirk (verified 2026-10-26):** `/v1/models` on this node returns
|
||||
`Invalid API key` even with the correct key, while `/v1/chat/completions` **works**
|
||||
(a `smart-route` call returned `minimaxai/minimax-m3.1-flash`). So a `/v1/models`
|
||||
check on `omni-101` will read as a false failure — **smoke-test the chat endpoint instead**.
|
||||
|
||||
### 4. Check fallback provider endpoints
|
||||
```bash
|
||||
@@ -84,10 +102,11 @@ Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or
|
||||
- Gateway journal shows no red alarm pattern.
|
||||
|
||||
## Current install references
|
||||
- Hermes: v0.21.5+6169.gab86a20 (2026.9.24), git install: `/home/hermes/.hermes/hermes-agent`
|
||||
- Hermes: **v0.22.3** (2026.9.24), local `5b147ab036d6`, git install: `/home/hermes/.hermes/hermes-agent` (`hermes update --check` reports stable **v0.22.5** available)
|
||||
- LiteLLM base URL: `http://litellm:4000/v1`
|
||||
- OmniRoute base URL: `http://omniroute:20128/v1`
|
||||
- OmniRoute direct IP: `100.88.81.19:20128`
|
||||
- OmniRoute base URL: `http://omniroute:20128/v1` (provider name `Omniroute:20128`, key env `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`)
|
||||
- OmniRoute direct IP (CT600): `100.88.81.19:20128`
|
||||
- Default provider `omni-101` (CT101): `http://100.78.63.20:20128` (key env `HERMES_CUSTOM_100_78_63_20_20128_API_KEY`) — ⚠ `/v1/models` returns a false `Invalid API key`; chat works
|
||||
- WebUI: `http://0.0.0.0:8787/health`
|
||||
|
||||
## Rollback
|
||||
@@ -95,7 +114,7 @@ Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or
|
||||
- If you change fallback providers during remediation, record and revert via `hermes fallback list` + previous wiki state
|
||||
|
||||
## Last tested
|
||||
2026-07-22
|
||||
2026-10-26 — LiteLLM 401 unauthed (expected), OmniRoute/CT600 6,480 models, WebUI `/health` 200, opencode/nous/ollama fallbacks all 200
|
||||
|
||||
## Related
|
||||
- [[model-providers]]
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Update Hermes safely
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-07-22
|
||||
last_tested: 2026-07-22
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
last_tested: 2026-10-26
|
||||
confidence: high
|
||||
tags: [runbook, hermes, update]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -24,6 +24,7 @@ Update Hermes to a newer release without losing configuration, skills, or sessio
|
||||
## Prerequisites
|
||||
- Hermes install method: git-based install at `/home/hermes/.hermes/hermes-agent`
|
||||
- Working Python virtualenv at `/home/hermes/.hermes/hermes-agent/venv`
|
||||
- **Managed runtimes live under `~/.hermes/installs/<install>/environments/<hash>/venv`** — these are what the gateway actually imports from; a missing dep there breaks cron jobs with no obvious trace (see [[incidents/2026-09-28-hermes-managed-runtime-deps]])
|
||||
- Available disk space in Hermes install and backup paths
|
||||
- No active Hermes chat/webui shell issues before starting
|
||||
|
||||
@@ -53,7 +54,17 @@ ls -la ~/.hermes/config.yaml.bak*
|
||||
```bash
|
||||
cd /home/hermes/.hermes/hermes-agent
|
||||
git status --short
|
||||
hermes update --yes --branch main
|
||||
hermes update --yes
|
||||
```
|
||||
Use `--branch main` only when deliberately tracking a branch other than the configured
|
||||
channel. The **stable** channel is currently the default and offers **v0.22.5**
|
||||
(`hermes update --check`, verified 2026-10-26; local install is v0.22.3 / `5b147ab036d6`).
|
||||
`--force-venv` rebuilds the virtualenv — avoid unless the venv itself is broken.
|
||||
|
||||
**After any update, re-assert the managed-runtime dependencies** (the updater can
|
||||
leave a runtime without `fastembed`/`dotenv`/`mnemosyne`):
|
||||
```bash
|
||||
~/.hermes/scripts/hermes-runtime-deps-guard.sh
|
||||
```
|
||||
|
||||
### 5. Verify the new version
|
||||
@@ -93,7 +104,7 @@ hermes dashboard restart
|
||||
- If update warns about config migration, revert the backup and re-run manually
|
||||
|
||||
## Notes
|
||||
- Hermes version on this install: v0.21.5+6169.gab86a20 (2026.9.24); install tree rebuilt 2026-10-03 10:11 AWST
|
||||
- Hermes version on this install: **v0.22.3** (2026.9.24), local `5b147ab036d6`; install tree rebuilt 2026-10-21 22:47→22:57 AWST. Stable **v0.22.5** available as of 2026-10-26
|
||||
- Install directory: `/home/hermes/.hermes/hermes-agent`
|
||||
- Update preserves venv by default unless `--force-venv` is used
|
||||
|
||||
@@ -118,7 +129,7 @@ pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"
|
||||
**After:** `pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"`, confirm CT460 production is untouched (separate config/skills/cron/memory). Full record: [[incidents/2026-10-03-ct450-hermes-update]].
|
||||
|
||||
## Last tested
|
||||
2026-07-22 on debian 13, git-based Hermes install
|
||||
2026-10-26 on debian 13, git-based Hermes install — `hermes update --check` (stable v0.22.5 available), `hermes backup --help` and `hermes update --help` flags confirmed
|
||||
|
||||
## Related
|
||||
- [[restart-hermes]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Memory Backend
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, agent]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -30,7 +30,11 @@ Persist and recall agent memory across sessions.
|
||||
- **Shared surface read:** true
|
||||
- **Sync roles:** user
|
||||
- **Skip contexts:** cron, flush, subagent, background, skill_loop
|
||||
- **Reflect max calls per session:** 3
|
||||
- **Reflect max calls per session:** 3 (`reflect.disabled_for_cron: true`)
|
||||
- **Sleep threshold:** 50 turns
|
||||
- **Nudge interval:** 10
|
||||
- **Ignore patterns:** none set
|
||||
- **Live totals (2026-10-26):** 2,350 memories — 842 working / 1,090 episodic / 418 `memoria_facts`
|
||||
|
||||
## Storage Location
|
||||
- Mnemosyne home: `~/.hermes/mnemosyne/`
|
||||
@@ -44,6 +48,7 @@ Persist and recall agent memory across sessions.
|
||||
- Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods.
|
||||
- The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
||||
- **Vector recall degraded silently until 2026-10-03** — `fastembed` was missing from every uv-managed runtime, so `mnemosyne_recall` fell back to FTS5 keyword scoring with no error raised. Repaired 2026-10-03; the 15-min dependency guard now asserts `fastembed` and the model cache path. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||||
- **Embedding coverage re-verified 2026-10-26** — `check_embedding_coverage.py` reports 1,899 / 2,350 (80.8 %) embedded; **working 815 / 842 (96.8 %)** and **episodic 1,084 / 1,090 (99.4 %)**. Both tiers are **above the 95 % reindex threshold**, so no reindex was triggered. `fastembed` 0.8.1 is present in **both** managed runtimes and 0.8.0 in the legacy venv; `FASTEMBED_CACHE_PATH` is pinned. The 2026-10-03 follow-up (assert `fastembed` in *all* managed runtimes) is now satisfied.
|
||||
|
||||
**Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]].
|
||||
|
||||
|
||||
Reference in New Issue
Block a user