stocktake(2026-10-26): correct 8 stale pages against live state — 0 stale remaining
wiki-maintain.sh ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted, service restarted). The freshness report listed 8 stale pages; each was diffed against live state and corrected only where reality could be verified. A second script run rewrote the same-day report with 0 stale / 103 fresh / 4 no-date, so the committed report matches the tree it ships with. - infrastructure/hosts.md: CT600 is LIVE (not retired); CT601 does not exist on either hypervisor; CT333 verified running; both hypervisors verified; config 33->51; skills 66->302; cron 25->28 (25 enabled / 3 disabled); Xvfb :102->:99; providers split into Omniroute:20128 (CT600) and omni-101 (CT101, default). - agents/hermes-production.md: Tailscale IP 100.96.100.82 -> 100.118.5.51 (the old value is the PVE hypervisor); v0.21.5 -> v0.22.3; dead 8082 claim removed; fallback chain rewritten from live config (first entry is the dead nous credential); all 8 auxiliary slots now no-think/freeapi/auto; cron 30->28. - agents/hermes-chaos.md: v0.22.3 (same build as CT460); "3 commits behind" -> 0 behind origin/master; default model smart-route on custom:omni-101; cron "11 paused" -> 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460. - infrastructure/tailscale.md: node table rebuilt from status --json; octoagent and omarchymac-1 were missing entirely (52 -> 54 non-funnel nodes); m93p now online; every offline age refreshed; omni row corrected to "CT601 removed from both hypervisors"; Serve tables re-probed per host. - systems/memory-backend.md: re-verified against live config; coverage 1899/2350 (80.8%), working 96.8%, episodic 99.4% — above the 95% reindex threshold. - runbooks/provider-health.md: the step-3 OmniRoute snippet raised IndexError (provider renamed omniroute -> Omniroute:20128; key moved to key_env) — fixed; new step 3b documents the omni-101 /v1/models false-failure quirk. - runbooks/update-hermes-safely.md: added managed-runtime dependency re-assert step; clarified --branch vs stable channel (v0.22.5 available). - incidents/2026-10-03-ct450-hermes-update.md + incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md: Current-state sections recording current versions and embedding coverage. All 9 corrected pages verified 200 at http://127.0.0.1:9120/<path>. Vault mirror at /home/hermes/wiki synced for all touched files (22 copies, sha256-verified) and committed locally; the vault is local-only and is not pushed. No secrets written — IDs, URLs, key env var names, counts and timestamps only.
This commit is contained in:
@@ -1310,3 +1310,75 @@ No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe st
|
||||
- **No secrets written.** Job IDs, incident IDs, base URLs, counts, paths and timestamps only.
|
||||
- **Not changed:** `current-state.md` (its dated sections are historical records), and the dead `custom:omni`
|
||||
provider entry (still named by 2 jobs — Anthony's call, not the wiki's).
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-26 — Stocktake: 8 stale pages corrected against live state (0 stale remaining)
|
||||
|
||||
`wiki-maintain.sh` ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted, `hermes-wiki-serve.service`
|
||||
restarted). The first freshness report listed **8 stale pages**; each was then **diffed against live state** and
|
||||
corrected where reality could be verified. A second script run then rewrote the same-day report with
|
||||
**0 stale / 103 fresh / 4 no-date**, so the committed report matches the tree it ships with.
|
||||
|
||||
- **`infrastructure/hosts.md`** — **CT600 is live, not retired.** Live `pct list` on both hypervisors shows
|
||||
**CT600 running** on `pve` (100.88.81.19) and **CT101 running** on `m93p` (100.78.63.20); **CT601 does not
|
||||
exist on either host** (`pct config 601` → no such config), so the old "retired → CT601" note was wrong on both
|
||||
counts. **CT333** (`clawtest`) moved from "unverified" to verified running. Both hypervisors now carry a
|
||||
verified row (`pve` 100.96.100.82 / 17 LXCs; `m93p` 100.104.51.98 / 19 LXCs). Config version **33 → 51**;
|
||||
skills **66 → 302**; cron **"25 active" → 28 (25 enabled / 3 disabled)**; Xvfb display **:102 → :99**
|
||||
(`Xvfb :99 -screen 0 1600x1200x24`). The `omni` provider entry was replaced by the two real ones —
|
||||
`custom:Omniroute:20128` → `http://omniroute:20128/v1` (6,480 models) and **`custom:omni-101`** →
|
||||
`http://100.78.63.20:20128` (**the live default provider**). The authoring tree vs the vault mirror are now
|
||||
distinguished by path.
|
||||
- **`agents/hermes-production.md`** — **Tailscale IP corrected: `100.96.100.82` → `100.118.5.51`.** The old value
|
||||
is the **PVE hypervisor**, not CT460. Version **v0.21.5 → v0.22.3** (`5b147ab036d6`); stable **v0.22.5** is
|
||||
available (`hermes update --check`). The **dead 8082** port claim was removed (no listener; live ports are
|
||||
9119 gateway + 8085 webhook). Default provider is **`custom:omni-101`**, and the **fallback chain was rewritten
|
||||
from live config** — its **first entry is the dead `nous` credential**, so a fallover tries a broken provider
|
||||
first. All **eight `auxiliary:` slots** are `no-think/freeapi/auto` on `custom:omni-101` (moved 2026-10-25
|
||||
19:13). Cron **30 (27/3) → 28 (25/3)** with the 5 non-clean `last_status` named. Mattermost still unreachable —
|
||||
CT255 `stopped`, peer offline **36 d**.
|
||||
- **`agents/hermes-chaos.md`** — version **v0.22.3**, and **both CT450 and CT460 now run the same build**
|
||||
(`5b147ab036d6`); the "**3 commits behind**" claim is superseded (**0 behind `origin/main`**). Default model
|
||||
is **`smart-route`** on `custom` → `http://100.78.63.20:20128`, not the old `deepseek-flash-mix` on
|
||||
`custom:omniroute`. Cron **"11 paused" → 25 jobs (13 enabled / 12 disabled)**, with **9 names shared with
|
||||
CT460** (8 disabled here, `Important-mail monitor` the reverse).
|
||||
- **`infrastructure/tailscale.md`** — node table rebuilt from `tailscale status --json` (keyed on `DNSName`,
|
||||
IPv4-filtered) and container identity re-confirmed from **inside** each LXC via
|
||||
`pct exec <vmid> -- tailscale ip -4` on both hypervisors. **`octoagent` (100.117.10.68) and
|
||||
`omarchymac-1` (100.126.173.115) were missing entirely** and are now listed — **52 → 54 non-funnel nodes**
|
||||
(53 live IPv4 peers + `hermes`). `m93p` now reads **online** (it was a stale peer record). **Every offline age
|
||||
was refreshed** (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**,
|
||||
`ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**,
|
||||
`openmausbot` 3d → **4d**). The `omni` row now reads "**CT601 removed from both hypervisors**" rather than
|
||||
"CT601 stopped". Serve tables re-probed per host: **CT460 matched live exactly** (12 paths on :443 plus the
|
||||
:8443 Funnel and :9443 route); **CT201 still 502s on `/`** (asset proxy → dead 8082) while `/mgr` and `/steel`
|
||||
return **200** — the 502 claim was re-confirmed, not merely carried over.
|
||||
- **`systems/memory-backend.md`** — re-verified against live `config.yaml`: added `reflect.disabled_for_cron`,
|
||||
`sleep_threshold: 50`, `nudge_interval: 10`, and live totals (**2,350 memories — 842 working / 1,090 episodic /
|
||||
418 `memoria_facts`**). Embedding coverage re-measured: **1,899 / 2,350 (80.8 %)** overall, **working 815 / 842
|
||||
(96.8 %)** and **episodic 1,084 / 1,090 (99.4 %)** — both tiers **above the 95 % reindex threshold**, so no
|
||||
reindex was triggered.
|
||||
- **`runbooks/provider-health.md`** — the step-3 OmniRoute snippet was **broken and raised `IndexError`**: the
|
||||
provider was renamed `omniroute` → **`Omniroute:20128`** and the key moved from an inline `api_key` to a
|
||||
**`key_env`** indirection. The snippet now resolves the key from `key_env`. A **new step 3b** documents the
|
||||
**`omni-101` quirk** — its `/v1/models` returns `Invalid API key` even with the correct key while
|
||||
`/v1/chat/completions` **works** (a `smart-route` call returned `minimaxai/minimax-m3.1-flash`), so a models
|
||||
check on that node is a **false failure**. Also corrected the typo "config slocated" → "located".
|
||||
Re-tested 2026-10-26: LiteLLM 401 unauthed (expected), CT600 OmniRoute 6,480 models, WebUI `/health` 200,
|
||||
opencode/nous/ollama fallbacks all 200.
|
||||
- **`runbooks/update-hermes-safely.md`** — added the **managed-runtime dependency re-assert step**
|
||||
(`hermes-runtime-deps-guard.sh`) after an update, since the updater can leave a runtime without
|
||||
`fastembed`/`dotenv`/`mnemosyne`. Clarified `--branch main` vs the **stable** channel (currently **v0.22.5**,
|
||||
local **v0.22.3**). Version and `last_tested` brought to 2026-10-26.
|
||||
- **`incidents/2026-10-03-ct450-hermes-update.md`** — added a **Current state** section: both instances now run
|
||||
**v0.22.3** (`5b147ab036d6`), CT450 is **0 behind `origin/main`**, and the CT450 Mattermost timeout is still an
|
||||
infrastructure condition (CT255 `stopped`), not an update regression.
|
||||
- **`incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md`** — added a **Current state** section: the
|
||||
follow-up is **satisfied** — `fastembed` 0.8.1 imports in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`,
|
||||
`d41b16c8b6674454a4a819bd4acced1f`) and 0.8.0 in the legacy venv, `FASTEMBED_CACHE_PATH` pinned. Current
|
||||
coverage figures recorded; both tiers above the 95 % threshold, so monitor job `b63ed5817212`
|
||||
(`last_status: ok`) has nothing to fire on.
|
||||
- **All 9 corrected pages verified at `http://127.0.0.1:9120/<path>` — every one returns 200**, and the served
|
||||
`infrastructure/tailscale` contains `octoagent`, `omarchymac-1` and `54 non-funnel`.
|
||||
- **No secrets written.** Job IDs, incident IDs, base URLs, key *env var names*, counts, paths and timestamps only.
|
||||
|
||||
Reference in New Issue
Block a user