stocktake(2026-10-26): correct 8 stale pages against live state — 0 stale remaining
wiki-maintain.sh ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted, service restarted). The freshness report listed 8 stale pages; each was diffed against live state and corrected only where reality could be verified. A second script run rewrote the same-day report with 0 stale / 103 fresh / 4 no-date, so the committed report matches the tree it ships with. - infrastructure/hosts.md: CT600 is LIVE (not retired); CT601 does not exist on either hypervisor; CT333 verified running; both hypervisors verified; config 33->51; skills 66->302; cron 25->28 (25 enabled / 3 disabled); Xvfb :102->:99; providers split into Omniroute:20128 (CT600) and omni-101 (CT101, default). - agents/hermes-production.md: Tailscale IP 100.96.100.82 -> 100.118.5.51 (the old value is the PVE hypervisor); v0.21.5 -> v0.22.3; dead 8082 claim removed; fallback chain rewritten from live config (first entry is the dead nous credential); all 8 auxiliary slots now no-think/freeapi/auto; cron 30->28. - agents/hermes-chaos.md: v0.22.3 (same build as CT460); "3 commits behind" -> 0 behind origin/master; default model smart-route on custom:omni-101; cron "11 paused" -> 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460. - infrastructure/tailscale.md: node table rebuilt from status --json; octoagent and omarchymac-1 were missing entirely (52 -> 54 non-funnel nodes); m93p now online; every offline age refreshed; omni row corrected to "CT601 removed from both hypervisors"; Serve tables re-probed per host. - systems/memory-backend.md: re-verified against live config; coverage 1899/2350 (80.8%), working 96.8%, episodic 99.4% — above the 95% reindex threshold. - runbooks/provider-health.md: the step-3 OmniRoute snippet raised IndexError (provider renamed omniroute -> Omniroute:20128; key moved to key_env) — fixed; new step 3b documents the omni-101 /v1/models false-failure quirk. - runbooks/update-hermes-safely.md: added managed-runtime dependency re-assert step; clarified --branch vs stable channel (v0.22.5 available). - incidents/2026-10-03-ct450-hermes-update.md + incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md: Current-state sections recording current versions and embedding coverage. All 9 corrected pages verified 200 at http://127.0.0.1:9120/<path>. Vault mirror at /home/hermes/wiki synced for all touched files (22 copies, sha256-verified) and committed locally; the vault is local-only and is not pushed. No secrets written — IDs, URLs, key env var names, counts and timestamps only.
This commit is contained in:
@@ -3,8 +3,8 @@ title: CT450 Hermes Update to v0.21.5
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-04
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
confidence: high
|
||||
tags: [hermes, ct450, chaos, update, incident]
|
||||
sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
|
||||
@@ -48,6 +48,9 @@ Two gotchas that cost time:
|
||||
- `clang` present; no stale locks present.
|
||||
- CT460 production untouched — separate config, skills, cron and memory state.
|
||||
|
||||
## Current state (re-verified 2026-10-26)
|
||||
Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.
|
||||
|
||||
## Follow-ups
|
||||
- CT450 reports `Update available: 3 commits behind`. Harmless; re-run `hermes update` if the chaos box should track upstream exactly.
|
||||
- CT450's Mattermost connection is failing (`mattermost connect timed out after 30s`, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See [[agents/hermes-chaos]].
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Mnemosyne Vector Recall Ran FTS-Only
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-04
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
|
||||
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
|
||||
@@ -49,6 +49,22 @@ Embeddings generated (24h): 13 (working: 7, episodic: 6)
|
||||
## Benign noise
|
||||
`[E:onnxruntime:Default, env.cc:228 ThreadMain] pthread_setaffinity_np failed for thread: … mask: {0,}` — a container CPU-affinity warning emitted on every model load. Not an error; do not chase it.
|
||||
|
||||
## Current state (re-verified 2026-10-26)
|
||||
The 2026-10-03 follow-up is **satisfied**: `fastembed` 0.8.1 is importable in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`) as well as 0.8.0 in the legacy venv, and `FASTEMBED_CACHE_PATH` is pinned in `~/.hermes/.env`.
|
||||
|
||||
Current `check_embedding_coverage.py` output:
|
||||
```
|
||||
Memories with embeddings: 1899 / 2350 (80.8%)
|
||||
Working memory: 815 / 842 (96.8%)
|
||||
Episodic memory: 1084 / 1090 (99.4%)
|
||||
vec_episodes_rowids: 1086
|
||||
vec_working_rowids: 828
|
||||
Embeddings generated (24h): 15 (working: 9, episodic: 6)
|
||||
```
|
||||
Both tiers are **above the 95 % working-coverage reindex threshold**, so the 6-hourly
|
||||
monitor job `b63ed5817212` (`last_status: ok`) has nothing to fire on. The 2026-10-04
|
||||
figures below are superseded.
|
||||
|
||||
## Follow-ups
|
||||
- Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.
|
||||
- Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.
|
||||
|
||||
Reference in New Issue
Block a user