wiki audit 2026-10-04: CT450 v0.21.5 update incident, mnemosyne vector-recall FTS-only repair + coverage refresh, managed runtimes 10->3, doctor vec0 caveat
This commit is contained in:
@@ -0,0 +1,59 @@
|
||||
---
|
||||
title: CT450 Hermes Update to v0.21.5
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-04
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
confidence: high
|
||||
tags: [hermes, ct450, chaos, update, incident]
|
||||
sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
|
||||
---
|
||||
|
||||
# CT450 Hermes Update to v0.21.5
|
||||
|
||||
## Summary
|
||||
The Chaos instance (LXC CT450, `hermesagent`) was updated **v0.19.0 → v0.21.5** on 2026-10-03 — its first update since July. CT460 (production) was rebuilt the same morning, so both instances now carry the same semver token from separate builds.
|
||||
|
||||
## Result (live-verified 2026-10-04)
|
||||
| | CT460 (production) | CT450 (chaos) |
|
||||
|---|---|---|
|
||||
| Version | `v0.21.5+6169.gab86a20` (2026.9.24) | `v0.21.5+6170.ga510d6c` (2026.9.24) |
|
||||
| Upstream | `ab86a20a` | `a510d6c6` — 3 commits behind, `hermes update` available |
|
||||
| Python | managed runtime 3.14 (legacy `venv/` 3.11.15) | 3.14.7 |
|
||||
| Install tree mtime | 2026-10-03 10:11 AWST | 2026-10-03 10:22 AWST |
|
||||
|
||||
## Access
|
||||
**CT450 has no SSH.** Reach it through the PVE host:
|
||||
|
||||
```bash
|
||||
ssh root@100.96.100.82
|
||||
pct exec 450 -- <command> # or: pct enter 450
|
||||
pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"
|
||||
```
|
||||
|
||||
Two gotchas that cost time:
|
||||
- `pct exec 450 -- hermes ...` **fails** with `Failed to exec "hermes"` — the binary is not on `lxc-attach`'s default PATH. Use the absolute path, or `su - hermes -c` (the dash matters: only a login shell puts `~/.local/bin` on PATH).
|
||||
- A `hermes` login shell prints a `hermes-setup` banner ahead of command output; parse with `| tail -n` rather than expecting clean stdout.
|
||||
|
||||
## Pre-flight checks (each cost one failed run on 2026-10-03)
|
||||
1. **Build toolchain.** `clang` must be present *before* updating — source-compiled Python dependencies will not build without it. Verified present on CT450 (`/usr/bin/clang`, 2026-10-04).
|
||||
2. **Stale update locks.** A leftover `.install.lock` / `.lazy-refresh-incomplete` in the Hermes home makes the updater refuse or silently skip the refresh step. Neither exists on CT450 now.
|
||||
3. **Install-tree ownership.** `~/.hermes/hermes-agent` must stay owned by `hermes:hermes` (uid 1000); a root-owned tree left by an earlier root-run update fails the in-place build.
|
||||
4. **Caddy keyring permissions.** Host-level Caddy/`keyring` file permissions (`0600`) blocked a step mid-update — check them before starting, not during.
|
||||
|
||||
## Verification
|
||||
- `hermes --version` on CT450 → `Hermes Agent v0.21.5+6170.ga510d6c (2026.9.24) · upstream a510d6c6`, Python 3.14.7.
|
||||
- Install tree mtime `2026-10-03 10:22` matches the update window.
|
||||
- `clang` present; no stale locks present.
|
||||
- CT460 production untouched — separate config, skills, cron and memory state.
|
||||
|
||||
## Follow-ups
|
||||
- CT450 reports `Update available: 3 commits behind`. Harmless; re-run `hermes update` if the chaos box should track upstream exactly.
|
||||
- CT450's Mattermost connection is failing (`mattermost connect timed out after 30s`, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See [[agents/hermes-chaos]].
|
||||
|
||||
## Related
|
||||
- [[agents/hermes-chaos]]
|
||||
- [[agents/hermes-production]]
|
||||
- [[runbooks/update-hermes-safely]]
|
||||
- [[infrastructure/hosts]]
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
title: Mnemosyne Vector Recall Ran FTS-Only
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-04
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
|
||||
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
|
||||
---
|
||||
|
||||
# Mnemosyne Vector Recall Ran FTS-Only
|
||||
|
||||
## Summary
|
||||
Until 2026-10-03, `mnemosyne_recall` was scoring **keyword-only**. `fastembed` was missing from every UV-managed runtime, so the dense (vector) leg of recall contributed no candidates. Nothing errored — recall still returned results, just worse ones. Detected and repaired 2026-10-03; re-verified live 2026-10-04.
|
||||
|
||||
## Impact
|
||||
- Semantic similarity search silently degraded to keyword/`FTS5` matching.
|
||||
- No user-visible error, no failed cron job, no alert. The failure mode of a missing embedding backend is **quiet recall degradation**, not a crash — which is why it survived unnoticed.
|
||||
- Nothing was lost: the residue is *un-embedded* rows, not missing rows (26 working / 6 episodic as of 2026-10-04). New writes embed normally.
|
||||
|
||||
## Root Cause
|
||||
Three overlapping conditions:
|
||||
1. **`fastembed` absent from the managed runtimes.** The gateway runs from a uv-managed venv under `~/.hermes/installs/<install>/environments/<hash>/venv`, and none of them had `fastembed`. The *legacy* `/home/hermes/.hermes/hermes-agent/venv` had it (0.8.0), which is why CLI-side checks looked healthy while the gateway could not embed.
|
||||
2. **`FASTEMBED_CACHE_PATH` unset** in `~/.hermes/.env`, so the local model directory was resolved implicitly rather than pinned.
|
||||
3. **Isolated-mode launches.** The embedding worker can be launched with `python -I`, which ignores `PYTHONPATH`/`PYTHONHOME` — any path-based injection of the dependency does not survive that launch mode.
|
||||
|
||||
## Fix (2026-10-03)
|
||||
- Installed `fastembed` (0.8.1) into the runtime the bootstrap actually selects.
|
||||
- Pinned `FASTEMBED_CACHE_PATH` (models cached under `~/.hermes/cache/fastembed`, `BAAI/bge-small-en-v1.5`).
|
||||
- Extended `~/.hermes/scripts/hermes-runtime-deps-guard.sh` — run every 15 min by `mnemosyne-watchdog.py` — so it now asserts, in the bootstrap-selected venv: `fastembed` importable, `FASTEMBED_CACHE_PATH` set, and the model directory present, on top of its existing `dotenv` + `mnemosyne` checks in the installer python and every managed venv.
|
||||
- No gateway restart required.
|
||||
|
||||
## Verification (live, 2026-10-04)
|
||||
```
|
||||
Memories with embeddings: 1759 / 2194 (80.2%)
|
||||
Working memory: 726 / 752 (96.5%)
|
||||
Episodic memory: 1033 / 1039 (99.4%)
|
||||
vec_episodes_rowids: 1035
|
||||
vec_working_rowids: 739
|
||||
Embeddings generated (24h): 13 (working: 7, episodic: 6)
|
||||
```
|
||||
`onnxruntime` threads now appear in `~/.hermes/logs/mcp-stderr.log` on each model load — direct evidence the embedding model is executing inside the gateway runtime. The dependency guard exits 0.
|
||||
|
||||
## Caveat — do not misread `mnemosyne doctor`
|
||||
`mnemosyne doctor` and the recall CLI run under the **legacy** `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable. Doctor therefore prints `vec0: present_but_unloadable` and marks the working/episodic vector tiers `unavailable`, even while the gateway embeds normally. The managed Python 3.14 venvs cannot import `sqlite_vec` either when probed directly. **Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line.**
|
||||
|
||||
## Benign noise
|
||||
`[E:onnxruntime:Default, env.cc:228 ThreadMain] pthread_setaffinity_np failed for thread: … mask: {0,}` — a container CPU-affinity warning emitted on every model load. Not an error; do not chase it.
|
||||
|
||||
## Follow-ups
|
||||
- Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.
|
||||
- Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.
|
||||
|
||||
## Related
|
||||
- [[systems/mnemosyne]]
|
||||
- [[systems/memory-backend]]
|
||||
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
|
||||
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]]
|
||||
Reference in New Issue
Block a user