4.0 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mnemosyne Vector Recall Ran FTS-Only | incident | resolved | 2026-10-04 | 2026-10-04 | 2026-10-04 | high |
|
|
Mnemosyne Vector Recall Ran FTS-Only
Summary
Until 2026-10-03, mnemosyne_recall was scoring keyword-only. fastembed was missing from every UV-managed runtime, so the dense (vector) leg of recall contributed no candidates. Nothing errored — recall still returned results, just worse ones. Detected and repaired 2026-10-03; re-verified live 2026-10-04.
Impact
- Semantic similarity search silently degraded to keyword/
FTS5matching. - No user-visible error, no failed cron job, no alert. The failure mode of a missing embedding backend is quiet recall degradation, not a crash — which is why it survived unnoticed.
- Nothing was lost: the residue is un-embedded rows, not missing rows (26 working / 6 episodic as of 2026-10-04). New writes embed normally.
Root Cause
Three overlapping conditions:
fastembedabsent from the managed runtimes. The gateway runs from a uv-managed venv under~/.hermes/installs/<install>/environments/<hash>/venv, and none of them hadfastembed. The legacy/home/hermes/.hermes/hermes-agent/venvhad it (0.8.0), which is why CLI-side checks looked healthy while the gateway could not embed.FASTEMBED_CACHE_PATHunset in~/.hermes/.env, so the local model directory was resolved implicitly rather than pinned.- Isolated-mode launches. The embedding worker can be launched with
python -I, which ignoresPYTHONPATH/PYTHONHOME— any path-based injection of the dependency does not survive that launch mode.
Fix (2026-10-03)
- Installed
fastembed(0.8.1) into the runtime the bootstrap actually selects. - Pinned
FASTEMBED_CACHE_PATH(models cached under~/.hermes/cache/fastembed,BAAI/bge-small-en-v1.5). - Extended
~/.hermes/scripts/hermes-runtime-deps-guard.sh— run every 15 min bymnemosyne-watchdog.py— so it now asserts, in the bootstrap-selected venv:fastembedimportable,FASTEMBED_CACHE_PATHset, and the model directory present, on top of its existingdotenv+mnemosynechecks in the installer python and every managed venv. - No gateway restart required.
Verification (live, 2026-10-04)
Memories with embeddings: 1759 / 2194 (80.2%)
Working memory: 726 / 752 (96.5%)
Episodic memory: 1033 / 1039 (99.4%)
vec_episodes_rowids: 1035
vec_working_rowids: 739
Embeddings generated (24h): 13 (working: 7, episodic: 6)
onnxruntime threads now appear in ~/.hermes/logs/mcp-stderr.log on each model load — direct evidence the embedding model is executing inside the gateway runtime. The dependency guard exits 0.
Caveat — do not misread mnemosyne doctor
mnemosyne doctor and the recall CLI run under the legacy venv/ (Python 3.11), where the sqlite_vec extension is not importable. Doctor therefore prints vec0: present_but_unloadable and marks the working/episodic vector tiers unavailable, even while the gateway embeds normally. The managed Python 3.14 venvs cannot import sqlite_vec either when probed directly. Judge vector health with ~/.hermes/scripts/check_embedding_coverage.py, not with doctor's vec0 line.
Benign noise
[E:onnxruntime:Default, env.cc:228 ThreadMain] pthread_setaffinity_np failed for thread: … mask: {0,} — a container CPU-affinity warning emitted on every model load. Not an error; do not chase it.
Follow-ups
- Confirm the embedding-coverage alert job (
b63ed5817212, every 6h, Telegram topic 102) still fires when working coverage drops below 95%. - Consider asserting
fastembedin all managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.