--- title: Mnemosyne Vector Recall Ran FTS-Only type: incident status: resolved created: 2026-10-04 updated: 2026-10-04 verified_on: 2026-10-04 confidence: high tags: [mnemosyne, memory, embeddings, fastembed, recall, incident] sources: [live-probe, check_embedding_coverage.py, session-2026-10-03] --- # Mnemosyne Vector Recall Ran FTS-Only ## Summary Until 2026-10-03, `mnemosyne_recall` was scoring **keyword-only**. `fastembed` was missing from every UV-managed runtime, so the dense (vector) leg of recall contributed no candidates. Nothing errored — recall still returned results, just worse ones. Detected and repaired 2026-10-03; re-verified live 2026-10-04. ## Impact - Semantic similarity search silently degraded to keyword/`FTS5` matching. - No user-visible error, no failed cron job, no alert. The failure mode of a missing embedding backend is **quiet recall degradation**, not a crash — which is why it survived unnoticed. - Nothing was lost: the residue is *un-embedded* rows, not missing rows (26 working / 6 episodic as of 2026-10-04). New writes embed normally. ## Root Cause Three overlapping conditions: 1. **`fastembed` absent from the managed runtimes.** The gateway runs from a uv-managed venv under `~/.hermes/installs//environments//venv`, and none of them had `fastembed`. The *legacy* `/home/hermes/.hermes/hermes-agent/venv` had it (0.8.0), which is why CLI-side checks looked healthy while the gateway could not embed. 2. **`FASTEMBED_CACHE_PATH` unset** in `~/.hermes/.env`, so the local model directory was resolved implicitly rather than pinned. 3. **Isolated-mode launches.** The embedding worker can be launched with `python -I`, which ignores `PYTHONPATH`/`PYTHONHOME` — any path-based injection of the dependency does not survive that launch mode. ## Fix (2026-10-03) - Installed `fastembed` (0.8.1) into the runtime the bootstrap actually selects. - Pinned `FASTEMBED_CACHE_PATH` (models cached under `~/.hermes/cache/fastembed`, `BAAI/bge-small-en-v1.5`). - Extended `~/.hermes/scripts/hermes-runtime-deps-guard.sh` — run every 15 min by `mnemosyne-watchdog.py` — so it now asserts, in the bootstrap-selected venv: `fastembed` importable, `FASTEMBED_CACHE_PATH` set, and the model directory present, on top of its existing `dotenv` + `mnemosyne` checks in the installer python and every managed venv. - No gateway restart required. ## Verification (live, 2026-10-04) ``` Memories with embeddings: 1759 / 2194 (80.2%) Working memory: 726 / 752 (96.5%) Episodic memory: 1033 / 1039 (99.4%) vec_episodes_rowids: 1035 vec_working_rowids: 739 Embeddings generated (24h): 13 (working: 7, episodic: 6) ``` `onnxruntime` threads now appear in `~/.hermes/logs/mcp-stderr.log` on each model load — direct evidence the embedding model is executing inside the gateway runtime. The dependency guard exits 0. ## Caveat — do not misread `mnemosyne doctor` `mnemosyne doctor` and the recall CLI run under the **legacy** `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable. Doctor therefore prints `vec0: present_but_unloadable` and marks the working/episodic vector tiers `unavailable`, even while the gateway embeds normally. The managed Python 3.14 venvs cannot import `sqlite_vec` either when probed directly. **Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line.** ## Benign noise `[E:onnxruntime:Default, env.cc:228 ThreadMain] pthread_setaffinity_np failed for thread: … mask: {0,}` — a container CPU-affinity warning emitted on every model load. Not an error; do not chase it. ## Follow-ups - Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%. - Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this. ## Related - [[systems/mnemosyne]] - [[systems/memory-backend]] - [[incidents/2026-09-28-hermes-managed-runtime-deps]] - [[incidents/2026-08-18-mnemosyne-capture-pipeline]]