Files
agent-estate-wiki/incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
T

5.7 KiB

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Mnemosyne Vector Recall Ran FTS-Only incident resolved 2026-10-04 2026-11-18 2026-11-18 high
mnemosyne
memory
embeddings
fastembed
recall
incident
live-probe
check_embedding_coverage.py
session-2026-10-03

Mnemosyne Vector Recall Ran FTS-Only

Summary

Until 2026-10-03, mnemosyne_recall was scoring keyword-only. fastembed was missing from every UV-managed runtime, so the dense (vector) leg of recall contributed no candidates. Nothing errored — recall still returned results, just worse ones. Detected and repaired 2026-10-03; re-verified live 2026-10-04.

Impact

  • Semantic similarity search silently degraded to keyword/FTS5 matching.
  • No user-visible error, no failed cron job, no alert. The failure mode of a missing embedding backend is quiet recall degradation, not a crash — which is why it survived unnoticed.
  • Nothing was lost: the residue is un-embedded rows, not missing rows (26 working / 6 episodic as of 2026-10-04). New writes embed normally.

Root Cause

Three overlapping conditions:

  1. fastembed absent from the managed runtimes. The gateway runs from a uv-managed venv under ~/.hermes/installs/<install>/environments/<hash>/venv, and none of them had fastembed. The legacy /home/hermes/.hermes/hermes-agent/venv had it (0.8.0), which is why CLI-side checks looked healthy while the gateway could not embed.
  2. FASTEMBED_CACHE_PATH unset in ~/.hermes/.env, so the local model directory was resolved implicitly rather than pinned.
  3. Isolated-mode launches. The embedding worker can be launched with python -I, which ignores PYTHONPATH/PYTHONHOME — any path-based injection of the dependency does not survive that launch mode.

Fix (2026-10-03)

  • Installed fastembed (0.8.1) into the runtime the bootstrap actually selects.
  • Pinned FASTEMBED_CACHE_PATH (models cached under ~/.hermes/cache/fastembed, BAAI/bge-small-en-v1.5).
  • Extended ~/.hermes/scripts/hermes-runtime-deps-guard.sh — run every 15 min by mnemosyne-watchdog.py — so it now asserts, in the bootstrap-selected venv: fastembed importable, FASTEMBED_CACHE_PATH set, and the model directory present, on top of its existing dotenv + mnemosyne checks in the installer python and every managed venv.
  • No gateway restart required.

Verification (live, 2026-10-04)

Memories with embeddings:   1759 / 2194 (80.2%)
Working memory:              726 / 752  (96.5%)
Episodic memory:            1033 / 1039 (99.4%)
vec_episodes_rowids:        1035
vec_working_rowids:          739
Embeddings generated (24h):   13  (working: 7, episodic: 6)

onnxruntime threads now appear in ~/.hermes/logs/mcp-stderr.log on each model load — direct evidence the embedding model is executing inside the gateway runtime. The dependency guard exits 0.

Caveat — do not misread mnemosyne doctor

mnemosyne doctor and the recall CLI run under the legacy venv/ (Python 3.11), where the sqlite_vec extension is not importable. Doctor therefore prints vec0: present_but_unloadable and marks the working/episodic vector tiers unavailable, even while the gateway embeds normally. The managed Python 3.14 venvs cannot import sqlite_vec either when probed directly. Judge vector health with ~/.hermes/scripts/check_embedding_coverage.py, not with doctor's vec0 line.

Benign noise

[E:onnxruntime:Default, env.cc:228 ThreadMain] pthread_setaffinity_np failed for thread: … mask: {0,} — a container CPU-affinity warning emitted on every model load. Not an error; do not chase it.

Current state (re-verified 2026-10-26)

The 2026-10-03 follow-up is satisfied: fastembed 0.8.1 is importable in both managed runtimes (19da945341ad4794ac2bc8ff9fdd3643, d41b16c8b6674454a4a819bd4acced1f) as well as 0.8.0 in the legacy venv, and FASTEMBED_CACHE_PATH is pinned in ~/.hermes/.env.

Current check_embedding_coverage.py output:

Memories with embeddings:   1899 / 2350 (80.8%)
Working memory:               815 / 842  (96.8%)
Episodic memory:             1084 / 1090 (99.4%)
vec_episodes_rowids:         1086
vec_working_rowids:           828
Embeddings generated (24h):    15 (working: 9, episodic: 6)

Both tiers are above the 95 % working-coverage reindex threshold, so the 6-hourly monitor job b63ed5817212 (last_status: ok) has nothing to fire on. The 2026-10-04 figures below are superseded.

Current state (re-verified 2026-11-18)

The fix continues to hold. The 6-hourly coverage monitor b63ed5817212 is enabled, last_status: ok, failure_streak: 0, last run 2026-11-18 00:04 — it has had nothing to fire on, so working coverage remains at/above the 95 % threshold. (The coverage script itself could not be re-run this session — the gateway's lifecycle guard blocks scripts touching the >1 MiB Mnemosyne DB — so the monitor's clean streak is the verification, per the monitor's own design.)

Follow-ups

  • Confirm the embedding-coverage alert job (b63ed5817212, every 6h, Telegram topic 102) still fires when working coverage drops below 95%. CONFIRMED 2026-10-26 — the job is enabled with last_status: ok; working coverage is 96.8 %, above the threshold, so it correctly has nothing to fire on.
  • Consider asserting fastembed in all managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this. DONE 2026-10-26 — fastembed 0.8.1 now imports in both managed runtimes (19da945341ad4794ac2bc8ff9fdd3643, d41b16c8b6674454a4a819bd4acced1f) as well as 0.8.0 in the legacy venv.