wiki audit 2026-10-04: CT450 v0.21.5 update incident, mnemosyne vector-recall FTS-only repair + coverage refresh, managed runtimes 10->3, doctor vec0 caveat
This commit is contained in:
+25
-14
@@ -3,8 +3,8 @@ title: Mnemosyne
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-01
|
||||
verified_on: 2026-10-01
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, backend]
|
||||
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
||||
@@ -33,25 +33,34 @@ Primary agent memory store for Hermes.
|
||||
- **Embedding model:** `BAAI/bge-small-en-v1.5` (384 dims)
|
||||
- **Embeddings via API:** false (local fastembed)
|
||||
|
||||
## Database Stats (2026-08-02)
|
||||
## Database Stats (live, 2026-10-04)
|
||||
| Table | Rows | Notes |
|
||||
|---|---|---|
|
||||
| `working_memory` | 1,476 | Active memories |
|
||||
| `episodic_memory` | 458 | Consolidated summaries |
|
||||
| `memory_embeddings` | 158 | Only working memories — **0 episodic** |
|
||||
| `working_memory` | 752 | Active memories |
|
||||
| `episodic_memory` | 1,039 | Consolidated summaries |
|
||||
| `memory_embeddings` | 739 | Working-tier embeddings |
|
||||
| `memory` (legacy) | 2,181 | Old table, not migrated |
|
||||
| `vec_working` | 159 | Vector index rows |
|
||||
| `vec_working` / `vec_working_rowids` | 739 | Working vector index |
|
||||
| `vec_episodes` / `vec_episodes_rowids` | 1,035 | Episodic vector index |
|
||||
| `vec_facts_rowids` | 0 | `memoria_facts` is intentionally left unvectorised |
|
||||
| Shared surface | 0 | Empty — cross-agent sharing not functional |
|
||||
|
||||
**Embedding coverage (live, 2026-10-04):** 1,759/2,194 memories embedded (80.2%) — working **726/752 (96.5%)**, episodic **1,033/1,039 (99.4%)**; 13 embeddings generated in the previous 24h. Working coverage is above the 95% alert threshold enforced by the Embedding Coverage job (`b63ed5817212`, every 6h, Telegram topic 102).
|
||||
|
||||
## Known Issues
|
||||
|
||||
### Vector embedding coverage gap
|
||||
- **Only 159/1,476 working memories (10.8%)** have vector embeddings
|
||||
- **0/458 episodic memories** have embeddings
|
||||
- Semantic search (`mnemosyne_recall`) falls back to FTS5 keyword matching for ~90% of working and 100% of episodic memories
|
||||
- This means recall quality is degraded — vector similarity can only search a tiny fraction of the memory store
|
||||
- **Root cause:** The embedding pipeline (fastembed) never processed the bulk of memories. The `vec_working` table was rebuilt once (Jul 18) but only captured 159 rows.
|
||||
- **Fix needed:** Backfill embeddings for all working and episodic memories. This requires running the embedding pipeline across the full dataset.
|
||||
### Vector embedding coverage gap — RESOLVED (2026-08-02)
|
||||
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
|
||||
- **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only.
|
||||
- Standard repair if coverage drops below 95% working: `~/.hermes/hermes-agent/venv/bin/mnemosyne reindex --yes` (the Embedding Coverage job `b63ed5817212` triggers the same path every 6h).
|
||||
|
||||
### Vector recall silently ran FTS-only (2026-10-03 — repaired)
|
||||
- `fastembed` was absent from **every** uv-managed runtime, so the dense leg of `mnemosyne_recall` produced no candidates and recall fell back to FTS5/keyword scoring **without raising an error**. Sessions kept working, which is why it went unnoticed.
|
||||
- Compounded by `FASTEMBED_CACHE_PATH` being unset in `~/.hermes/.env`, and by the embedding worker sometimes launching in isolated (`python -I`) mode, which does not inherit `PYTHONPATH`.
|
||||
- **Repaired 2026-10-03:** `fastembed` installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min alongside its existing `dotenv`/`mnemosyne` checks.
|
||||
- **CLI caveat — do not misread `mnemosyne doctor`.** Doctor and the recall CLI run under the legacy `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable; doctor therefore prints `vec0: present_but_unloadable` and marks the vector tiers `unavailable` even while the gateway embeds normally. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line.
|
||||
- Benign noise: `pthread_setaffinity_np failed … mask: {0,}` from onnxruntime in `mcp-stderr.log` is a container affinity warning emitted on every model load — not an error.
|
||||
- See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||||
|
||||
### Profile isolation chosen over shared surface
|
||||
|- The ops profile was enabled with mnemosyne on 2026-08-02
|
||||
@@ -75,9 +84,11 @@ Primary agent memory store for Hermes.
|
||||
- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
|
||||
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
|
||||
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception
|
||||
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
|
||||
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
|
||||
|
||||
## History and Evidence
|
||||
- **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||||
- **2026-09-30 memory-hygiene pass:** 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (`hermes_cron_model_default` → `custom:omni-101`, job-radar model). No unresolved conflicts left in the working tier.
|
||||
- **2026-08-19 Mnemosyne Watchdog incident:** Watchdog job (`96eb95ce12f2`) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See [[systems/scheduled-tasks]] Known Issues section.
|
||||
- **2026-08-18 incident:** Working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See [[incidents/2026-08-18-mnemosyne-capture-pipeline]].
|
||||
|
||||
Reference in New Issue
Block a user