Embedding coverage (live, 2026-10-04): 1,759/2,194 memories embedded (80.2%) — working 726/752 (96.5%), episodic 1,033/1,039 (99.4%); 13 embeddings generated in the previous 24h. Working coverage is above the 95% alert threshold enforced by the Embedding Coverage job (b63ed5817212, every 6h, Telegram topic 102).
Known Issues
Vector embedding coverage gap — RESOLVED (2026-08-02)
Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
Resolved by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is 96.5% working / 99.4% episodic, and new embeddings are still being generated. The figures above are kept as the before-state only.
Standard repair if coverage drops below 95% working: ~/.hermes/hermes-agent/venv/bin/mnemosyne reindex --yes (the Embedding Coverage job b63ed5817212 triggers the same path every 6h).
Vector recall silently ran FTS-only (2026-10-03 — repaired)
fastembed was absent from every uv-managed runtime, so the dense leg of mnemosyne_recall produced no candidates and recall fell back to FTS5/keyword scoring without raising an error. Sessions kept working, which is why it went unnoticed.
Compounded by FASTEMBED_CACHE_PATH being unset in ~/.hermes/.env, and by the embedding worker sometimes launching in isolated (python -I) mode, which does not inherit PYTHONPATH.
Repaired 2026-10-03:fastembed installed into the bootstrap-selected runtime, FASTEMBED_CACHE_PATH pinned, and hermes-runtime-deps-guard.sh extended to assert fastembed + cache path + model dir every 15 min alongside its existing dotenv/mnemosyne checks.
CLI caveat — do not misread mnemosyne doctor. Doctor and the recall CLI run under the legacy venv/ (Python 3.11), where the sqlite_vec extension is not importable; doctor therefore prints vec0: present_but_unloadable and marks the vector tiers unavailable even while the gateway embeds normally. Judge vector health with ~/.hermes/scripts/check_embedding_coverage.py, not with doctor's vec0 line.
Benign noise: pthread_setaffinity_np failed … mask: {0,} from onnxruntime in mcp-stderr.log is a container affinity warning emitted on every model load — not an error.
|- The ops profile was enabled with mnemosyne on 2026-08-02
|- Decision:profile_isolation: true was set on ops, giving each profile its own independent memory bank — zero cross-contamination
|- data/shared/mnemosyne.db exists with 0 rows; cross-profile sharing is intentionally unused
|- This means Tusk (ops) and Rhino (default) do not share memories, which is the desired behaviour
|- Config in ~/.hermes/profiles/ops/config.yaml: memory.mnemosyne.profile_isolation: true, allow_tool_override: false, shared_surface_read: false
|- Verified end-to-end on 2026-08-02: memory writes land correctly in both profiles independently
Legacy table not migrated
2,181 rows in the old memory table are not in the new working_memory/episodic_memory system
These are only accessible via the legacy memory tool, not via mnemosyne_recall
Gateway notifications were captured as user memories (fixed 2026-09-30)
Hermes's background-process notifications (long-running terminal jobs reporting completion) were being written into Mnemosyne as [USER] [Anthony Martin] memories — i.e. machine messages were stored as if the user had said them, and then surfaced during recall as if they were his words.
36 such rows were found and purged on 2026-09-30 (working tier plus the corresponding episodic rows). Recall after the purge returned clean.
Two canonical facts were also wrong because of the same class of drift: hermes_cron_model_default named the removed provider custom:omni instead of custom:omni-101, and the Daily Marketing Job Radar's model fact was stale. Both corrected.
Rule: treat a recalled memory attributed to a person as untrusted until the attribution makes sense in context. Machine-generated status text should never enter the person-attributed tier.
Known Management Jobs
Mnemosyne Auto-Consolidate every 30m — script: consolidate_mnemosyne.sh, mode: no-agent
Mnemosyne Health Monitor every 12h — skill: mnemosyne-operations, reports only on exception
Mnemosyne Embedding Coverage every 6h — job b63ed5817212, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
Mnemosyne Watchdog every 15m — script: mnemosyne-watchdog.py, mode: no-agent
History and Evidence
2026-10-03: vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and incidents/2026-10-03-mnemosyne-vector-recall-fts-only.
2026-09-30 memory-hygiene pass: 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (hermes_cron_model_default → custom:omni-101, job-radar model). No unresolved conflicts left in the working tier.
2026-08-19 Mnemosyne Watchdog incident: Watchdog job (96eb95ce12f2) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See systems/scheduled-tasks Known Issues section.
2026-08-18 incident: Working memory capture (sync_turn) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in run_agent.py:_sync_external_memory_for_turn(). Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See incidents/2026-08-18-mnemosyne-capture-pipeline.
2026-08-01 incident: Automatic conversational capture was broken by stringified structured YAML, especially sync_roles: "['user']". Fixed by rewriting with native YAML types. See incidents/2026-08-01-mnemosyne-capture-failure.
2026-08-01 cleanup: Legacy always-injected memory reduced from 8,013 bytes to 4,998 bytes (memory) and 2,958 bytes (user).
2026-07-29 incident:memory.mnemosyne.tools: None caused schema loading warnings. Fixed by removing the line. See incidents/2026-07-29-mnemosyne-relay.
2026-07-29 relay scope corruption:RuntimeError: scope handle is not at the top of the stack — 3 occurrences, self-resolved.