--- title: Mnemosyne type: system status: active created: 2026-07-22 updated: 2026-11-15 verified_on: 2026-11-15 review_after: 2026-11-13 confidence: high tags: [memory, mnemosyne, backend] sources: [live-probe, mnemosyne-db, config, ops-profile-config, live-secrets-scan, live-state-db, live-health-monitor-log] --- # Mnemosyne ## Purpose Primary agent memory store for Hermes. ## Current Configuration - **Provider:** `mnemosyne` - **Enabled:** true - **Auto sleep:** true - **Default scope:** `session` - **Memory char limit:** 5000 - **User char limit:** 3000 - **Vector type:** int8 - **Profile isolation:** true - **Shared surface read:** true - **Shared surface path:** `data/shared/mnemosyne.db` - **Skip contexts:** cron, flush, subagent, background, skill_loop - **Reflect max calls per session:** 3 - **Sync roles:** user - **Flush every:** 6 turns - **Embedding model:** `BAAI/bge-small-en-v1.5` (384 dims) - **Embeddings via API:** false (local fastembed) ## Database Stats (live, 2026-10-04) | Table | Rows | Notes | |---|---|---| | `working_memory` | 773 | Active memories | | `episodic_memory` | 1,047 | Consolidated summaries | | `memory_embeddings` | 760 | Working-tier embeddings | | `memory` (legacy) | 2,181 | Old table, not migrated | | `vec_working` / `vec_working_rowids` | 760 | Working vector index | | `vec_episodes` / `vec_episodes_rowids` | 1,035 | Episodic vector index | | `vec_facts_rowids` | 0 | `memoria_facts` is intentionally left unvectorised | | Shared surface | 0 | Empty — cross-agent sharing not functional | **Embedding coverage (live, 2026-10-04):** 1,759/2,194 memories embedded (80.2%) — working **726/752 (96.5%)**, episodic **1,033/1,039 (99.4%)**; 13 embeddings generated in the previous 24h. Working coverage is above the 95% alert threshold enforced by the Embedding Coverage job (`b63ed5817212`, every 6h, Telegram topic 102). ## Known Issues **⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed; its `failure_type: auth` is the scheduler's classifier, not evidence of a credential fault). **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, delivery `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run itself** (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, `-589.5` at line 1046, ❌ at 1047). `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`, default **120** min) but **activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last captured message was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the **19:22 AWST desktop** turn — after the 12:00 slot. **The quiet was the `0 */12` cadence, not a sign-vs-magnitude bug.** **What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, printed as `(0 items)` on every run yet tested only for age (`> 36 h`) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: `consolidation_log.created_at` mixes formats (recent rows `+00:00`, older rows naive **AWST**), so treating the column as one axis misplaces events by 8 h. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). → [[incidents/2026-11-14-mnemosyne-capture-stall]] **⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**. **Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below. **Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. ~~Ranking relevance is the live issue, not coverage~~ — see [[incidents/2026-10-09-memory-recall-relevance]]. **Tier counts (live, 2026-11-01 03:0x — the 2026-10-09 ranking incident is now RESOLVED):** working `941`, episodic `1,124`, `canonical_facts` `90`, `conflicts` `175` (**11 open** — ids **165–175**, all `contradiction`, all created in one batch at 2026-10-31 19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature), working embeddings `927/941 (98.5%)`, episodic embeddings `1,124/1,124 (100%)` — both tiers above the 95% floor, no `reindex` needed. **The recall weights are live in `~/.hermes/mnemosyne/config.yaml`** (`vec_weight: 0.5`, `fts_weight: 0.3`, `importance_weight: 0.3`, plus `MNEMOSYNE_IMPORTANCE_WEIGHT=0.3` in the environment) — **not** in `~/.hermes/config.yaml`'s `memory.mnemosyne` block, which is why the 2026-10-10 probe wrongly concluded the fix was never applied. The original regression query now returns only substantive current-state facts; the stale Uber Eats episode no longer outranks them → [[incidents/2026-10-09-memory-recall-relevance]] (**resolved**). **Tier counts (live, 2026-10-12 12:02 — health job `ea79a2107e36`, green):** working **760/786 embedded (96.7%)**, episodic **1050/1056 (99.4%)** — both above the 95% floor, no `reindex` needed (`b63ed5817212` continues to backfill). Orphan embeddings **0**. FTS5 exactly aligned: `fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876. **0 unconsolidated** working rows; **1 unresolved conflict** (threshold 10); last consolidation 0.3 h earlier; 2,252 memories total (786 working / 1,056 episodic / 1 triple); disk 15 GB free (81% used). Interpretation: coverage is *not* the live issue — ranking relevance is (see [[incidents/2026-10-09-memory-recall-relevance]]). **Tier counts (live, 2026-10-12 00:1x):** working `786`, episodic `1,056`, `facts` `876` (`fts_facts` 876 — FTS aligned), `fts_working` 786, `memoria_facts` `410`, `conflicts` `152`. Embeddings: working `773/786 (98.3%)`, episodic `1,052/1,056 (99.6%)` — both above the 95% reindex threshold; the coverage job `b63ed5817212` backfilled 15 vectors since the 2026-10-11 00:08 check. **Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST. **2026-11-14/15 recurrence, new flavour:** the monitor's own escalation text printed `last capture 2026-11-14 01:32:59` (raw UTC) for a write that landed at **09:32:59 AWST**, so the alert *reads* as a ~23 h hole when the true gap was **14 h 31 m** — the same trap, now inside the monitor's output rather than the audit's reading → [[incidents/2026-11-14-mnemosyne-capture-stall]]. ### Vector embedding coverage gap — RESOLVED (2026-08-02) - Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories. - **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only. - Standard repair if coverage drops below 95% working: `~/.hermes/hermes-agent/venv/bin/mnemosyne reindex --yes` (the Embedding Coverage job `b63ed5817212` triggers the same path every 6h). ### Vector recall silently ran FTS-only (2026-10-03 — repaired) - `fastembed` was absent from **every** uv-managed runtime, so the dense leg of `mnemosyne_recall` produced no candidates and recall fell back to FTS5/keyword scoring **without raising an error**. Sessions kept working, which is why it went unnoticed. - Compounded by `FASTEMBED_CACHE_PATH` being unset in `~/.hermes/.env`, and by the embedding worker sometimes launching in isolated (`python -I`) mode, which does not inherit `PYTHONPATH`. - **Repaired 2026-10-03:** `fastembed` installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min alongside its existing `dotenv`/`mnemosyne` checks. - **CLI caveat — do not misread `mnemosyne doctor`.** Doctor and the recall CLI run under the legacy `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable; doctor therefore prints `vec0: present_but_unloadable` and marks the vector tiers `unavailable` even while the gateway embeds normally. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line. - Benign noise: `pthread_setaffinity_np failed … mask: {0,}` from onnxruntime in `mcp-stderr.log` is a container affinity warning emitted on every model load — not an error. - See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]]. ### Profile isolation chosen over shared surface |- The ops profile was enabled with mnemosyne on 2026-08-02 |- **Decision:** `profile_isolation: true` was set on ops, giving each profile its own independent memory bank — zero cross-contamination |- `data/shared/mnemosyne.db` exists with 0 rows; cross-profile sharing is intentionally unused |- This means Tusk (ops) and Rhino (default) do not share memories, which is the desired behaviour |- Config in `~/.hermes/profiles/ops/config.yaml`: `memory.mnemosyne.profile_isolation: true`, `allow_tool_override: false`, `shared_surface_read: false` |- Verified end-to-end on 2026-08-02: memory writes land correctly in both profiles independently ### Legacy table not migrated - 2,181 rows in the old `memory` table are not in the new `working_memory`/`episodic_memory` system - These are only accessible via the legacy `memory` tool, not via `mnemosyne_recall` ### Gateway notifications were captured as user memories (fixed 2026-09-30) - Hermes's background-process notifications (long-running `terminal` jobs reporting completion) were being written into Mnemosyne as **`[USER] [Anthony Martin]`** memories — i.e. machine messages were stored as if the user had said them, and then surfaced during recall as if they were his words. - **36 such rows** were found and purged on 2026-09-30 (working tier plus the corresponding episodic rows). Recall after the purge returned clean. - Two **canonical facts** were also wrong because of the same class of drift: `hermes_cron_model_default` named the removed provider `custom:omni` instead of `custom:omni-101`, and the Daily Marketing Job Radar's model fact was stale. Both corrected. - **Rule:** treat a recalled memory attributed to a person as untrusted until the attribution makes sense in context. Machine-generated status text should never enter the person-attributed tier. ## Known Management Jobs - Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent - Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent - Mnemosyne Health Monitor every 12h — **script**: `mnemosyne-health-monitor.sh`, mode: **no-agent** since the 2026-11-08 conversion (job `ea79a2107e36`), reports only on exception. ⚠ **The no-agent conversion was verified end-to-end on 2026-11-08:** the 18:00 slot (agent mode) ran **12:00:51 → 18:00:42 = 5 h 59 m 51 s**, and the converted job's first slot ran **2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, **silent** — `output/ea79a2107e36/2026-11-09_00-02-00.md` is 159 bytes against 105,982 B for the last agent-mode file. Live read at 2026-11-09 00:0x: **working 1030/1030 = 100.0 %, episodic 1192/1192 = 100.0 %** (so the `< 95 %` alert is correctly quiet), 1043 embeddings generated in 24 h. The script compares with `awk`, not `bc` — **`bc` is not installed on this host** (`mnemosyne-health-monitor.sh:47`), which is why the old monitor's arithmetic silently defaulted instead of failing loudly. ⚠ **2026-10-30: the monitor was twice flagged and neither flag survived contact with `finished_at`.** Its 00:01:08 slot was auto-paused at 01:04:22 on an *"execution unverified / gateway connection failed"* admission ([[incidents/2026-10-30-mnemosyne-health-monitor-paused]]) — but the slot was **still running** and closed **`completed` at 10:02:48** after **10 h 01 m 40 s**, delivering normally and auto-closing incident `ea79a2_d93c3acdc4f2`; the job is **enabled**, `last_status: ok`, streak 0. Its **12:01 slot was then lost to the CT460 host outage** (5 h 30 m dark, [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]) and fired in the 16:37:21 catch-up. **Check `finished_at` before believing a long-running slot failed.** - Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95% - Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent - ⚠ **Blast-radius note (2026-10-16).** Both monitors above (`ea79a2107e36` Health every 12 h, `b63ed5817212` Embedding Coverage every 6 h) are pinned to **`custom:omni-101`** — the same node the memory system's own reporting depends on. During the **2026-10-14 → 10-15 `omniroute-1` outage both failed on every slot**, so **no coverage figures were refreshed for ~34 hours**; the last explicit read remains the **2026-10-12 12:02** run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %), and the **2026-10-16 00:09:48** Health Monitor run is the first clean read since. The two script-only jobs (`consolidate_mnemosyne.sh`, `mnemosyne-backup.sh`) were **unaffected** — they make no model call. Monitoring that shares a blast radius with its subject is degraded monitoring; a coverage alert that cannot fire during a router outage is exactly the case this has to cover. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] ## History and Evidence - **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]]. - **2026-09-30 memory-hygiene pass:** 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (`hermes_cron_model_default` → `custom:omni-101`, job-radar model). No unresolved conflicts left in the working tier. - **2026-08-19 Mnemosyne Watchdog incident:** Watchdog job (`96eb95ce12f2`) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See [[systems/scheduled-tasks]] Known Issues section. - **2026-08-18 incident:** Working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See [[incidents/2026-08-18-mnemosyne-capture-pipeline]]. - **2026-08-01 incident:** Automatic conversational capture was broken by stringified structured YAML, especially `sync_roles: "['user']"`. Fixed by rewriting with native YAML types. See [[incidents/2026-08-01-mnemosyne-capture-failure]]. - **2026-08-01 cleanup:** Legacy always-injected memory reduced from 8,013 bytes to 4,998 bytes (memory) and 2,958 bytes (user). - **2026-07-29 incident:** `memory.mnemosyne.tools: None` caused schema loading warnings. Fixed by removing the line. See [[incidents/2026-07-29-mnemosyne-relay]]. - **2026-07-29 relay scope corruption:** `RuntimeError: scope handle is not at the top of the stack` — 3 occurrences, self-resolved. ## Related - [[concepts/agent-memory]] - [[comparisons/memory-backends]] - [[incidents/2026-08-01-mnemosyne-capture-failure]] - [[incidents/2026-08-18-mnemosyne-capture-pipeline]] - [[incidents/2026-07-29-mnemosyne-relay]] --- ## Re-verified 2026-10-10 (audit window; live probes 2026-10-11 00:14-00:20 AWST) - **Gateway load path repaired.** The gateway's `activate_dependencies()` selects one committed uv env (`795020dbf24343cd9c52df24723dca83`); it was missing `mnemosyne` / `mnemosyne_hermes` and the native dependencies. Repaired 2026-10-10 21:26-21:34 and confirmed live from 22:01:19 (`registered (40 tools)` -> `activated`). Detail: [[incidents/2026-09-28-hermes-managed-runtime-deps]]. - **"Capture silent since 10-09 15:20" was a false alarm.** It resolved to *no interactive sessions in the window* - the window was cron-only traffic. What the surrounding hours did capture was dominated by gateway background-process notifications stored as `[USER] [Anthony Martin] [IMPORTANT: Background process ...]` rows: recall-visible, low signal. A capture filter would help; **no fix applied** (carried open item). - **Coverage script executable again:** `~/.hermes/scripts/check_embedding_coverage.py` now carries mode `0755`, so the monitor's non-executable finding is resolved. - **Open, unverified:** 13 orphan rows in `memory_embeddings` (oldest 2026-08-22) reported by a 2026-10-10 monitor run - neither confirmed nor cleaned by this audit. ## Re-verified 2026-10-18 (audit window; live probes 2026-10-18 00:30-00:36 AWST) - **Both monitors rode the `omniroute-1` 503 window; both are green again.** `ea79a2107e36` (health, every 12 h) failed **10-17 00:00** with `RuntimeError: HTTP 503 ... resource pressure` and delivered its alert; `b63ed5817212` (coverage, every 6 h) failed **10-16 12:03** the same way. Since **10-17 12:00** every slot of both has completed with `delivery_outcome: suppressed` (nothing to report) — including **10-18 00:00**. - **No fresh coverage figures exist in the ledger.** These jobs report to Telegram, not to the execution row, so the **last explicit read remains 2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056 (99.4 %)**, both above the 95 % reindex threshold. Nothing here indicates a reindex is due. - **The watchdog triple-checks the same thing.** `96eb95ce12f2` (every 15 min) exited 1 four times — 10-16 12:15 + 12:30, 10-17 00:15 + 00:30 — each row quoting the **coverage monitor's own 503 error**, i.e. one sentinel producing four failure rows. Read failures **per job**, not per row. It has been clean on every slot since (latest 10-18 00:30). ## Re-verified 2026-10-24 (audit window 2026-10-23; live probes 2026-10-24 00:1x–00:3x AWST) - **Cold start is the new failure vector for consumers: 10–45 s.** The Conduit voice plugin (`conduit_push`) gave one request — provider start included — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`). Mnemosyne's cold start for the `conduit-voice` session regularly exceeds that, so **three calls were abandoned by the memory layer** (10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16: `Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`), and the caller got an agent with no recall. Fixed in the plugin's working tree on 10-23: default **60 s**, overridable with `CONDUIT_MEMORY_TIMEOUT_S`, plus a `TokenError` guard that logs and proceeds without recall instead of aborting. Dashboard restarted 18:54:43; no abort line exists after it. → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]] - **Rule for other consumers:** a 10 s memory budget is below the observed cold-start floor. Any new surface that reads Mnemosyne on a first request needs ≥60 s, or must degrade to "no recall" rather than fail. The **underlying cold start was not optimised** — only the budget was raised. - **Capture and coverage unchanged** from the 10-18 read (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %; nothing indicates a reindex is due). No monitor reds in the window.