Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline. NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450). Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
134 lines
12 KiB
Markdown
134 lines
12 KiB
Markdown
---
|
||
title: Mnemosyne
|
||
type: system
|
||
status: active
|
||
created: 2026-07-22
|
||
updated: 2026-10-13
|
||
verified_on: 2026-10-13
|
||
confidence: high
|
||
tags: [memory, mnemosyne, backend]
|
||
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
||
---
|
||
|
||
# Mnemosyne
|
||
|
||
## Purpose
|
||
Primary agent memory store for Hermes.
|
||
|
||
## Current Configuration
|
||
- **Provider:** `mnemosyne`
|
||
- **Enabled:** true
|
||
- **Auto sleep:** true
|
||
- **Default scope:** `session`
|
||
- **Memory char limit:** 5000
|
||
- **User char limit:** 3000
|
||
- **Vector type:** int8
|
||
- **Profile isolation:** true
|
||
- **Shared surface read:** true
|
||
- **Shared surface path:** `data/shared/mnemosyne.db`
|
||
- **Skip contexts:** cron, flush, subagent, background, skill_loop
|
||
- **Reflect max calls per session:** 3
|
||
- **Sync roles:** user
|
||
- **Flush every:** 6 turns
|
||
- **Embedding model:** `BAAI/bge-small-en-v1.5` (384 dims)
|
||
- **Embeddings via API:** false (local fastembed)
|
||
|
||
## Database Stats (live, 2026-10-04)
|
||
| Table | Rows | Notes |
|
||
|---|---|---|
|
||
| `working_memory` | 773 | Active memories |
|
||
| `episodic_memory` | 1,047 | Consolidated summaries |
|
||
| `memory_embeddings` | 760 | Working-tier embeddings |
|
||
| `memory` (legacy) | 2,181 | Old table, not migrated |
|
||
| `vec_working` / `vec_working_rowids` | 760 | Working vector index |
|
||
| `vec_episodes` / `vec_episodes_rowids` | 1,035 | Episodic vector index |
|
||
| `vec_facts_rowids` | 0 | `memoria_facts` is intentionally left unvectorised |
|
||
| Shared surface | 0 | Empty — cross-agent sharing not functional |
|
||
|
||
**Embedding coverage (live, 2026-10-04):** 1,759/2,194 memories embedded (80.2%) — working **726/752 (96.5%)**, episodic **1,033/1,039 (99.4%)**; 13 embeddings generated in the previous 24h. Working coverage is above the 95% alert threshold enforced by the Embedding Coverage job (`b63ed5817212`, every 6h, Telegram topic 102).
|
||
|
||
## Known Issues
|
||
|
||
**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. Ranking relevance is the live issue, not coverage — see [[incidents/2026-10-09-memory-recall-relevance]].
|
||
|
||
**Tier counts (live, 2026-10-12 12:02 — health job `ea79a2107e36`, green):** working **760/786 embedded (96.7%)**, episodic **1050/1056 (99.4%)** — both above the 95% floor, no `reindex` needed (`b63ed5817212` continues to backfill). Orphan embeddings **0**. FTS5 exactly aligned: `fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876. **0 unconsolidated** working rows; **1 unresolved conflict** (threshold 10); last consolidation 0.3 h earlier; 2,252 memories total (786 working / 1,056 episodic / 1 triple); disk 15 GB free (81% used). Interpretation: coverage is *not* the live issue — ranking relevance is (see [[incidents/2026-10-09-memory-recall-relevance]]).
|
||
|
||
**Tier counts (live, 2026-10-12 00:1x):** working `786`, episodic `1,056`, `facts` `876` (`fts_facts` 876 — FTS aligned), `fts_working` 786, `memoria_facts` `410`, `conflicts` `152`. Embeddings: working `773/786 (98.3%)`, episodic `1,052/1,056 (99.6%)` — both above the 95% reindex threshold; the coverage job `b63ed5817212` backfilled 15 vectors since the 2026-10-11 00:08 check.
|
||
|
||
**Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST.
|
||
|
||
### Vector embedding coverage gap — RESOLVED (2026-08-02)
|
||
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
|
||
- **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only.
|
||
- Standard repair if coverage drops below 95% working: `~/.hermes/hermes-agent/venv/bin/mnemosyne reindex --yes` (the Embedding Coverage job `b63ed5817212` triggers the same path every 6h).
|
||
|
||
### Vector recall silently ran FTS-only (2026-10-03 — repaired)
|
||
- `fastembed` was absent from **every** uv-managed runtime, so the dense leg of `mnemosyne_recall` produced no candidates and recall fell back to FTS5/keyword scoring **without raising an error**. Sessions kept working, which is why it went unnoticed.
|
||
- Compounded by `FASTEMBED_CACHE_PATH` being unset in `~/.hermes/.env`, and by the embedding worker sometimes launching in isolated (`python -I`) mode, which does not inherit `PYTHONPATH`.
|
||
- **Repaired 2026-10-03:** `fastembed` installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min alongside its existing `dotenv`/`mnemosyne` checks.
|
||
- **CLI caveat — do not misread `mnemosyne doctor`.** Doctor and the recall CLI run under the legacy `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable; doctor therefore prints `vec0: present_but_unloadable` and marks the vector tiers `unavailable` even while the gateway embeds normally. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line.
|
||
- Benign noise: `pthread_setaffinity_np failed … mask: {0,}` from onnxruntime in `mcp-stderr.log` is a container affinity warning emitted on every model load — not an error.
|
||
- See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||
|
||
### Profile isolation chosen over shared surface
|
||
|- The ops profile was enabled with mnemosyne on 2026-08-02
|
||
|- **Decision:** `profile_isolation: true` was set on ops, giving each profile its own independent memory bank — zero cross-contamination
|
||
|- `data/shared/mnemosyne.db` exists with 0 rows; cross-profile sharing is intentionally unused
|
||
|- This means Tusk (ops) and Rhino (default) do not share memories, which is the desired behaviour
|
||
|- Config in `~/.hermes/profiles/ops/config.yaml`: `memory.mnemosyne.profile_isolation: true`, `allow_tool_override: false`, `shared_surface_read: false`
|
||
|- Verified end-to-end on 2026-08-02: memory writes land correctly in both profiles independently
|
||
|
||
### Legacy table not migrated
|
||
- 2,181 rows in the old `memory` table are not in the new `working_memory`/`episodic_memory` system
|
||
- These are only accessible via the legacy `memory` tool, not via `mnemosyne_recall`
|
||
|
||
### Gateway notifications were captured as user memories (fixed 2026-09-30)
|
||
- Hermes's background-process notifications (long-running `terminal` jobs reporting completion) were being written into Mnemosyne as **`[USER] [Anthony Martin]`** memories — i.e. machine messages were stored as if the user had said them, and then surfaced during recall as if they were his words.
|
||
- **36 such rows** were found and purged on 2026-09-30 (working tier plus the corresponding episodic rows). Recall after the purge returned clean.
|
||
- Two **canonical facts** were also wrong because of the same class of drift: `hermes_cron_model_default` named the removed provider `custom:omni` instead of `custom:omni-101`, and the Daily Marketing Job Radar's model fact was stale. Both corrected.
|
||
- **Rule:** treat a recalled memory attributed to a person as untrusted until the attribution makes sense in context. Machine-generated status text should never enter the person-attributed tier.
|
||
|
||
## Known Management Jobs
|
||
- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
|
||
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
|
||
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception
|
||
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
|
||
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
|
||
|
||
## History and Evidence
|
||
- **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||
- **2026-09-30 memory-hygiene pass:** 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (`hermes_cron_model_default` → `custom:omni-101`, job-radar model). No unresolved conflicts left in the working tier.
|
||
- **2026-08-19 Mnemosyne Watchdog incident:** Watchdog job (`96eb95ce12f2`) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See [[systems/scheduled-tasks]] Known Issues section.
|
||
- **2026-08-18 incident:** Working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See [[incidents/2026-08-18-mnemosyne-capture-pipeline]].
|
||
- **2026-08-01 incident:** Automatic conversational capture was broken by stringified structured YAML, especially `sync_roles: "['user']"`. Fixed by rewriting with native YAML types. See [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
||
- **2026-08-01 cleanup:** Legacy always-injected memory reduced from 8,013 bytes to 4,998 bytes (memory) and 2,958 bytes (user).
|
||
- **2026-07-29 incident:** `memory.mnemosyne.tools: None` caused schema loading warnings. Fixed by removing the line. See [[incidents/2026-07-29-mnemosyne-relay]].
|
||
- **2026-07-29 relay scope corruption:** `RuntimeError: scope handle is not at the top of the stack` — 3 occurrences, self-resolved.
|
||
|
||
## Related
|
||
- [[concepts/agent-memory]]
|
||
- [[comparisons/memory-backends]]
|
||
- [[incidents/2026-08-01-mnemosyne-capture-failure]]
|
||
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]]
|
||
- [[incidents/2026-07-29-mnemosyne-relay]]
|
||
|
||
|
||
---
|
||
|
||
## Re-verified 2026-10-10 (audit window; live probes 2026-10-11 00:14-00:20 AWST)
|
||
|
||
- **Gateway load path repaired.** The gateway's `activate_dependencies()` selects one committed uv
|
||
env (`795020dbf24343cd9c52df24723dca83`); it was missing `mnemosyne` / `mnemosyne_hermes` and the
|
||
native dependencies. Repaired 2026-10-10 21:26-21:34 and confirmed live from 22:01:19
|
||
(`registered (40 tools)` -> `activated`). Detail:
|
||
[[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||
- **"Capture silent since 10-09 15:20" was a false alarm.** It resolved to *no interactive sessions
|
||
in the window* - the window was cron-only traffic. What the surrounding hours did capture was
|
||
dominated by gateway background-process notifications stored as
|
||
`[USER] [Anthony Martin] [IMPORTANT: Background process ...]` rows: recall-visible, low signal.
|
||
A capture filter would help; **no fix applied** (carried open item).
|
||
- **Coverage script executable again:** `~/.hermes/scripts/check_embedding_coverage.py` now carries
|
||
mode `0755`, so the monitor's non-executable finding is resolved.
|
||
- **Open, unverified:** 13 orphan rows in `memory_embeddings` (oldest 2026-08-22) reported by a
|
||
2026-10-10 monitor run - neither confirmed nor cleaned by this audit.
|