BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic. Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)". A scan of all 117 pages found exactly one such line; it is now properly quoted. Quartz now builds cleanly: 690 files emitted and deployed. RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page). The page claimed the recall-ranking fix "was not present in live config" and that conflict 152 was "the only open conflict". Both claims were false: - The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5, fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 — not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the 2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks. - The original regression test now passes: "mnemosyne recall 'Are you up to date'" returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale Uber Eats episode no longer outranks anything. - Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise. All three open items closed; page moved to status: resolved. LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01): - infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping times out); m93p peer record reads false for the 4th consecutive time while the host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages refreshed; no IP changes, no missing peers. - infrastructure/docker-services.md: CT201's Tailscale IP was wrong (100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale worldmonitor-seeder-fast entry removed from the CT200 list, which is now regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to thirsty_payne. - infrastructure/proxmox.md: storage percentages refreshed on both hosts from live pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an elevated load average recorded. - systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md: re-probed and verified_on bumped. - systems/skills-index.md: regenerated — 301 unique skills in 68 categories, 46 archived. (301 is the unique (category, skill_name) count; find returns 302 files due to the known nested obsidian-git-sync alias duplicate.) Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
187 lines
18 KiB
Markdown
187 lines
18 KiB
Markdown
---
|
||
title: Mnemosyne
|
||
type: system
|
||
status: active
|
||
created: 2026-07-22
|
||
updated: 2026-11-01
|
||
verified_on: 2026-11-01
|
||
review_after: 2026-11-07
|
||
confidence: high
|
||
tags: [memory, mnemosyne, backend]
|
||
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
||
---
|
||
|
||
# Mnemosyne
|
||
|
||
## Purpose
|
||
Primary agent memory store for Hermes.
|
||
|
||
## Current Configuration
|
||
- **Provider:** `mnemosyne`
|
||
- **Enabled:** true
|
||
- **Auto sleep:** true
|
||
- **Default scope:** `session`
|
||
- **Memory char limit:** 5000
|
||
- **User char limit:** 3000
|
||
- **Vector type:** int8
|
||
- **Profile isolation:** true
|
||
- **Shared surface read:** true
|
||
- **Shared surface path:** `data/shared/mnemosyne.db`
|
||
- **Skip contexts:** cron, flush, subagent, background, skill_loop
|
||
- **Reflect max calls per session:** 3
|
||
- **Sync roles:** user
|
||
- **Flush every:** 6 turns
|
||
- **Embedding model:** `BAAI/bge-small-en-v1.5` (384 dims)
|
||
- **Embeddings via API:** false (local fastembed)
|
||
|
||
## Database Stats (live, 2026-10-04)
|
||
| Table | Rows | Notes |
|
||
|---|---|---|
|
||
| `working_memory` | 773 | Active memories |
|
||
| `episodic_memory` | 1,047 | Consolidated summaries |
|
||
| `memory_embeddings` | 760 | Working-tier embeddings |
|
||
| `memory` (legacy) | 2,181 | Old table, not migrated |
|
||
| `vec_working` / `vec_working_rowids` | 760 | Working vector index |
|
||
| `vec_episodes` / `vec_episodes_rowids` | 1,035 | Episodic vector index |
|
||
| `vec_facts_rowids` | 0 | `memoria_facts` is intentionally left unvectorised |
|
||
| Shared surface | 0 | Empty — cross-agent sharing not functional |
|
||
|
||
**Embedding coverage (live, 2026-10-04):** 1,759/2,194 memories embedded (80.2%) — working **726/752 (96.5%)**, episodic **1,033/1,039 (99.4%)**; 13 embeddings generated in the previous 24h. Working coverage is above the 95% alert threshold enforced by the Embedding Coverage job (`b63ed5817212`, every 6h, Telegram topic 102).
|
||
|
||
## Known Issues
|
||
|
||
**Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below.
|
||
|
||
**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. ~~Ranking relevance is the live issue, not coverage~~ — see [[incidents/2026-10-09-memory-recall-relevance]].
|
||
|
||
**Tier counts (live, 2026-11-01 03:0x — the 2026-10-09 ranking incident is now RESOLVED):** working `941`, episodic `1,124`, `canonical_facts` `90`, `conflicts` `175` (**11 open** — ids **165–175**, all `contradiction`, all created in one batch at 2026-10-31 19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature), working embeddings `927/941 (98.5%)`, episodic embeddings `1,124/1,124 (100%)` — both tiers above the 95% floor, no `reindex` needed. **The recall weights are live in `~/.hermes/mnemosyne/config.yaml`** (`vec_weight: 0.5`, `fts_weight: 0.3`, `importance_weight: 0.3`, plus `MNEMOSYNE_IMPORTANCE_WEIGHT=0.3` in the environment) — **not** in `~/.hermes/config.yaml`'s `memory.mnemosyne` block, which is why the 2026-10-10 probe wrongly concluded the fix was never applied. The original regression query now returns only substantive current-state facts; the stale Uber Eats episode no longer outranks them → [[incidents/2026-10-09-memory-recall-relevance]] (**resolved**).
|
||
|
||
**Tier counts (live, 2026-10-12 12:02 — health job `ea79a2107e36`, green):** working **760/786 embedded (96.7%)**, episodic **1050/1056 (99.4%)** — both above the 95% floor, no `reindex` needed (`b63ed5817212` continues to backfill). Orphan embeddings **0**. FTS5 exactly aligned: `fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876. **0 unconsolidated** working rows; **1 unresolved conflict** (threshold 10); last consolidation 0.3 h earlier; 2,252 memories total (786 working / 1,056 episodic / 1 triple); disk 15 GB free (81% used). Interpretation: coverage is *not* the live issue — ranking relevance is (see [[incidents/2026-10-09-memory-recall-relevance]]).
|
||
|
||
**Tier counts (live, 2026-10-12 00:1x):** working `786`, episodic `1,056`, `facts` `876` (`fts_facts` 876 — FTS aligned), `fts_working` 786, `memoria_facts` `410`, `conflicts` `152`. Embeddings: working `773/786 (98.3%)`, episodic `1,052/1,056 (99.6%)` — both above the 95% reindex threshold; the coverage job `b63ed5817212` backfilled 15 vectors since the 2026-10-11 00:08 check.
|
||
|
||
**Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST.
|
||
|
||
### Vector embedding coverage gap — RESOLVED (2026-08-02)
|
||
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
|
||
- **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only.
|
||
- Standard repair if coverage drops below 95% working: `~/.hermes/hermes-agent/venv/bin/mnemosyne reindex --yes` (the Embedding Coverage job `b63ed5817212` triggers the same path every 6h).
|
||
|
||
### Vector recall silently ran FTS-only (2026-10-03 — repaired)
|
||
- `fastembed` was absent from **every** uv-managed runtime, so the dense leg of `mnemosyne_recall` produced no candidates and recall fell back to FTS5/keyword scoring **without raising an error**. Sessions kept working, which is why it went unnoticed.
|
||
- Compounded by `FASTEMBED_CACHE_PATH` being unset in `~/.hermes/.env`, and by the embedding worker sometimes launching in isolated (`python -I`) mode, which does not inherit `PYTHONPATH`.
|
||
- **Repaired 2026-10-03:** `fastembed` installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min alongside its existing `dotenv`/`mnemosyne` checks.
|
||
- **CLI caveat — do not misread `mnemosyne doctor`.** Doctor and the recall CLI run under the legacy `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable; doctor therefore prints `vec0: present_but_unloadable` and marks the vector tiers `unavailable` even while the gateway embeds normally. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line.
|
||
- Benign noise: `pthread_setaffinity_np failed … mask: {0,}` from onnxruntime in `mcp-stderr.log` is a container affinity warning emitted on every model load — not an error.
|
||
- See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||
|
||
### Profile isolation chosen over shared surface
|
||
|- The ops profile was enabled with mnemosyne on 2026-08-02
|
||
|- **Decision:** `profile_isolation: true` was set on ops, giving each profile its own independent memory bank — zero cross-contamination
|
||
|- `data/shared/mnemosyne.db` exists with 0 rows; cross-profile sharing is intentionally unused
|
||
|- This means Tusk (ops) and Rhino (default) do not share memories, which is the desired behaviour
|
||
|- Config in `~/.hermes/profiles/ops/config.yaml`: `memory.mnemosyne.profile_isolation: true`, `allow_tool_override: false`, `shared_surface_read: false`
|
||
|- Verified end-to-end on 2026-08-02: memory writes land correctly in both profiles independently
|
||
|
||
### Legacy table not migrated
|
||
- 2,181 rows in the old `memory` table are not in the new `working_memory`/`episodic_memory` system
|
||
- These are only accessible via the legacy `memory` tool, not via `mnemosyne_recall`
|
||
|
||
### Gateway notifications were captured as user memories (fixed 2026-09-30)
|
||
- Hermes's background-process notifications (long-running `terminal` jobs reporting completion) were being written into Mnemosyne as **`[USER] [Anthony Martin]`** memories — i.e. machine messages were stored as if the user had said them, and then surfaced during recall as if they were his words.
|
||
- **36 such rows** were found and purged on 2026-09-30 (working tier plus the corresponding episodic rows). Recall after the purge returned clean.
|
||
- Two **canonical facts** were also wrong because of the same class of drift: `hermes_cron_model_default` named the removed provider `custom:omni` instead of `custom:omni-101`, and the Daily Marketing Job Radar's model fact was stale. Both corrected.
|
||
- **Rule:** treat a recalled memory attributed to a person as untrusted until the attribution makes sense in context. Machine-generated status text should never enter the person-attributed tier.
|
||
|
||
## Known Management Jobs
|
||
- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
|
||
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
|
||
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception.
|
||
⚠ **2026-10-30: the monitor was twice flagged and neither flag survived contact with `finished_at`.** Its
|
||
00:01:08 slot was auto-paused at 01:04:22 on an *"execution unverified / gateway connection failed"* admission
|
||
([[incidents/2026-10-30-mnemosyne-health-monitor-paused]]) — but the slot was **still running** and closed
|
||
**`completed` at 10:02:48** after **10 h 01 m 40 s**, delivering normally and auto-closing incident
|
||
`ea79a2_d93c3acdc4f2`; the job is **enabled**, `last_status: ok`, streak 0. Its **12:01 slot was then lost to
|
||
the CT460 host outage** (5 h 30 m dark, [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]) and fired in
|
||
the 16:37:21 catch-up. **Check `finished_at` before believing a long-running slot failed.**
|
||
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
|
||
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
|
||
- ⚠ **Blast-radius note (2026-10-16).** Both monitors above (`ea79a2107e36` Health every 12 h,
|
||
`b63ed5817212` Embedding Coverage every 6 h) are pinned to **`custom:omni-101`** — the same node the
|
||
memory system's own reporting depends on. During the **2026-10-14 → 10-15 `omniroute-1` outage both
|
||
failed on every slot**, so **no coverage figures were refreshed for ~34 hours**; the last explicit read
|
||
remains the **2026-10-12 12:02** run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %), and the
|
||
**2026-10-16 00:09:48** Health Monitor run is the first clean read since. The two script-only jobs
|
||
(`consolidate_mnemosyne.sh`, `mnemosyne-backup.sh`) were **unaffected** — they make no model call.
|
||
Monitoring that shares a blast radius with its subject is degraded monitoring; a coverage alert that
|
||
cannot fire during a router outage is exactly the case this has to cover.
|
||
→ [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
|
||
|
||
## History and Evidence
|
||
- **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||
- **2026-09-30 memory-hygiene pass:** 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (`hermes_cron_model_default` → `custom:omni-101`, job-radar model). No unresolved conflicts left in the working tier.
|
||
- **2026-08-19 Mnemosyne Watchdog incident:** Watchdog job (`96eb95ce12f2`) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See [[systems/scheduled-tasks]] Known Issues section.
|
||
- **2026-08-18 incident:** Working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See [[incidents/2026-08-18-mnemosyne-capture-pipeline]].
|
||
- **2026-08-01 incident:** Automatic conversational capture was broken by stringified structured YAML, especially `sync_roles: "['user']"`. Fixed by rewriting with native YAML types. See [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
||
- **2026-08-01 cleanup:** Legacy always-injected memory reduced from 8,013 bytes to 4,998 bytes (memory) and 2,958 bytes (user).
|
||
- **2026-07-29 incident:** `memory.mnemosyne.tools: None` caused schema loading warnings. Fixed by removing the line. See [[incidents/2026-07-29-mnemosyne-relay]].
|
||
- **2026-07-29 relay scope corruption:** `RuntimeError: scope handle is not at the top of the stack` — 3 occurrences, self-resolved.
|
||
|
||
## Related
|
||
- [[concepts/agent-memory]]
|
||
- [[comparisons/memory-backends]]
|
||
- [[incidents/2026-08-01-mnemosyne-capture-failure]]
|
||
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]]
|
||
- [[incidents/2026-07-29-mnemosyne-relay]]
|
||
|
||
|
||
---
|
||
|
||
## Re-verified 2026-10-10 (audit window; live probes 2026-10-11 00:14-00:20 AWST)
|
||
|
||
- **Gateway load path repaired.** The gateway's `activate_dependencies()` selects one committed uv
|
||
env (`795020dbf24343cd9c52df24723dca83`); it was missing `mnemosyne` / `mnemosyne_hermes` and the
|
||
native dependencies. Repaired 2026-10-10 21:26-21:34 and confirmed live from 22:01:19
|
||
(`registered (40 tools)` -> `activated`). Detail:
|
||
[[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||
- **"Capture silent since 10-09 15:20" was a false alarm.** It resolved to *no interactive sessions
|
||
in the window* - the window was cron-only traffic. What the surrounding hours did capture was
|
||
dominated by gateway background-process notifications stored as
|
||
`[USER] [Anthony Martin] [IMPORTANT: Background process ...]` rows: recall-visible, low signal.
|
||
A capture filter would help; **no fix applied** (carried open item).
|
||
- **Coverage script executable again:** `~/.hermes/scripts/check_embedding_coverage.py` now carries
|
||
mode `0755`, so the monitor's non-executable finding is resolved.
|
||
- **Open, unverified:** 13 orphan rows in `memory_embeddings` (oldest 2026-08-22) reported by a
|
||
2026-10-10 monitor run - neither confirmed nor cleaned by this audit.
|
||
|
||
## Re-verified 2026-10-18 (audit window; live probes 2026-10-18 00:30-00:36 AWST)
|
||
|
||
- **Both monitors rode the `omniroute-1` 503 window; both are green again.** `ea79a2107e36` (health, every
|
||
12 h) failed **10-17 00:00** with `RuntimeError: HTTP 503 ... resource pressure` and delivered its alert;
|
||
`b63ed5817212` (coverage, every 6 h) failed **10-16 12:03** the same way. Since **10-17 12:00** every slot
|
||
of both has completed with `delivery_outcome: suppressed` (nothing to report) — including **10-18 00:00**.
|
||
- **No fresh coverage figures exist in the ledger.** These jobs report to Telegram, not to the execution
|
||
row, so the **last explicit read remains 2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056
|
||
(99.4 %)**, both above the 95 % reindex threshold. Nothing here indicates a reindex is due.
|
||
- **The watchdog triple-checks the same thing.** `96eb95ce12f2` (every 15 min) exited 1 four times —
|
||
10-16 12:15 + 12:30, 10-17 00:15 + 00:30 — each row quoting the **coverage monitor's own 503 error**, i.e.
|
||
one sentinel producing four failure rows. Read failures **per job**, not per row. It has been clean on
|
||
every slot since (latest 10-18 00:30).
|
||
|
||
## Re-verified 2026-10-24 (audit window 2026-10-23; live probes 2026-10-24 00:1x–00:3x AWST)
|
||
|
||
- **Cold start is the new failure vector for consumers: 10–45 s.** The Conduit voice plugin
|
||
(`conduit_push`) gave one request — provider start included — a hard-coded **10 s** budget
|
||
(`MEMORY_TIMEOUT_S`). Mnemosyne's cold start for the `conduit-voice` session regularly exceeds
|
||
that, so **three calls were abandoned by the memory layer** (10-22 11:28:17, 10-22 13:16:37,
|
||
10-23 18:29:16: `Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`),
|
||
and the caller got an agent with no recall. Fixed in the plugin's working tree on 10-23: default
|
||
**60 s**, overridable with `CONDUIT_MEMORY_TIMEOUT_S`, plus a `TokenError` guard that logs and
|
||
proceeds without recall instead of aborting. Dashboard restarted 18:54:43; no abort line exists
|
||
after it. → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]
|
||
- **Rule for other consumers:** a 10 s memory budget is below the observed cold-start floor. Any
|
||
new surface that reads Mnemosyne on a first request needs ≥60 s, or must degrade to "no recall"
|
||
rather than fail. The **underlying cold start was not optimised** — only the budget was raised.
|
||
- **Capture and coverage unchanged** from the 10-18 read (working 760/786 = 96.7 %, episodic
|
||
1050/1056 = 99.4 %; nothing indicates a reindex is due). No monitor reds in the window.
|