Files
agent-estate-wiki/systems/mnemosyne.md
T
Tony0410 a0b3734c95 stocktake(2026-11-01): fix the Quartz build (YAML title), close the "unfixed memory ranking" incident, and correct live drift in tailscale/docker/proxmox
BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic.
Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an
unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a
mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)".
A scan of all 117 pages found exactly one such line; it is now properly quoted.
Quartz now builds cleanly: 690 files emitted and deployed.

RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page).
The page claimed the recall-ranking fix "was not present in live config" and that
conflict 152 was "the only open conflict". Both claims were false:
- The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5,
  fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 —
  not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the
  2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks.
- The original regression test now passes: "mnemosyne recall 'Are you up to date'"
  returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale
  Uber Eats episode no longer outranks anything.
- Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise.
All three open items closed; page moved to status: resolved.

LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01):
- infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping
  times out); m93p peer record reads false for the 4th consecutive time while the
  host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages
  refreshed; no IP changes, no missing peers.
- infrastructure/docker-services.md: CT201's Tailscale IP was wrong
  (100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale
  worldmonitor-seeder-fast entry removed from the CT200 list, which is now
  regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to
  thirsty_payne.
- infrastructure/proxmox.md: storage percentages refreshed on both hosts from live
  pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an
  elevated load average recorded.
- systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md:
  re-probed and verified_on bumped.
- systems/skills-index.md: regenerated — 301 unique skills in 68 categories,
  46 archived. (301 is the unique (category, skill_name) count; find returns 302
  files due to the known nested obsidian-git-sync alias duplicate.)

Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
2026-11-01 04:48:30 +08:00

187 lines
18 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Mnemosyne
type: system
status: active
created: 2026-07-22
updated: 2026-11-01
verified_on: 2026-11-01
review_after: 2026-11-07
confidence: high
tags: [memory, mnemosyne, backend]
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
---
# Mnemosyne
## Purpose
Primary agent memory store for Hermes.
## Current Configuration
- **Provider:** `mnemosyne`
- **Enabled:** true
- **Auto sleep:** true
- **Default scope:** `session`
- **Memory char limit:** 5000
- **User char limit:** 3000
- **Vector type:** int8
- **Profile isolation:** true
- **Shared surface read:** true
- **Shared surface path:** `data/shared/mnemosyne.db`
- **Skip contexts:** cron, flush, subagent, background, skill_loop
- **Reflect max calls per session:** 3
- **Sync roles:** user
- **Flush every:** 6 turns
- **Embedding model:** `BAAI/bge-small-en-v1.5` (384 dims)
- **Embeddings via API:** false (local fastembed)
## Database Stats (live, 2026-10-04)
| Table | Rows | Notes |
|---|---|---|
| `working_memory` | 773 | Active memories |
| `episodic_memory` | 1,047 | Consolidated summaries |
| `memory_embeddings` | 760 | Working-tier embeddings |
| `memory` (legacy) | 2,181 | Old table, not migrated |
| `vec_working` / `vec_working_rowids` | 760 | Working vector index |
| `vec_episodes` / `vec_episodes_rowids` | 1,035 | Episodic vector index |
| `vec_facts_rowids` | 0 | `memoria_facts` is intentionally left unvectorised |
| Shared surface | 0 | Empty — cross-agent sharing not functional |
**Embedding coverage (live, 2026-10-04):** 1,759/2,194 memories embedded (80.2%) — working **726/752 (96.5%)**, episodic **1,033/1,039 (99.4%)**; 13 embeddings generated in the previous 24h. Working coverage is above the 95% alert threshold enforced by the Embedding Coverage job (`b63ed5817212`, every 6h, Telegram topic 102).
## Known Issues
**Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below.
**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. ~~Ranking relevance is the live issue, not coverage~~ — see [[incidents/2026-10-09-memory-recall-relevance]].
**Tier counts (live, 2026-11-01 03:0x — the 2026-10-09 ranking incident is now RESOLVED):** working `941`, episodic `1,124`, `canonical_facts` `90`, `conflicts` `175` (**11 open** — ids **165–175**, all `contradiction`, all created in one batch at 2026-10-31 19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature), working embeddings `927/941 (98.5%)`, episodic embeddings `1,124/1,124 (100%)` — both tiers above the 95% floor, no `reindex` needed. **The recall weights are live in `~/.hermes/mnemosyne/config.yaml`** (`vec_weight: 0.5`, `fts_weight: 0.3`, `importance_weight: 0.3`, plus `MNEMOSYNE_IMPORTANCE_WEIGHT=0.3` in the environment) — **not** in `~/.hermes/config.yaml`'s `memory.mnemosyne` block, which is why the 2026-10-10 probe wrongly concluded the fix was never applied. The original regression query now returns only substantive current-state facts; the stale Uber Eats episode no longer outranks them → [[incidents/2026-10-09-memory-recall-relevance]] (**resolved**).
**Tier counts (live, 2026-10-12 12:02 — health job `ea79a2107e36`, green):** working **760/786 embedded (96.7%)**, episodic **1050/1056 (99.4%)** — both above the 95% floor, no `reindex` needed (`b63ed5817212` continues to backfill). Orphan embeddings **0**. FTS5 exactly aligned: `fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876. **0 unconsolidated** working rows; **1 unresolved conflict** (threshold 10); last consolidation 0.3 h earlier; 2,252 memories total (786 working / 1,056 episodic / 1 triple); disk 15 GB free (81% used). Interpretation: coverage is *not* the live issue — ranking relevance is (see [[incidents/2026-10-09-memory-recall-relevance]]).
**Tier counts (live, 2026-10-12 00:1x):** working `786`, episodic `1,056`, `facts` `876` (`fts_facts` 876 — FTS aligned), `fts_working` 786, `memoria_facts` `410`, `conflicts` `152`. Embeddings: working `773/786 (98.3%)`, episodic `1,052/1,056 (99.6%)` — both above the 95% reindex threshold; the coverage job `b63ed5817212` backfilled 15 vectors since the 2026-10-11 00:08 check.
**Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST.
### Vector embedding coverage gap — RESOLVED (2026-08-02)
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
- **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only.
- Standard repair if coverage drops below 95% working: `~/.hermes/hermes-agent/venv/bin/mnemosyne reindex --yes` (the Embedding Coverage job `b63ed5817212` triggers the same path every 6h).
### Vector recall silently ran FTS-only (2026-10-03 — repaired)
- `fastembed` was absent from **every** uv-managed runtime, so the dense leg of `mnemosyne_recall` produced no candidates and recall fell back to FTS5/keyword scoring **without raising an error**. Sessions kept working, which is why it went unnoticed.
- Compounded by `FASTEMBED_CACHE_PATH` being unset in `~/.hermes/.env`, and by the embedding worker sometimes launching in isolated (`python -I`) mode, which does not inherit `PYTHONPATH`.
- **Repaired 2026-10-03:** `fastembed` installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min alongside its existing `dotenv`/`mnemosyne` checks.
- **CLI caveat — do not misread `mnemosyne doctor`.** Doctor and the recall CLI run under the legacy `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable; doctor therefore prints `vec0: present_but_unloadable` and marks the vector tiers `unavailable` even while the gateway embeds normally. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line.
- Benign noise: `pthread_setaffinity_np failed … mask: {0,}` from onnxruntime in `mcp-stderr.log` is a container affinity warning emitted on every model load — not an error.
- See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
### Profile isolation chosen over shared surface
|- The ops profile was enabled with mnemosyne on 2026-08-02
|- **Decision:** `profile_isolation: true` was set on ops, giving each profile its own independent memory bank — zero cross-contamination
|- `data/shared/mnemosyne.db` exists with 0 rows; cross-profile sharing is intentionally unused
|- This means Tusk (ops) and Rhino (default) do not share memories, which is the desired behaviour
|- Config in `~/.hermes/profiles/ops/config.yaml`: `memory.mnemosyne.profile_isolation: true`, `allow_tool_override: false`, `shared_surface_read: false`
|- Verified end-to-end on 2026-08-02: memory writes land correctly in both profiles independently
### Legacy table not migrated
- 2,181 rows in the old `memory` table are not in the new `working_memory`/`episodic_memory` system
- These are only accessible via the legacy `memory` tool, not via `mnemosyne_recall`
### Gateway notifications were captured as user memories (fixed 2026-09-30)
- Hermes's background-process notifications (long-running `terminal` jobs reporting completion) were being written into Mnemosyne as **`[USER] [Anthony Martin]`** memories — i.e. machine messages were stored as if the user had said them, and then surfaced during recall as if they were his words.
- **36 such rows** were found and purged on 2026-09-30 (working tier plus the corresponding episodic rows). Recall after the purge returned clean.
- Two **canonical facts** were also wrong because of the same class of drift: `hermes_cron_model_default` named the removed provider `custom:omni` instead of `custom:omni-101`, and the Daily Marketing Job Radar's model fact was stale. Both corrected.
- **Rule:** treat a recalled memory attributed to a person as untrusted until the attribution makes sense in context. Machine-generated status text should never enter the person-attributed tier.
## Known Management Jobs
- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception.
⚠ **2026-10-30: the monitor was twice flagged and neither flag survived contact with `finished_at`.** Its
00:01:08 slot was auto-paused at 01:04:22 on an *"execution unverified / gateway connection failed"* admission
([[incidents/2026-10-30-mnemosyne-health-monitor-paused]]) — but the slot was **still running** and closed
**`completed` at 10:02:48** after **10 h 01 m 40 s**, delivering normally and auto-closing incident
`ea79a2_d93c3acdc4f2`; the job is **enabled**, `last_status: ok`, streak 0. Its **12:01 slot was then lost to
the CT460 host outage** (5 h 30 m dark, [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]) and fired in
the 16:37:21 catch-up. **Check `finished_at` before believing a long-running slot failed.**
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
- ⚠ **Blast-radius note (2026-10-16).** Both monitors above (`ea79a2107e36` Health every 12 h,
`b63ed5817212` Embedding Coverage every 6 h) are pinned to **`custom:omni-101`** — the same node the
memory system's own reporting depends on. During the **2026-10-14 → 10-15 `omniroute-1` outage both
failed on every slot**, so **no coverage figures were refreshed for ~34 hours**; the last explicit read
remains the **2026-10-12 12:02** run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %), and the
**2026-10-16 00:09:48** Health Monitor run is the first clean read since. The two script-only jobs
(`consolidate_mnemosyne.sh`, `mnemosyne-backup.sh`) were **unaffected** — they make no model call.
Monitoring that shares a blast radius with its subject is degraded monitoring; a coverage alert that
cannot fire during a router outage is exactly the case this has to cover.
→ [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
## History and Evidence
- **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
- **2026-09-30 memory-hygiene pass:** 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (`hermes_cron_model_default` → `custom:omni-101`, job-radar model). No unresolved conflicts left in the working tier.
- **2026-08-19 Mnemosyne Watchdog incident:** Watchdog job (`96eb95ce12f2`) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See [[systems/scheduled-tasks]] Known Issues section.
- **2026-08-18 incident:** Working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See [[incidents/2026-08-18-mnemosyne-capture-pipeline]].
- **2026-08-01 incident:** Automatic conversational capture was broken by stringified structured YAML, especially `sync_roles: "['user']"`. Fixed by rewriting with native YAML types. See [[incidents/2026-08-01-mnemosyne-capture-failure]].
- **2026-08-01 cleanup:** Legacy always-injected memory reduced from 8,013 bytes to 4,998 bytes (memory) and 2,958 bytes (user).
- **2026-07-29 incident:** `memory.mnemosyne.tools: None` caused schema loading warnings. Fixed by removing the line. See [[incidents/2026-07-29-mnemosyne-relay]].
- **2026-07-29 relay scope corruption:** `RuntimeError: scope handle is not at the top of the stack` — 3 occurrences, self-resolved.
## Related
- [[concepts/agent-memory]]
- [[comparisons/memory-backends]]
- [[incidents/2026-08-01-mnemosyne-capture-failure]]
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]]
- [[incidents/2026-07-29-mnemosyne-relay]]
---
## Re-verified 2026-10-10 (audit window; live probes 2026-10-11 00:14-00:20 AWST)
- **Gateway load path repaired.** The gateway's `activate_dependencies()` selects one committed uv
env (`795020dbf24343cd9c52df24723dca83`); it was missing `mnemosyne` / `mnemosyne_hermes` and the
native dependencies. Repaired 2026-10-10 21:26-21:34 and confirmed live from 22:01:19
(`registered (40 tools)` -> `activated`). Detail:
[[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **"Capture silent since 10-09 15:20" was a false alarm.** It resolved to *no interactive sessions
in the window* - the window was cron-only traffic. What the surrounding hours did capture was
dominated by gateway background-process notifications stored as
`[USER] [Anthony Martin] [IMPORTANT: Background process ...]` rows: recall-visible, low signal.
A capture filter would help; **no fix applied** (carried open item).
- **Coverage script executable again:** `~/.hermes/scripts/check_embedding_coverage.py` now carries
mode `0755`, so the monitor's non-executable finding is resolved.
- **Open, unverified:** 13 orphan rows in `memory_embeddings` (oldest 2026-08-22) reported by a
2026-10-10 monitor run - neither confirmed nor cleaned by this audit.
## Re-verified 2026-10-18 (audit window; live probes 2026-10-18 00:30-00:36 AWST)
- **Both monitors rode the `omniroute-1` 503 window; both are green again.** `ea79a2107e36` (health, every
12 h) failed **10-17 00:00** with `RuntimeError: HTTP 503 ... resource pressure` and delivered its alert;
`b63ed5817212` (coverage, every 6 h) failed **10-16 12:03** the same way. Since **10-17 12:00** every slot
of both has completed with `delivery_outcome: suppressed` (nothing to report) — including **10-18 00:00**.
- **No fresh coverage figures exist in the ledger.** These jobs report to Telegram, not to the execution
row, so the **last explicit read remains 2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056
(99.4 %)**, both above the 95 % reindex threshold. Nothing here indicates a reindex is due.
- **The watchdog triple-checks the same thing.** `96eb95ce12f2` (every 15 min) exited 1 four times —
10-16 12:15 + 12:30, 10-17 00:15 + 00:30 — each row quoting the **coverage monitor's own 503 error**, i.e.
one sentinel producing four failure rows. Read failures **per job**, not per row. It has been clean on
every slot since (latest 10-18 00:30).
## Re-verified 2026-10-24 (audit window 2026-10-23; live probes 2026-10-24 00:1x–00:3x AWST)
- **Cold start is the new failure vector for consumers: 10–45 s.** The Conduit voice plugin
(`conduit_push`) gave one request — provider start included — a hard-coded **10 s** budget
(`MEMORY_TIMEOUT_S`). Mnemosyne's cold start for the `conduit-voice` session regularly exceeds
that, so **three calls were abandoned by the memory layer** (10-22 11:28:17, 10-22 13:16:37,
10-23 18:29:16: `Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`),
and the caller got an agent with no recall. Fixed in the plugin's working tree on 10-23: default
**60 s**, overridable with `CONDUIT_MEMORY_TIMEOUT_S`, plus a `TokenError` guard that logs and
proceeds without recall instead of aborting. Dashboard restarted 18:54:43; no abort line exists
after it. → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]
- **Rule for other consumers:** a 10 s memory budget is below the observed cold-start floor. Any
new surface that reads Mnemosyne on a first request needs ≥60 s, or must degrade to "no recall"
rather than fail. The **underlying cold start was not optimised** — only the budget was raised.
- **Capture and coverage unchanged** from the 10-18 read (working 760/786 = 96.7 %, episodic
1050/1056 = 99.4 %; nothing indicates a reindex is due). No monitor reds in the window.