Files
agent-estate-wiki/systems/mnemosyne.md
T
hermes 2061b8607b audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections
omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07
(httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error).
36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog
symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1).

STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable
pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls
completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence.

New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live).
Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute,
systems/mnemosyne (monitor blast-radius note), current-state, index, log.
2026-10-16 00:19:14 +08:00

13 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Mnemosyne system active 2026-07-22 2026-10-16 2026-10-16 high
memory
mnemosyne
backend
live-probe
mnemosyne-db
config
ops-profile-config

Mnemosyne

Purpose

Primary agent memory store for Hermes.

Current Configuration

  • Provider: mnemosyne
  • Enabled: true
  • Auto sleep: true
  • Default scope: session
  • Memory char limit: 5000
  • User char limit: 3000
  • Vector type: int8
  • Profile isolation: true
  • Shared surface read: true
  • Shared surface path: data/shared/mnemosyne.db
  • Skip contexts: cron, flush, subagent, background, skill_loop
  • Reflect max calls per session: 3
  • Sync roles: user
  • Flush every: 6 turns
  • Embedding model: BAAI/bge-small-en-v1.5 (384 dims)
  • Embeddings via API: false (local fastembed)

Database Stats (live, 2026-10-04)

Table Rows Notes
working_memory 773 Active memories
episodic_memory 1,047 Consolidated summaries
memory_embeddings 760 Working-tier embeddings
memory (legacy) 2,181 Old table, not migrated
vec_working / vec_working_rowids 760 Working vector index
vec_episodes / vec_episodes_rowids 1,035 Episodic vector index
vec_facts_rowids 0 memoria_facts is intentionally left unvectorised
Shared surface 0 Empty — cross-agent sharing not functional

Embedding coverage (live, 2026-10-04): 1,759/2,194 memories embedded (80.2%) — working 726/752 (96.5%), episodic 1,033/1,039 (99.4%); 13 embeddings generated in the previous 24h. Working coverage is above the 95% alert threshold enforced by the Embedding Coverage job (b63ed5817212, every 6h, Telegram topic 102).

Known Issues

Tier counts (live, 2026-10-09 22:27): working 773, episodic 1,047, canonical_facts 97, facts 871, memoria_facts 407, memoria_preferences 132, conflicts 152 (1 open — id 152, contradiction, created 2026-10-09 13:11:38), working embeddings 760/773, episodic embeddings 1,043/1,047 (99.6%). Ranking relevance is the live issue, not coverage — see incidents/2026-10-09-memory-recall-relevance.

Tier counts (live, 2026-10-12 12:02 — health job ea79a2107e36, green): working 760/786 embedded (96.7%), episodic 1050/1056 (99.4%) — both above the 95% floor, no reindex needed (b63ed5817212 continues to backfill). Orphan embeddings 0. FTS5 exactly aligned: fts_working 786/786, fts_episodes 1056/1056, fts_facts 876/876. 0 unconsolidated working rows; 1 unresolved conflict (threshold 10); last consolidation 0.3 h earlier; 2,252 memories total (786 working / 1,056 episodic / 1 triple); disk 15 GB free (81% used). Interpretation: coverage is not the live issue — ranking relevance is (see incidents/2026-10-09-memory-recall-relevance).

Tier counts (live, 2026-10-12 00:1x): working 786, episodic 1,056, facts 876 (fts_facts 876 — FTS aligned), fts_working 786, memoria_facts 410, conflicts 152. Embeddings: working 773/786 (98.3%), episodic 1,052/1,056 (99.6%) — both above the 95% reindex threshold; the coverage job b63ed5817212 backfilled 15 vectors since the 2026-10-11 00:08 check.

Capture clock is naive UTC — never compare it to AWST now (2026-10-11 false alarm). At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing working_memory MAX(created_at) = 2026-10-10 13:24:53 against the local clock. created_at is stored as a naive UTC string (the same trap as consolidation_log), so the real gap was 2.7 h: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — working_memory gained 2 rows on 2026-10-11 (11:19:37 and 11:51:39 UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST.

Vector embedding coverage gap — RESOLVED (2026-08-02)

  • Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
  • Resolved by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is 96.5% working / 99.4% episodic, and new embeddings are still being generated. The figures above are kept as the before-state only.
  • Standard repair if coverage drops below 95% working: ~/.hermes/hermes-agent/venv/bin/mnemosyne reindex --yes (the Embedding Coverage job b63ed5817212 triggers the same path every 6h).

Vector recall silently ran FTS-only (2026-10-03 — repaired)

  • fastembed was absent from every uv-managed runtime, so the dense leg of mnemosyne_recall produced no candidates and recall fell back to FTS5/keyword scoring without raising an error. Sessions kept working, which is why it went unnoticed.
  • Compounded by FASTEMBED_CACHE_PATH being unset in ~/.hermes/.env, and by the embedding worker sometimes launching in isolated (python -I) mode, which does not inherit PYTHONPATH.
  • Repaired 2026-10-03: fastembed installed into the bootstrap-selected runtime, FASTEMBED_CACHE_PATH pinned, and hermes-runtime-deps-guard.sh extended to assert fastembed + cache path + model dir every 15 min alongside its existing dotenv/mnemosyne checks.
  • CLI caveat — do not misread mnemosyne doctor. Doctor and the recall CLI run under the legacy venv/ (Python 3.11), where the sqlite_vec extension is not importable; doctor therefore prints vec0: present_but_unloadable and marks the vector tiers unavailable even while the gateway embeds normally. Judge vector health with ~/.hermes/scripts/check_embedding_coverage.py, not with doctor's vec0 line.
  • Benign noise: pthread_setaffinity_np failed … mask: {0,} from onnxruntime in mcp-stderr.log is a container affinity warning emitted on every model load — not an error.
  • See incidents/2026-10-03-mnemosyne-vector-recall-fts-only.

Profile isolation chosen over shared surface

|- The ops profile was enabled with mnemosyne on 2026-08-02 |- Decision: profile_isolation: true was set on ops, giving each profile its own independent memory bank — zero cross-contamination |- data/shared/mnemosyne.db exists with 0 rows; cross-profile sharing is intentionally unused |- This means Tusk (ops) and Rhino (default) do not share memories, which is the desired behaviour |- Config in ~/.hermes/profiles/ops/config.yaml: memory.mnemosyne.profile_isolation: true, allow_tool_override: false, shared_surface_read: false |- Verified end-to-end on 2026-08-02: memory writes land correctly in both profiles independently

Legacy table not migrated

  • 2,181 rows in the old memory table are not in the new working_memory/episodic_memory system
  • These are only accessible via the legacy memory tool, not via mnemosyne_recall

Gateway notifications were captured as user memories (fixed 2026-09-30)

  • Hermes's background-process notifications (long-running terminal jobs reporting completion) were being written into Mnemosyne as [USER] [Anthony Martin] memories — i.e. machine messages were stored as if the user had said them, and then surfaced during recall as if they were his words.
  • 36 such rows were found and purged on 2026-09-30 (working tier plus the corresponding episodic rows). Recall after the purge returned clean.
  • Two canonical facts were also wrong because of the same class of drift: hermes_cron_model_default named the removed provider custom:omni instead of custom:omni-101, and the Daily Marketing Job Radar's model fact was stale. Both corrected.
  • Rule: treat a recalled memory attributed to a person as untrusted until the attribution makes sense in context. Machine-generated status text should never enter the person-attributed tier.

Known Management Jobs

  • Mnemosyne Auto-Consolidate every 30m — script: consolidate_mnemosyne.sh, mode: no-agent
  • Mnemosyne DB Backup weekly Sunday 03:00 — script: mnemosyne-backup.sh, mode: no-agent
  • Mnemosyne Health Monitor every 12h — skill: mnemosyne-operations, reports only on exception
  • Mnemosyne Embedding Coverage every 6h — job b63ed5817212, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
  • Mnemosyne Watchdog every 15m — script: mnemosyne-watchdog.py, mode: no-agent
  • ⚠ Blast-radius note (2026-10-16). Both monitors above (ea79a2107e36 Health every 12 h, b63ed5817212 Embedding Coverage every 6 h) are pinned to custom:omni-101 — the same node the memory system's own reporting depends on. During the 2026-10-14 → 10-15 omniroute-1 outage both failed on every slot, so no coverage figures were refreshed for ~34 hours; the last explicit read remains the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %), and the 2026-10-16 00:09:48 Health Monitor run is the first clean read since. The two script-only jobs (consolidate_mnemosyne.sh, mnemosyne-backup.sh) were unaffected — they make no model call. Monitoring that shares a blast radius with its subject is degraded monitoring; a coverage alert that cannot fire during a router outage is exactly the case this has to cover. → incidents/2026-10-14-omniroute-node-connection-refused-storm

History and Evidence

  • 2026-10-03: vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and incidents/2026-10-03-mnemosyne-vector-recall-fts-only.
  • 2026-09-30 memory-hygiene pass: 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (hermes_cron_model_default → custom:omni-101, job-radar model). No unresolved conflicts left in the working tier.
  • 2026-08-19 Mnemosyne Watchdog incident: Watchdog job (96eb95ce12f2) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See systems/scheduled-tasks Known Issues section.
  • 2026-08-18 incident: Working memory capture (sync_turn) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in run_agent.py:_sync_external_memory_for_turn(). Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See incidents/2026-08-18-mnemosyne-capture-pipeline.
  • 2026-08-01 incident: Automatic conversational capture was broken by stringified structured YAML, especially sync_roles: "['user']". Fixed by rewriting with native YAML types. See incidents/2026-08-01-mnemosyne-capture-failure.
  • 2026-08-01 cleanup: Legacy always-injected memory reduced from 8,013 bytes to 4,998 bytes (memory) and 2,958 bytes (user).
  • 2026-07-29 incident: memory.mnemosyne.tools: None caused schema loading warnings. Fixed by removing the line. See incidents/2026-07-29-mnemosyne-relay.
  • 2026-07-29 relay scope corruption: RuntimeError: scope handle is not at the top of the stack — 3 occurrences, self-resolved.

Re-verified 2026-10-10 (audit window; live probes 2026-10-11 00:14-00:20 AWST)

  • Gateway load path repaired. The gateway's activate_dependencies() selects one committed uv env (795020dbf24343cd9c52df24723dca83); it was missing mnemosyne / mnemosyne_hermes and the native dependencies. Repaired 2026-10-10 21:26-21:34 and confirmed live from 22:01:19 (registered (40 tools) -> activated). Detail: incidents/2026-09-28-hermes-managed-runtime-deps.
  • "Capture silent since 10-09 15:20" was a false alarm. It resolved to no interactive sessions in the window - the window was cron-only traffic. What the surrounding hours did capture was dominated by gateway background-process notifications stored as [USER] [Anthony Martin] [IMPORTANT: Background process ...] rows: recall-visible, low signal. A capture filter would help; no fix applied (carried open item).
  • Coverage script executable again: ~/.hermes/scripts/check_embedding_coverage.py now carries mode 0755, so the monitor's non-executable finding is resolved.
  • Open, unverified: 13 orphan rows in memory_embeddings (oldest 2026-08-22) reported by a 2026-10-10 monitor run - neither confirmed nor cleaned by this audit.