5.4 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mnemosyne Capture Pipeline Broken (Aug 18) | incident | superseded | 2026-08-18 | 2026-11-04 | 2026-11-04 | high |
|
|
Mnemosyne Capture Pipeline Broken — 2026-08-18
Impact
Working memory capture (sync_turn) stopped functioning after 2026-08-16 23:15:39. As of verification on 2026-08-18, the pipeline had been broken for ~48 hours. Multiple active conversation turns during this period were not captured into working memory. The plugin loads, initializes, and tools work — but the post-turn sync path (_sync_external_memory_for_turn → sync_all → sync_turn) is not being triggered.
What remains healthy:
- DB: 27M, 326 WM / 693 EP / 352 facts
- FTS indexes: all aligned (OK)
- Embedding coverage: 100% (WM), 201.6% total
- Last consolidation: within threshold
- Plugin: loads and registers correctly; config types are proper YAML
Root Cause
One of the silent guard conditions in run_agent.py:_sync_external_memory_for_turn() (line ~4333–4343) is blocking every turn. Three candidates:
if interrupted: return— an interrupted flag may be stuckif not (self._memory_manager and final_response and original_user_message): return— one of these is falsyif not (user_text and response_text): return— text extraction failing
Unlike the Aug 1 incident (stringified YAML sync_roles), the config is now clean — native YAML types verified. The failure is in the runtime guard logic, not configuration.
Diagnostic Steps
Add temporary diagnostic logging before each guard in run_agent.py:_sync_external_memory_for_turn() (around line 4333):
if interrupted:
logger.warning("MEMDEBUG: skip - interrupted=True")
if not (self._memory_manager and final_response and original_user_message):
logger.warning("MEMDEBUG: skip - manager=%s final=%s orig=%s",
bool(self._memory_manager), bool(final_response), bool(original_user_message))
if not (user_text and response_text):
logger.warning("MEMDEBUG: skip - user_text=%s response_text=%s",
bool(user_text), bool(response_text))
Then restart gateway (hermes gateway restart from external shell) and grep MEMDEBUG ~/.hermes/logs/agent.log after one conversation turn to identify which guard fires.
Verification
python3 -c "
import sqlite3
db = '/home/hermes/.hermes/mnemosyne/data/mnemosyne.db'
conn = sqlite3.connect(db)
cur = conn.cursor()
cur.execute('SELECT MAX(created_at) FROM working_memory')
print('Last WM capture:', cur.fetchone()[0]) # Should be recent
"
Current Status (verified 2026-10-12)
Superseded — capture has been healthy since, and this page is kept for the diagnostic trail. The 2026-10-12
nightly audit found the full path working: working_memory at 786 rows, +2 on 2026-10-11 (11:19:37 and
11:51:39 UTC = 19:19 / 19:51 AWST, both from that day's interactive session), episodic_memory 1,056,
embedding coverage 98.3 % working / 99.6 % episodic, FTS aligned (fts_working 786, fts_facts 876).
The three guard conditions diagnosed below were never individually confirmed as the fix — the path resumed
capturing and has captured through every window since.
Re-verified 2026-11-04 — capture still healthy, and the counts have moved on
Live state of ~/.hermes/mnemosyne/data/mnemosyne.db (the path in the Verification snippet above is still
correct; note the DB is under data/, not the ~/.hermes/mnemosyne/*.db glob path):
| Metric | 2026-10-12 (page) | 2026-11-04 (live) |
|---|---|---|
working_memory |
786 | 970 |
episodic_memory |
1,056 | 1,146 |
memoria_facts |
— | 446 |
memory_embeddings |
— | 943 |
| Working-memory vector coverage | 98.3 % | 98.6 % (vec_working 956 / working_memory 970) |
| Episodic vector coverage | 99.6 % | 100 % (vec_episodes 1,146 / episodic_memory 1,146) |
fts_working |
786 | 970 — aligned with working_memory |
fts_facts |
876 | 927 (vs memoria_facts 446 — FTS indexes a superset) |
Last working_memory entry |
2026-10-11 | 2026-11-03 17:12:40 |
mnemosyne stats independently reports Total 2,562 / Working 970 / Episodic 1,146, and the vec0 extension
loads cleanly (sqlite-vec from the agent venv), confirming vector search is live rather than silently FTS-only.
Working-memory coverage is above the 95 % reindex threshold, so no mnemosyne reindex was triggered by this
audit. Capture has continued to work without incident for a full month — the original failure has not
recurred.
Caution for the next alarm: the health monitor raised a false EXCEPTION on 2026-10-11 00:08 by comparing
naive-UTC created_at against AWST now ("stalled 10.7 h" — real gap 2.7 h). Do not triage that as a repeat
of this incident; see systems/mnemosyne.
Prevention
- Never use
hermes config set memory.mnemosyne.<field>for list/dict/null values - After any config change, verify YAML types with
yaml.safe_load+type()on each field - After any gateway restart, verify capture within 2 hours via health monitor
Related
- systems/mnemosyne
- incidents/2026-08-01-mnemosyne-capture-failure — Previous capture failure (YAML stringification)
- incidents/2026-07-29-mnemosyne-relay — Earlier relay scope corruption