--- title: "Mnemosyne Capture Pipeline Broken (Aug 18)" type: incident status: superseded created: 2026-08-18 updated: 2026-10-12 verified_on: 2026-10-12 confidence: high tags: [memory, mnemosyne, hermes, incident, capture] sources: [mnemosyne-db, health-monitor, run_agent.py] --- # Mnemosyne Capture Pipeline Broken — 2026-08-18 ## Impact Working memory capture (`sync_turn`) stopped functioning after **2026-08-16 23:15:39**. As of verification on 2026-08-18, the pipeline had been broken for ~48 hours. Multiple active conversation turns during this period were **not captured** into working memory. The plugin loads, initializes, and tools work — but the post-turn sync path (`_sync_external_memory_for_turn` → `sync_all` → `sync_turn`) is not being triggered. **What remains healthy:** - DB: 27M, 326 WM / 693 EP / 352 facts - FTS indexes: all aligned (OK) - Embedding coverage: 100% (WM), 201.6% total - Last consolidation: within threshold - Plugin: loads and registers correctly; config types are proper YAML ## Root Cause One of the silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()` (line ~4333–4343) is blocking every turn. Three candidates: 1. `if interrupted: return` — an interrupted flag may be stuck 2. `if not (self._memory_manager and final_response and original_user_message): return` — one of these is falsy 3. `if not (user_text and response_text): return` — text extraction failing Unlike the Aug 1 incident (stringified YAML `sync_roles`), the config is now clean — native YAML types verified. The failure is in the runtime guard logic, not configuration. ## Diagnostic Steps Add temporary diagnostic logging before each guard in `run_agent.py:_sync_external_memory_for_turn()` (around line 4333): ```python if interrupted: logger.warning("MEMDEBUG: skip - interrupted=True") if not (self._memory_manager and final_response and original_user_message): logger.warning("MEMDEBUG: skip - manager=%s final=%s orig=%s", bool(self._memory_manager), bool(final_response), bool(original_user_message)) if not (user_text and response_text): logger.warning("MEMDEBUG: skip - user_text=%s response_text=%s", bool(user_text), bool(response_text)) ``` Then restart gateway (`hermes gateway restart` from external shell) and `grep MEMDEBUG ~/.hermes/logs/agent.log` after one conversation turn to identify which guard fires. ## Verification ```bash python3 -c " import sqlite3 db = '/home/hermes/.hermes/mnemosyne/data/mnemosyne.db' conn = sqlite3.connect(db) cur = conn.cursor() cur.execute('SELECT MAX(created_at) FROM working_memory') print('Last WM capture:', cur.fetchone()[0]) # Should be recent " ``` ## Current Status (verified 2026-10-12) **Superseded — capture has been healthy since, and this page is kept for the diagnostic trail.** The 2026-10-12 nightly audit found the full path working: `working_memory` at **786 rows**, +2 on 2026-10-11 (11:19:37 and 11:51:39 UTC = 19:19 / 19:51 AWST, both from that day's interactive session), `episodic_memory` 1,056, embedding coverage **98.3 % working / 99.6 % episodic**, FTS aligned (`fts_working` 786, `fts_facts` 876). The three guard conditions diagnosed below were never individually confirmed as the fix — the path resumed capturing and has captured through every window since. **Caution for the next alarm:** the health monitor raised a false EXCEPTION on 2026-10-11 00:08 by comparing naive-UTC `created_at` against AWST `now` ("stalled 10.7 h" — real gap 2.7 h). Do not triage that as a repeat of this incident; see [[systems/mnemosyne]]. ## Prevention - Never use `hermes config set memory.mnemosyne.` for list/dict/null values - After any config change, verify YAML types with `yaml.safe_load` + `type()` on each field - After any gateway restart, verify capture within 2 hours via health monitor ## Related - [[systems/mnemosyne]] - [[incidents/2026-08-01-mnemosyne-capture-failure]] — Previous capture failure (YAML stringification) - [[incidents/2026-07-29-mnemosyne-relay]] — Earlier relay scope corruption