Files
agent-estate-wiki/incidents/2026-08-18-mnemosyne-capture-pipeline.md
T

4.0 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Mnemosyne Capture Pipeline Broken (Aug 18) incident superseded 2026-08-18 2026-10-12 2026-10-12 high
memory
mnemosyne
hermes
incident
capture
mnemosyne-db
health-monitor
run_agent.py

Mnemosyne Capture Pipeline Broken — 2026-08-18

Impact

Working memory capture (sync_turn) stopped functioning after 2026-08-16 23:15:39. As of verification on 2026-08-18, the pipeline had been broken for ~48 hours. Multiple active conversation turns during this period were not captured into working memory. The plugin loads, initializes, and tools work — but the post-turn sync path (_sync_external_memory_for_turn → sync_all → sync_turn) is not being triggered.

What remains healthy:

  • DB: 27M, 326 WM / 693 EP / 352 facts
  • FTS indexes: all aligned (OK)
  • Embedding coverage: 100% (WM), 201.6% total
  • Last consolidation: within threshold
  • Plugin: loads and registers correctly; config types are proper YAML

Root Cause

One of the silent guard conditions in run_agent.py:_sync_external_memory_for_turn() (line ~4333–4343) is blocking every turn. Three candidates:

  1. if interrupted: return — an interrupted flag may be stuck
  2. if not (self._memory_manager and final_response and original_user_message): return — one of these is falsy
  3. if not (user_text and response_text): return — text extraction failing

Unlike the Aug 1 incident (stringified YAML sync_roles), the config is now clean — native YAML types verified. The failure is in the runtime guard logic, not configuration.

Diagnostic Steps

Add temporary diagnostic logging before each guard in run_agent.py:_sync_external_memory_for_turn() (around line 4333):

if interrupted:
    logger.warning("MEMDEBUG: skip - interrupted=True")
if not (self._memory_manager and final_response and original_user_message):
    logger.warning("MEMDEBUG: skip - manager=%s final=%s orig=%s",
                   bool(self._memory_manager), bool(final_response), bool(original_user_message))
if not (user_text and response_text):
    logger.warning("MEMDEBUG: skip - user_text=%s response_text=%s",
                   bool(user_text), bool(response_text))

Then restart gateway (hermes gateway restart from external shell) and grep MEMDEBUG ~/.hermes/logs/agent.log after one conversation turn to identify which guard fires.

Verification

python3 -c "
import sqlite3
db = '/home/hermes/.hermes/mnemosyne/data/mnemosyne.db'
conn = sqlite3.connect(db)
cur = conn.cursor()
cur.execute('SELECT MAX(created_at) FROM working_memory')
print('Last WM capture:', cur.fetchone()[0])  # Should be recent
"

Current Status (verified 2026-10-12)

Superseded — capture has been healthy since, and this page is kept for the diagnostic trail. The 2026-10-12 nightly audit found the full path working: working_memory at 786 rows, +2 on 2026-10-11 (11:19:37 and 11:51:39 UTC = 19:19 / 19:51 AWST, both from that day's interactive session), episodic_memory 1,056, embedding coverage 98.3 % working / 99.6 % episodic, FTS aligned (fts_working 786, fts_facts 876). The three guard conditions diagnosed below were never individually confirmed as the fix — the path resumed capturing and has captured through every window since.

Caution for the next alarm: the health monitor raised a false EXCEPTION on 2026-10-11 00:08 by comparing naive-UTC created_at against AWST now ("stalled 10.7 h" — real gap 2.7 h). Do not triage that as a repeat of this incident; see systems/mnemosyne.

Prevention

  • Never use hermes config set memory.mnemosyne.<field> for list/dict/null values
  • After any config change, verify YAML types with yaml.safe_load + type() on each field
  • After any gateway restart, verify capture within 2 hours via health monitor