Files
agent-estate-wiki/incidents/2026-08-18-mnemosyne-capture-pipeline.md

107 lines
5.4 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: "Mnemosyne Capture Pipeline Broken (Aug 18)"
type: incident
status: superseded
created: 2026-08-18
updated: 2026-11-04
verified_on: 2026-11-04
confidence: high
tags: [memory, mnemosyne, hermes, incident, capture]
sources: [mnemosyne-db, health-monitor, run_agent.py]
---
# Mnemosyne Capture Pipeline Broken — 2026-08-18
## Impact
Working memory capture (`sync_turn`) stopped functioning after **2026-08-16 23:15:39**. As of verification on 2026-08-18, the pipeline had been broken for ~48 hours. Multiple active conversation turns during this period were **not captured** into working memory. The plugin loads, initializes, and tools work — but the post-turn sync path (`_sync_external_memory_for_turn` → `sync_all` → `sync_turn`) is not being triggered.
**What remains healthy:**
- DB: 27M, 326 WM / 693 EP / 352 facts
- FTS indexes: all aligned (OK)
- Embedding coverage: 100% (WM), 201.6% total
- Last consolidation: within threshold
- Plugin: loads and registers correctly; config types are proper YAML
## Root Cause
One of the silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()` (line ~4333–4343) is blocking every turn. Three candidates:
1. `if interrupted: return` — an interrupted flag may be stuck
2. `if not (self._memory_manager and final_response and original_user_message): return` — one of these is falsy
3. `if not (user_text and response_text): return` — text extraction failing
Unlike the Aug 1 incident (stringified YAML `sync_roles`), the config is now clean — native YAML types verified. The failure is in the runtime guard logic, not configuration.
## Diagnostic Steps
Add temporary diagnostic logging before each guard in `run_agent.py:_sync_external_memory_for_turn()` (around line 4333):
```python
if interrupted:
logger.warning("MEMDEBUG: skip - interrupted=True")
if not (self._memory_manager and final_response and original_user_message):
logger.warning("MEMDEBUG: skip - manager=%s final=%s orig=%s",
bool(self._memory_manager), bool(final_response), bool(original_user_message))
if not (user_text and response_text):
logger.warning("MEMDEBUG: skip - user_text=%s response_text=%s",
bool(user_text), bool(response_text))
```
Then restart gateway (`hermes gateway restart` from external shell) and `grep MEMDEBUG ~/.hermes/logs/agent.log` after one conversation turn to identify which guard fires.
## Verification
```bash
python3 -c "
import sqlite3
db = '/home/hermes/.hermes/mnemosyne/data/mnemosyne.db'
conn = sqlite3.connect(db)
cur = conn.cursor()
cur.execute('SELECT MAX(created_at) FROM working_memory')
print('Last WM capture:', cur.fetchone()[0]) # Should be recent
"
```
## Current Status (verified 2026-10-12)
**Superseded — capture has been healthy since, and this page is kept for the diagnostic trail.** The 2026-10-12
nightly audit found the full path working: `working_memory` at **786 rows**, +2 on 2026-10-11 (11:19:37 and
11:51:39 UTC = 19:19 / 19:51 AWST, both from that day's interactive session), `episodic_memory` 1,056,
embedding coverage **98.3 % working / 99.6 % episodic**, FTS aligned (`fts_working` 786, `fts_facts` 876).
The three guard conditions diagnosed below were never individually confirmed as the fix — the path resumed
capturing and has captured through every window since.
### Re-verified 2026-11-04 — capture still healthy, and the counts have moved on
Live state of `~/.hermes/mnemosyne/data/mnemosyne.db` (the path in the Verification snippet above is still
correct; note the DB is under `data/`, not the `~/.hermes/mnemosyne/*.db` glob path):
| Metric | 2026-10-12 (page) | 2026-11-04 (live) |
|---|---|---|
| `working_memory` | 786 | **970** |
| `episodic_memory` | 1,056 | **1,146** |
| `memoria_facts` | — | **446** |
| `memory_embeddings` | — | **943** |
| Working-memory vector coverage | 98.3 % | **98.6 %** (`vec_working` 956 / `working_memory` 970) |
| Episodic vector coverage | 99.6 % | **100 %** (`vec_episodes` 1,146 / `episodic_memory` 1,146) |
| `fts_working` | 786 | **970** — aligned with `working_memory` |
| `fts_facts` | 876 | **927** (vs `memoria_facts` 446 — FTS indexes a superset) |
| Last `working_memory` entry | 2026-10-11 | **2026-11-03 17:12:40** |
`mnemosyne stats` independently reports **Total 2,562 / Working 970 / Episodic 1,146**, and the `vec0` extension
loads cleanly (`sqlite-vec` from the agent venv), confirming vector search is live rather than silently FTS-only.
**Working-memory coverage is above the 95 % reindex threshold**, so no `mnemosyne reindex` was triggered by this
audit. **Capture has continued to work without incident for a full month** — the original failure has not
recurred.
**Caution for the next alarm:** the health monitor raised a false EXCEPTION on 2026-10-11 00:08 by comparing
naive-UTC `created_at` against AWST `now` ("stalled 10.7 h" — real gap 2.7 h). Do not triage that as a repeat
of this incident; see [[systems/mnemosyne]].
## Prevention
- Never use `hermes config set memory.mnemosyne.<field>` for list/dict/null values
- After any config change, verify YAML types with `yaml.safe_load` + `type()` on each field
- After any gateway restart, verify capture within 2 hours via health monitor
## Related
- [[systems/mnemosyne]]
- [[incidents/2026-08-01-mnemosyne-capture-failure]] — Previous capture failure (YAML stringification)
- [[incidents/2026-07-29-mnemosyne-relay]] — Earlier relay scope corruption