Files
agent-estate-wiki/incidents/2026-08-18-mnemosyne-capture-pipeline.md
T
Tony0410 0c4ed3350b wiki audit 2026-10-02: 2 new cron failures, cliprox/terra connection auth incident, delivery-failure semantics corrected, profile rewrite, broken wikilinks repaired
- new: incidents/2026-10-01-cliproxy-connection-auth-expired.md (HTTP 401, credential fault, cannot self-recover)
- systems/scheduled-tasks.md: 30 jobs / 27 enabled; failing 3 -> 6; streaks advanced; wiki-tree drift recorded
- systems/messaging-integrations.md: two distinct thread-failure variants (lost vs thread-less fallback)
- systems/current-profile.md: full rewrite from live config (no profiles/default/, smart-route, OmniRoute)
- current-state.md / index.md / log.md: 2026-10-02 entry
- repair: copied 5 vault-only pages referenced by index.md (ignis, nextcloud, 3 incidents) so wikilinks resolve
- include pending stocktake output: skills-index.md 2026-09-27, raw/stocktake/freshness-2026-09-2{6,7}.txt
2026-10-02 00:17:15 +08:00

71 lines
3.1 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: "Mnemosyne Capture Pipeline Broken (Aug 18)"
type: incident
status: active
created: 2026-08-18
updated: 2026-08-18
verified_on: 2026-08-18
confidence: high
tags: [memory, mnemosyne, hermes, incident, capture]
sources: [mnemosyne-db, health-monitor, run_agent.py]
---
# Mnemosyne Capture Pipeline Broken — 2026-08-18
## Impact
Working memory capture (`sync_turn`) stopped functioning after **2026-08-16 23:15:39**. As of verification on 2026-08-18, the pipeline had been broken for ~48 hours. Multiple active conversation turns during this period were **not captured** into working memory. The plugin loads, initializes, and tools work — but the post-turn sync path (`_sync_external_memory_for_turn` → `sync_all` → `sync_turn`) is not being triggered.
**What remains healthy:**
- DB: 27M, 326 WM / 693 EP / 352 facts
- FTS indexes: all aligned (OK)
- Embedding coverage: 100% (WM), 201.6% total
- Last consolidation: within threshold
- Plugin: loads and registers correctly; config types are proper YAML
## Root Cause
One of the silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()` (line ~4333–4343) is blocking every turn. Three candidates:
1. `if interrupted: return` — an interrupted flag may be stuck
2. `if not (self._memory_manager and final_response and original_user_message): return` — one of these is falsy
3. `if not (user_text and response_text): return` — text extraction failing
Unlike the Aug 1 incident (stringified YAML `sync_roles`), the config is now clean — native YAML types verified. The failure is in the runtime guard logic, not configuration.
## Diagnostic Steps
Add temporary diagnostic logging before each guard in `run_agent.py:_sync_external_memory_for_turn()` (around line 4333):
```python
if interrupted:
logger.warning("MEMDEBUG: skip - interrupted=True")
if not (self._memory_manager and final_response and original_user_message):
logger.warning("MEMDEBUG: skip - manager=%s final=%s orig=%s",
bool(self._memory_manager), bool(final_response), bool(original_user_message))
if not (user_text and response_text):
logger.warning("MEMDEBUG: skip - user_text=%s response_text=%s",
bool(user_text), bool(response_text))
```
Then restart gateway (`hermes gateway restart` from external shell) and `grep MEMDEBUG ~/.hermes/logs/agent.log` after one conversation turn to identify which guard fires.
## Verification
```bash
python3 -c "
import sqlite3
db = '/home/hermes/.hermes/mnemosyne/data/mnemosyne.db'
conn = sqlite3.connect(db)
cur = conn.cursor()
cur.execute('SELECT MAX(created_at) FROM working_memory')
print('Last WM capture:', cur.fetchone()[0]) # Should be recent
"
```
## Prevention
- Never use `hermes config set memory.mnemosyne.<field>` for list/dict/null values
- After any config change, verify YAML types with `yaml.safe_load` + `type()` on each field
- After any gateway restart, verify capture within 2 hours via health monitor
## Related
- [[systems/mnemosyne]]
- [[incidents/2026-08-01-mnemosyne-capture-failure]] — Previous capture failure (YAML stringification)
- [[incidents/2026-07-29-mnemosyne-relay]] — Earlier relay scope corruption