wiki audit 2026-10-02: 2 new cron failures, cliprox/terra connection auth incident, delivery-failure semantics corrected, profile rewrite, broken wikilinks repaired
- new: incidents/2026-10-01-cliproxy-connection-auth-expired.md (HTTP 401, credential fault, cannot self-recover)
- systems/scheduled-tasks.md: 30 jobs / 27 enabled; failing 3 -> 6; streaks advanced; wiki-tree drift recorded
- systems/messaging-integrations.md: two distinct thread-failure variants (lost vs thread-less fallback)
- systems/current-profile.md: full rewrite from live config (no profiles/default/, smart-route, OmniRoute)
- current-state.md / index.md / log.md: 2026-10-02 entry
- repair: copied 5 vault-only pages referenced by index.md (ignis, nextcloud, 3 incidents) so wikilinks resolve
- include pending stocktake output: skills-index.md 2026-09-27, raw/stocktake/freshness-2026-09-2{6,7}.txt
This commit is contained in:
@@ -0,0 +1,70 @@
|
||||
---
|
||||
title: "Mnemosyne Capture Pipeline Broken (Aug 18)"
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-08-18
|
||||
updated: 2026-08-18
|
||||
verified_on: 2026-08-18
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, hermes, incident, capture]
|
||||
sources: [mnemosyne-db, health-monitor, run_agent.py]
|
||||
---
|
||||
|
||||
# Mnemosyne Capture Pipeline Broken — 2026-08-18
|
||||
|
||||
## Impact
|
||||
Working memory capture (`sync_turn`) stopped functioning after **2026-08-16 23:15:39**. As of verification on 2026-08-18, the pipeline had been broken for ~48 hours. Multiple active conversation turns during this period were **not captured** into working memory. The plugin loads, initializes, and tools work — but the post-turn sync path (`_sync_external_memory_for_turn` → `sync_all` → `sync_turn`) is not being triggered.
|
||||
|
||||
**What remains healthy:**
|
||||
- DB: 27M, 326 WM / 693 EP / 352 facts
|
||||
- FTS indexes: all aligned (OK)
|
||||
- Embedding coverage: 100% (WM), 201.6% total
|
||||
- Last consolidation: within threshold
|
||||
- Plugin: loads and registers correctly; config types are proper YAML
|
||||
|
||||
## Root Cause
|
||||
One of the silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()` (line ~4333–4343) is blocking every turn. Three candidates:
|
||||
|
||||
1. `if interrupted: return` — an interrupted flag may be stuck
|
||||
2. `if not (self._memory_manager and final_response and original_user_message): return` — one of these is falsy
|
||||
3. `if not (user_text and response_text): return` — text extraction failing
|
||||
|
||||
Unlike the Aug 1 incident (stringified YAML `sync_roles`), the config is now clean — native YAML types verified. The failure is in the runtime guard logic, not configuration.
|
||||
|
||||
## Diagnostic Steps
|
||||
Add temporary diagnostic logging before each guard in `run_agent.py:_sync_external_memory_for_turn()` (around line 4333):
|
||||
|
||||
```python
|
||||
if interrupted:
|
||||
logger.warning("MEMDEBUG: skip - interrupted=True")
|
||||
if not (self._memory_manager and final_response and original_user_message):
|
||||
logger.warning("MEMDEBUG: skip - manager=%s final=%s orig=%s",
|
||||
bool(self._memory_manager), bool(final_response), bool(original_user_message))
|
||||
if not (user_text and response_text):
|
||||
logger.warning("MEMDEBUG: skip - user_text=%s response_text=%s",
|
||||
bool(user_text), bool(response_text))
|
||||
```
|
||||
|
||||
Then restart gateway (`hermes gateway restart` from external shell) and `grep MEMDEBUG ~/.hermes/logs/agent.log` after one conversation turn to identify which guard fires.
|
||||
|
||||
## Verification
|
||||
```bash
|
||||
python3 -c "
|
||||
import sqlite3
|
||||
db = '/home/hermes/.hermes/mnemosyne/data/mnemosyne.db'
|
||||
conn = sqlite3.connect(db)
|
||||
cur = conn.cursor()
|
||||
cur.execute('SELECT MAX(created_at) FROM working_memory')
|
||||
print('Last WM capture:', cur.fetchone()[0]) # Should be recent
|
||||
"
|
||||
```
|
||||
|
||||
## Prevention
|
||||
- Never use `hermes config set memory.mnemosyne.<field>` for list/dict/null values
|
||||
- After any config change, verify YAML types with `yaml.safe_load` + `type()` on each field
|
||||
- After any gateway restart, verify capture within 2 hours via health monitor
|
||||
|
||||
## Related
|
||||
- [[systems/mnemosyne]]
|
||||
- [[incidents/2026-08-01-mnemosyne-capture-failure]] — Previous capture failure (YAML stringification)
|
||||
- [[incidents/2026-07-29-mnemosyne-relay]] — Earlier relay scope corruption
|
||||
Reference in New Issue
Block a user