Files
agent-estate-wiki/incidents/2026-08-01-mnemosyne-capture-failure.md
T
Hermes 341dda95dd stocktake 2026-11-11: 36 stale pages re-verified (0 stale); fallback chain rewritten 4/6 slots; CT333 'clawtest' naming claim corrected; nous 404 credits signature replaces refresh-token 401; estate shrank (VM 310 + CT601 gone)
- freshness: 36 stale / 93 fresh / 3 no-date -> 0 stale / 129 fresh / 3 no-date of 132 (same-day report regenerated after the corrections, per pattern 13)
- skills: 299 skills / 67 categories (count unchanged; generator arithmetic re-verified by hand against the live tree: 300 SKILL.md files - 1 nested duplicate = 299)
- runbooks/test-openrouter: live fallback chain rewritten (primary back to smart-route/custom:omni-101; mimo-v2.5, stepfun, qwen3.5/ollama-cloud out; guacano + kilo-reflection-beam-free in); ollama.com dropped from the URL test list; guacano/litellm noted as internal + key-gated
- comparisons/hermes-vs-nanobot: the 2026-10-20 'CT333 is zeroclaw in Proxmox naming' claim was itself wrong - pct list and hostname both return clawtest
- incidents/2026-10-22-nous: refresh-token failure gone from logs; new signature 404 insufficient_credits_for_paid_model from auxiliary.pulse_memory (ninth slot, back on nous, paid model) - open item revised
- incidents/2026-08-29-gmail: email crons now ok but root cause unfixed 74 days (himalaya message read still in fetch_newsletters.py; goimapnotify back up PID 441; monitor still paused)
- 30 further pages re-verified live and de-flagged with dated probe notes (ct207 ingress, CT205 guanaco v0.8.15, CT270 nextcloud 33.0.2.2, headless stack, hermes-ui both hosts, systemd units + cgroup limits, terminal/tool_search/mnemosyne/custom_providers config blocks, ops profile still deleted, hosts/pve guest lists)
- log.md addendum + index.md date bump; no page deleted, no secrets written
2026-11-11 03:52:48 +08:00

5.4 KiB

title, type, status, created, updated, resolved_on, verified_on, confidence, tags, sources
title type status created updated resolved_on verified_on confidence tags sources
Mnemosyne Automatic Capture Failure incident active 2026-08-01 2026-11-11 2026-10-20 2026-11-11 high
memory
mnemosyne
hermes
incident
live-config
mnemosyne-db
hermes-cli-e2e

Mnemosyne Automatic Capture Failure — 2026-08-01

Impact

Ordinary Hermes conversations were not being written to Mnemosyne working_memory. Explicit memory tools, SQLite integrity, vectors, FTS indexes, consolidation, backups, and cron monitoring remained operational. The newest automatic/explicit working row before repair was approximately fourteen hours old despite active Hermes sessions.

The always-injected legacy files were also effectively full: MEMORY.md was 5,055 bytes against a 5,000-character limit and USER.md was 2,958 bytes against 3,000. Legacy writes had begun failing.

Root Cause

Structured values under memory.mnemosyne in ~/.hermes/config.yaml had been serialized as strings instead of native YAML types. The critical value was:

sync_roles: "['user']"

The provider interpreted this as the literal invalid role ['user'], so turn synchronization completed without persisting a user row. Other malformed values included skip_contexts, reflect, sleep_threshold, and ignore_patterns.

Repair

  • Backed up config to /home/hermes/.hermes/backups/mnemosyne-capture-fix-20260801_180249/config.yaml.before.
  • Rewrote structured Mnemosyne settings with native YAML types.
  • Set profile_isolation: true because the installation has multiple Hermes profiles.
  • Preserved sync_roles: [user] and the intended skipped contexts.
  • Ran hermes config check successfully.
  • Created authoritative Mnemosyne canonical slots for user preferences and key workflows, correcting stale light-theme, numbered-Obsidian, and CT460 route entries.
  • Backed up legacy files to /home/hermes/.hermes/backups/legacy-memory-slim-20260801_180702/.
  • Reduced legacy injection from 8,013 bytes to 1,080 bytes while retaining essential always-on rules.

Verification

A pre-fix CLI turn using marker MNEMO_E2E_20260801_180113_27053 completed but produced no database row.

A post-fix CLI turn using marker MNEMO_FIXED_20260801_180310_16583 wrote:

[USER] Reply with exactly OK. Memory capture test marker: MNEMO_FIXED_20260801_180310_16583

The row was verified directly in ~/.hermes/mnemosyne/data/mnemosyne.db. Final diagnostics reported zero failed checks, complete working-vector coverage, zero orphan embeddings, 1,482 working rows, and 463 episodic vectors.

Remaining Activation Step

The already-running Telegram gateway/provider instance was initialized before the config repair. Run hermes gateway restart from an external terminal to load the corrected typed configuration for messaging-platform turns. The gateway cannot safely restart itself from one of its own child tool calls.

Monitoring Note

The health script checks capture staleness only after 24 hours. It therefore remained green during this fourteen-hour active-capture outage. A future improvement should compare recent non-cron conversation activity with the newest working_memory row rather than lowering the threshold blindly and creating idle-period false positives.

Resolution confirmed (2026-10-20)

The remaining "Activation Step" on this page — restarting the gateway to load the corrected typed configuration — has been completed and the config is still correct 50 days later. Verified live in /home/hermes/.hermes/config.yaml under memory.mnemosyne:

Key Live value Was it a string?
sync_roles ['user'] (real list) no — now a native list
skip_contexts ['cron', 'flush', 'subagent', 'background', 'skill_loop'] no — native list
reflect {disabled_for_cron: True, max_calls_per_session: 3} (dict) no — native dict
sleep_threshold 50 (int) no — int
ignore_patterns [] (list) no — native list
profile_isolation True (bool) no — bool
auto_sleep / default_scope / vector_type True / session / int8 no

None of these are stringified. The hermes config set stringification bug that caused this incident has not recurred for this block.

One drift to note: skip_contexts now includes cron, which was not in the original repair list. Mnemosyne capture is therefore deliberately skipped in cron contexts — scheduled jobs produce no working memories by design. That explains why cron-driven runs of this wiki job leave no Mnemosyne trace.

Status: resolved. The back-up copies named in the repair section still exist: ~/.hermes/backups/mnemosyne-capture-fix-20260801_180249/ and ~/.hermes/backups/legacy-memory-slim-20260801_180702/.

Resolution re-confirmed 2026-11-11. The whole memory.mnemosyne block in ~/.hermes/config.yaml is still natively typed — sync_roles: [user] (list), skip_contexts: [cron, flush, subagent, background, skill_loop] (list), reflect: {disabled_for_cron: true, max_calls_per_session: 3} (dict), sleep_threshold: 50 (int), ignore_patterns: [] (list), profile_isolation: true (bool), auto_sleep: true, default_scope: session, vector_type: int8. No stringification regression. Both backup directories named in the repair still exist. status: resolved stands.