- freshness: 36 stale / 93 fresh / 3 no-date -> 0 stale / 129 fresh / 3 no-date of 132 (same-day report regenerated after the corrections, per pattern 13) - skills: 299 skills / 67 categories (count unchanged; generator arithmetic re-verified by hand against the live tree: 300 SKILL.md files - 1 nested duplicate = 299) - runbooks/test-openrouter: live fallback chain rewritten (primary back to smart-route/custom:omni-101; mimo-v2.5, stepfun, qwen3.5/ollama-cloud out; guacano + kilo-reflection-beam-free in); ollama.com dropped from the URL test list; guacano/litellm noted as internal + key-gated - comparisons/hermes-vs-nanobot: the 2026-10-20 'CT333 is zeroclaw in Proxmox naming' claim was itself wrong - pct list and hostname both return clawtest - incidents/2026-10-22-nous: refresh-token failure gone from logs; new signature 404 insufficient_credits_for_paid_model from auxiliary.pulse_memory (ninth slot, back on nous, paid model) - open item revised - incidents/2026-08-29-gmail: email crons now ok but root cause unfixed 74 days (himalaya message read still in fetch_newsletters.py; goimapnotify back up PID 441; monitor still paused) - 30 further pages re-verified live and de-flagged with dated probe notes (ct207 ingress, CT205 guanaco v0.8.15, CT270 nextcloud 33.0.2.2, headless stack, hermes-ui both hosts, systemd units + cgroup limits, terminal/tool_search/mnemosyne/custom_providers config blocks, ops profile still deleted, hosts/pve guest lists) - log.md addendum + index.md date bump; no page deleted, no secrets written
97 lines
5.4 KiB
Markdown
97 lines
5.4 KiB
Markdown
---
|
|
title: Mnemosyne Automatic Capture Failure
|
|
type: incident
|
|
status: active
|
|
created: 2026-08-01
|
|
updated: 2026-11-11
|
|
resolved_on: 2026-10-20
|
|
verified_on: 2026-11-11
|
|
confidence: high
|
|
tags: [memory, mnemosyne, hermes, incident]
|
|
sources: [live-config, mnemosyne-db, hermes-cli-e2e]
|
|
---
|
|
|
|
# Mnemosyne Automatic Capture Failure — 2026-08-01
|
|
|
|
## Impact
|
|
Ordinary Hermes conversations were not being written to Mnemosyne `working_memory`. Explicit memory tools, SQLite integrity, vectors, FTS indexes, consolidation, backups, and cron monitoring remained operational. The newest automatic/explicit working row before repair was approximately fourteen hours old despite active Hermes sessions.
|
|
|
|
The always-injected legacy files were also effectively full: `MEMORY.md` was 5,055 bytes against a 5,000-character limit and `USER.md` was 2,958 bytes against 3,000. Legacy writes had begun failing.
|
|
|
|
## Root Cause
|
|
Structured values under `memory.mnemosyne` in `~/.hermes/config.yaml` had been serialized as strings instead of native YAML types. The critical value was:
|
|
|
|
```yaml
|
|
sync_roles: "['user']"
|
|
```
|
|
|
|
The provider interpreted this as the literal invalid role `['user']`, so turn synchronization completed without persisting a user row. Other malformed values included `skip_contexts`, `reflect`, `sleep_threshold`, and `ignore_patterns`.
|
|
|
|
## Repair
|
|
- Backed up config to `/home/hermes/.hermes/backups/mnemosyne-capture-fix-20260801_180249/config.yaml.before`.
|
|
- Rewrote structured Mnemosyne settings with native YAML types.
|
|
- Set `profile_isolation: true` because the installation has multiple Hermes profiles.
|
|
- Preserved `sync_roles: [user]` and the intended skipped contexts.
|
|
- Ran `hermes config check` successfully.
|
|
- Created authoritative Mnemosyne canonical slots for user preferences and key workflows, correcting stale light-theme, numbered-Obsidian, and CT460 route entries.
|
|
- Backed up legacy files to `/home/hermes/.hermes/backups/legacy-memory-slim-20260801_180702/`.
|
|
- Reduced legacy injection from 8,013 bytes to 1,080 bytes while retaining essential always-on rules.
|
|
|
|
## Verification
|
|
A pre-fix CLI turn using marker `MNEMO_E2E_20260801_180113_27053` completed but produced no database row.
|
|
|
|
A post-fix CLI turn using marker `MNEMO_FIXED_20260801_180310_16583` wrote:
|
|
|
|
```text
|
|
[USER] Reply with exactly OK. Memory capture test marker: MNEMO_FIXED_20260801_180310_16583
|
|
```
|
|
|
|
The row was verified directly in `~/.hermes/mnemosyne/data/mnemosyne.db`. Final diagnostics reported zero failed checks, complete working-vector coverage, zero orphan embeddings, 1,482 working rows, and 463 episodic vectors.
|
|
|
|
## Remaining Activation Step
|
|
The already-running Telegram gateway/provider instance was initialized before the config repair. Run `hermes gateway restart` from an external terminal to load the corrected typed configuration for messaging-platform turns. The gateway cannot safely restart itself from one of its own child tool calls.
|
|
|
|
## Monitoring Note
|
|
The health script checks capture staleness only after 24 hours. It therefore remained green during this fourteen-hour active-capture outage. A future improvement should compare recent non-cron conversation activity with the newest `working_memory` row rather than lowering the threshold blindly and creating idle-period false positives.
|
|
|
|
## Resolution confirmed (2026-10-20)
|
|
|
|
The remaining "Activation Step" on this page — restarting the gateway to load the corrected typed
|
|
configuration — **has been completed and the config is still correct 50 days later.** Verified live
|
|
in `/home/hermes/.hermes/config.yaml` under `memory.mnemosyne`:
|
|
|
|
| Key | Live value | Was it a string? |
|
|
|---|---|---|
|
|
| `sync_roles` | `['user']` (real list) | no — now a native list |
|
|
| `skip_contexts` | `['cron', 'flush', 'subagent', 'background', 'skill_loop']` | no — native list |
|
|
| `reflect` | `{disabled_for_cron: True, max_calls_per_session: 3}` (dict) | no — native dict |
|
|
| `sleep_threshold` | `50` (int) | no — int |
|
|
| `ignore_patterns` | `[]` (list) | no — native list |
|
|
| `profile_isolation` | `True` (bool) | no — bool |
|
|
| `auto_sleep` / `default_scope` / `vector_type` | `True` / `session` / `int8` | no |
|
|
|
|
**None of these are stringified.** The `hermes config set` stringification bug that caused this
|
|
incident has not recurred for this block.
|
|
|
|
**One drift to note:** `skip_contexts` now includes `cron`, which was not in the original repair
|
|
list. Mnemosyne capture is therefore **deliberately skipped in cron contexts** — scheduled jobs
|
|
produce no working memories by design. That explains why cron-driven runs of this wiki job leave
|
|
no Mnemosyne trace.
|
|
|
|
**Status: `resolved`.** The back-up copies named in the repair section still exist:
|
|
`~/.hermes/backups/mnemosyne-capture-fix-20260801_180249/` and
|
|
`~/.hermes/backups/legacy-memory-slim-20260801_180702/`.
|
|
|
|
> **Resolution re-confirmed 2026-11-11.** The whole `memory.mnemosyne` block in
|
|
> `~/.hermes/config.yaml` is still natively typed — `sync_roles: [user]` (list),
|
|
> `skip_contexts: [cron, flush, subagent, background, skill_loop]` (list),
|
|
> `reflect: {disabled_for_cron: true, max_calls_per_session: 3}` (dict), `sleep_threshold: 50`
|
|
> (int), `ignore_patterns: []` (list), `profile_isolation: true` (bool), `auto_sleep: true`,
|
|
> `default_scope: session`, `vector_type: int8`. No stringification regression. Both backup
|
|
> directories named in the repair still exist. `status: resolved` stands.
|
|
|
|
## Related
|
|
- [[systems/mnemosyne]]
|
|
- [[systems/memory-backend]]
|
|
- [[incidents/2026-07-29-mnemosyne-relay]]
|