snapshot: preserve central wiki state 2026-08-15 (7 modified + 1 new file)
This commit is contained in:
Tony0410
@@ -0,0 +1,41 @@
|
||||
# mnemosyne-tools-none — 2026-07-29
|
||||
|
||||
## FIXED: `memory.mnemosyne.tools: None` in config.yaml
|
||||
|
||||
**Root cause:** `~/.hermes/config.yaml` had `tools: None` under the mnemosyne section (line 183). The memory manager interprets `None` as a literal tool name filter and fails schema loading with:
|
||||
```
|
||||
WARNING: Memory provider 'mnemosyne' get_tool_schemas() failed: Unknown Mnemosyne tool(s) in memory.mnemosyne.tools: None.
|
||||
```
|
||||
Fallback: the next line registers all 40 tools anyway — no data loss, just noise.
|
||||
|
||||
**Fix applied:** Removed the `tools: None` line. Config now defaults to all tools (correct behaviour).
|
||||
|
||||
**Timeline:** 86 occurrences logged between Jul 28–29, 19 unique sessions affected.
|
||||
|
||||
---
|
||||
|
||||
# relay-scope-corruption — 2026-07-29
|
||||
|
||||
## SYMPTOM
|
||||
```
|
||||
WARNING agent.relay_runtime: Hermes Relay turn finalization failed
|
||||
RuntimeError: invalid argument: scope handle is not at the top of the stack
|
||||
File relay_runtime.py:644 end_turn → run_in_session → invoke
|
||||
File nemo_relay/scope.py:144 pop()
|
||||
```
|
||||
|
||||
## DETAILS
|
||||
- Hermes Agent 0.19.0, nemo_relay 0.6.0
|
||||
- 3 occurrences across 3 sessions within 33 min (07:37–08:10 UTC 2026-07-29)
|
||||
- Sessions: `20260723_024537_95c0e6f7`, `20260728_233758_0c6b3c0d`, `20260729_070054_b8452262`
|
||||
- Exception is caught (WARNING, not ERROR). Turn finalization may be incomplete.
|
||||
- No recurrence after 08:10.
|
||||
|
||||
## LIKELY CAUSE
|
||||
Scope push/pop ordering mismatch. Something pushes a scope that isn't at the top when `end_turn` tries to pop it. Could be a recovery code path (tool failure → cleanup) pushing scopes in wrong order, or a subagent spawning pattern leaking a scope.
|
||||
|
||||
## REPORTED
|
||||
Filed (or should be filed) under NousResearch/hermes-agent issues. No token available to auto-create.
|
||||
|
||||
## MONITORING
|
||||
Watch for recurrence. If persists, check whether the affected sessions share a common tool call pattern (subagent delegation, tool failure, etc.).
|
||||
@@ -0,0 +1,59 @@
|
||||
---
|
||||
title: Mnemosyne Automatic Capture Failure
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-08-01
|
||||
updated: 2026-08-01
|
||||
verified_on: 2026-08-01
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, hermes, incident]
|
||||
sources: [live-config, mnemosyne-db, hermes-cli-e2e]
|
||||
---
|
||||
|
||||
# Mnemosyne Automatic Capture Failure — 2026-08-01
|
||||
|
||||
## Impact
|
||||
Ordinary Hermes conversations were not being written to Mnemosyne `working_memory`. Explicit memory tools, SQLite integrity, vectors, FTS indexes, consolidation, backups, and cron monitoring remained operational. The newest automatic/explicit working row before repair was approximately fourteen hours old despite active Hermes sessions.
|
||||
|
||||
The always-injected legacy files were also effectively full: `MEMORY.md` was 5,055 bytes against a 5,000-character limit and `USER.md` was 2,958 bytes against 3,000. Legacy writes had begun failing.
|
||||
|
||||
## Root Cause
|
||||
Structured values under `memory.mnemosyne` in `~/.hermes/config.yaml` had been serialized as strings instead of native YAML types. The critical value was:
|
||||
|
||||
```yaml
|
||||
sync_roles: "['user']"
|
||||
```
|
||||
|
||||
The provider interpreted this as the literal invalid role `['user']`, so turn synchronization completed without persisting a user row. Other malformed values included `skip_contexts`, `reflect`, `sleep_threshold`, and `ignore_patterns`.
|
||||
|
||||
## Repair
|
||||
- Backed up config to `/home/hermes/.hermes/backups/mnemosyne-capture-fix-20260801_180249/config.yaml.before`.
|
||||
- Rewrote structured Mnemosyne settings with native YAML types.
|
||||
- Set `profile_isolation: true` because the installation has multiple Hermes profiles.
|
||||
- Preserved `sync_roles: [user]` and the intended skipped contexts.
|
||||
- Ran `hermes config check` successfully.
|
||||
- Created authoritative Mnemosyne canonical slots for user preferences and key workflows, correcting stale light-theme, numbered-Obsidian, and CT460 route entries.
|
||||
- Backed up legacy files to `/home/hermes/.hermes/backups/legacy-memory-slim-20260801_180702/`.
|
||||
- Reduced legacy injection from 8,013 bytes to 1,080 bytes while retaining essential always-on rules.
|
||||
|
||||
## Verification
|
||||
A pre-fix CLI turn using marker `MNEMO_E2E_20260801_180113_27053` completed but produced no database row.
|
||||
|
||||
A post-fix CLI turn using marker `MNEMO_FIXED_20260801_180310_16583` wrote:
|
||||
|
||||
```text
|
||||
[USER] Reply with exactly OK. Memory capture test marker: MNEMO_FIXED_20260801_180310_16583
|
||||
```
|
||||
|
||||
The row was verified directly in `~/.hermes/mnemosyne/data/mnemosyne.db`. Final diagnostics reported zero failed checks, complete working-vector coverage, zero orphan embeddings, 1,482 working rows, and 463 episodic vectors.
|
||||
|
||||
## Remaining Activation Step
|
||||
The already-running Telegram gateway/provider instance was initialized before the config repair. Run `hermes gateway restart` from an external terminal to load the corrected typed configuration for messaging-platform turns. The gateway cannot safely restart itself from one of its own child tool calls.
|
||||
|
||||
## Monitoring Note
|
||||
The health script checks capture staleness only after 24 hours. It therefore remained green during this fourteen-hour active-capture outage. A future improvement should compare recent non-cron conversation activity with the newest `working_memory` row rather than lowering the threshold blindly and creating idle-period false positives.
|
||||
|
||||
## Related
|
||||
- [[systems/mnemosyne]]
|
||||
- [[systems/memory-backend]]
|
||||
- [[incidents/2026-07-29-mnemosyne-relay]]
|
||||
@@ -0,0 +1,44 @@
|
||||
# [2026-08-04] OmniRoute free-DeepSeek silent stops (empty-after-tool)
|
||||
|
||||
## Status
|
||||
Mitigated — combos reworked; upstream still degraded.
|
||||
|
||||
## Symptom
|
||||
On **both** Hermes instances (ops/Tusk + the other instance), turns using `deepseek-flash-free` (and `smart-route` via `good-free`) did one tool call, then went **silently** — no text, no error. "continue" → one more tool call → silent stop again.
|
||||
|
||||
## Root cause
|
||||
The free-DeepSeek upstreams behind the combos started returning **valid-but-empty HTTP 200s right after a tool_calls turn** (no content, no further tool call). Hermes can't distinguish "empty" from "model chose to stop" → ends turn silently. Began **2026-08-01 ~09:14 UTC**, still active. OmniRoute log markers:
|
||||
|
||||
```
|
||||
[STREAM] Empty assistant response after tool_calls completion (<provider>:<model>)
|
||||
[ProxyEgress] <provider>/<conn-hash> ... status=error
|
||||
```
|
||||
|
||||
Per-model evidence (counts in `storage.sqlite` era log):
|
||||
- `238d5978/deepseek-v4-flash` — 63× empty-after-tool (conn `d4b9fee0`, 189 proxy errors)
|
||||
- `238d5978/deepseek-v4-flash-0731` — 70× empty-after-tool
|
||||
- `eb5af0da/deepseek-v4-flash-thinking:free` (conn `f8169018`) — 504 proxy errors (combo #1 slot)
|
||||
- `nvidia` deepseek + `opencode-zen` deepseek-free — noisy, errors
|
||||
|
||||
Strategy trap: both combos used `priority`, which retries the **same** broken connection and never fails over on an empty 200.
|
||||
|
||||
## Change applied (2026-08-04, ~03:30 AWST)
|
||||
Edited `combos` in `/var/lib/omniroute/storage.sqlite` on CT600 (live DB, hot-reloads — no restart):
|
||||
|
||||
- **`deepseek-flash-free`** — strategy `priority` → `fill-first`; 8 → 7 models: healthy free first (`mimo-v2.5-free`, `laguna-s-2.1:free` ×2 providers, `step-3.7-flash`, `glm-5.2-cheap:free`), one deepseek (`deepseek-v4-flash-thinking:free`) demoted to last resort.
|
||||
- **`good-free`** — strategy `priority` → `fill-first`; 15 → 12 models. **Dropped** the proven culprits: `238d5978/deepseek-v4-flash`, `nvidia/deepseek-ai/deepseek-v4-pro`, `nvidia/deepseek-ai/deepseek-v4-flash`. `opencode/deepseek-v4-flash-free` kept as last resort.
|
||||
|
||||
Backup of both pre-change rows: `/tmp/combo_backup_20260804.json` on the Hermes gateway LXC.
|
||||
|
||||
## Verification
|
||||
- Live API (`/v1/combos`) reflects both changes immediately (hot reload).
|
||||
- 2-step tool-call round-trip through `deepseek-flash-free` returns proper final content.
|
||||
- Direct round-trips (stream + non-stream) through the OLD combo were also healthy — the failure needs the degraded upstream + real session, hence counts not repro.
|
||||
|
||||
## Follow-ups
|
||||
- Upstream free-DeepSeek tier still degraded; re-check counts in a few days and restore deepseek-first ordering if it recovers.
|
||||
- Consider `auto`+explorationRate for self-tuning if free tier stays flaky.
|
||||
- Hermes-side: empty-post-tool turns are nudge/retry/fallback handled, but still end silently once exhausted — possible future improvement: surface "(empty after tool calls)" more visibly.
|
||||
|
||||
## Refs
|
||||
- [[systems/omniroute]]
|
||||
Reference in New Issue
Block a user