Files
agent-estate-wiki/incidents/2026-07-29-mnemosyne-relay.md
T
Hermes ab3652aa20 Stocktake 2026-11-12: tailscale live-vs-page drift corrected; sftpgo incident re-verified; pre-dispatch-filter signature found recurring on a second job/pin
- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json.
  mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping
  (CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online.
  omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout).
  9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge
  (running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table
  unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed.
- incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer
  Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared.
- incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on
  cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved
  2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded.
- Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits),
  work-outlook (himalaya still login not supported), omniroute-free-deepseek
  (good-free/free-stack both HTTP 200, non-deepseek resolution).
- Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md.
- wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
2026-11-12 03:19:12 +08:00

70 lines
3.3 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: 'Mnemosyne Relay Scope Corruption + `tools: None` — 2026-07-29'
type: incident
status: resolved
created: 2026-07-29
updated: 2026-11-12
verified_on: 2026-11-12
confidence: high
tags: [mnemosyne, config, relay, incident, resolved]
sources: [live-probe, config, session-20260729]
---
# mnemosyne-tools-none — 2026-07-29
## FIXED: `memory.mnemosyne.tools: None` in config.yaml
**Root cause:** `~/.hermes/config.yaml` had `tools: None` under the mnemosyne section (line 183). The memory manager interprets `None` as a literal tool name filter and fails schema loading with:
```
WARNING: Memory provider 'mnemosyne' get_tool_schemas() failed: Unknown Mnemosyne tool(s) in memory.mnemosyne.tools: None.
```
Fallback: the next line registers all 40 tools anyway — no data loss, just noise.
**Fix applied:** Removed the `tools: None` line. Config now defaults to all tools (correct behaviour).
**Timeline:** 86 occurrences logged between Jul 28–29, 19 unique sessions affected.
---
# relay-scope-corruption — 2026-07-29
## SYMPTOM
```
WARNING agent.relay_runtime: Hermes Relay turn finalization failed
RuntimeError: invalid argument: scope handle is not at the top of the stack
File relay_runtime.py:644 end_turn → run_in_session → invoke
File nemo_relay/scope.py:144 pop()
```
## DETAILS
- Hermes Agent 0.19.0, nemo_relay 0.6.0
- 3 occurrences across 3 sessions within 33 min (07:37–08:10 UTC 2026-07-29)
- Sessions: `20260723_024537_95c0e6f7`, `20260728_233758_0c6b3c0d`, `20260729_070054_b8452262`
- Exception is caught (WARNING, not ERROR). Turn finalization may be incomplete.
- No recurrence after 08:10.
## LIKELY CAUSE
Scope push/pop ordering mismatch. Something pushes a scope that isn't at the top when `end_turn` tries to pop it. Could be a recovery code path (tool failure → cleanup) pushing scopes in wrong order, or a subagent spawning pattern leaking a scope.
## REPORTED
Filed (or should be filed) under NousResearch/hermes-agent issues. No token available to auto-create.
## MONITORING
Watch for recurrence. If persists, check whether the affected sessions share a common tool call pattern (subagent delegation, tool failure, etc.).
## Re-verified 2026-11-12 03:3x AWST (stocktake — page was 19d old, inside the staleness window)
Both halves of this page re-checked against live state and **the fix holds; neither symptom has recurred**
in ~3.5 months and 106 days of logs:
- `grep -n "tools:" ~/.hermes/config.yaml` → the `memory.mnemosyne` block no longer carries `tools: None`
(the literal `None` key is absent), so the `Unknown Mnemosyne tool(s)… None` schema warning has no
trigger left.
- `grep -c "Unknown Mnemosyne tool" ~/.hermes/logs/gateway.log` and `.log.1`–`.log.3` → **0 in every
rotated file** (4 generations, 0 occurrences).
- `grep -ah "scope handle is not at the top" ~/.hermes/logs/gateway.log*` → **no occurrences**; the
relay-scope corruption has not repeated since 2026-07-29 08:10 UTC.
- Mnemosyne itself is healthy on the current build: `mnemosyne stats` → **2782 memories (1062 working /
1217 episodic)** and `mnemosyne recall "tailscale serve routes"` returns substantive ranked results, so
the relay/memory path is functioning.
The `REPORTED` action above still stands (no token to auto-create a NousResearch issue); the **MONITORING**
condition is now satisfied — no recurrence across four log generations.