- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json. mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping (CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online. omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout). 9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge (running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed. - incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared. - incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved 2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded. - Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits), work-outlook (himalaya still login not supported), omniroute-free-deepseek (good-free/free-stack both HTTP 200, non-deepseek resolution). - Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md. - wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
70 lines
3.3 KiB
Markdown
70 lines
3.3 KiB
Markdown
---
|
||
title: 'Mnemosyne Relay Scope Corruption + `tools: None` — 2026-07-29'
|
||
type: incident
|
||
status: resolved
|
||
created: 2026-07-29
|
||
updated: 2026-11-12
|
||
verified_on: 2026-11-12
|
||
confidence: high
|
||
tags: [mnemosyne, config, relay, incident, resolved]
|
||
sources: [live-probe, config, session-20260729]
|
||
---
|
||
|
||
# mnemosyne-tools-none — 2026-07-29
|
||
|
||
## FIXED: `memory.mnemosyne.tools: None` in config.yaml
|
||
|
||
**Root cause:** `~/.hermes/config.yaml` had `tools: None` under the mnemosyne section (line 183). The memory manager interprets `None` as a literal tool name filter and fails schema loading with:
|
||
```
|
||
WARNING: Memory provider 'mnemosyne' get_tool_schemas() failed: Unknown Mnemosyne tool(s) in memory.mnemosyne.tools: None.
|
||
```
|
||
Fallback: the next line registers all 40 tools anyway — no data loss, just noise.
|
||
|
||
**Fix applied:** Removed the `tools: None` line. Config now defaults to all tools (correct behaviour).
|
||
|
||
**Timeline:** 86 occurrences logged between Jul 28–29, 19 unique sessions affected.
|
||
|
||
---
|
||
|
||
# relay-scope-corruption — 2026-07-29
|
||
|
||
## SYMPTOM
|
||
```
|
||
WARNING agent.relay_runtime: Hermes Relay turn finalization failed
|
||
RuntimeError: invalid argument: scope handle is not at the top of the stack
|
||
File relay_runtime.py:644 end_turn → run_in_session → invoke
|
||
File nemo_relay/scope.py:144 pop()
|
||
```
|
||
|
||
## DETAILS
|
||
- Hermes Agent 0.19.0, nemo_relay 0.6.0
|
||
- 3 occurrences across 3 sessions within 33 min (07:37–08:10 UTC 2026-07-29)
|
||
- Sessions: `20260723_024537_95c0e6f7`, `20260728_233758_0c6b3c0d`, `20260729_070054_b8452262`
|
||
- Exception is caught (WARNING, not ERROR). Turn finalization may be incomplete.
|
||
- No recurrence after 08:10.
|
||
|
||
## LIKELY CAUSE
|
||
Scope push/pop ordering mismatch. Something pushes a scope that isn't at the top when `end_turn` tries to pop it. Could be a recovery code path (tool failure → cleanup) pushing scopes in wrong order, or a subagent spawning pattern leaking a scope.
|
||
|
||
## REPORTED
|
||
Filed (or should be filed) under NousResearch/hermes-agent issues. No token available to auto-create.
|
||
|
||
## MONITORING
|
||
Watch for recurrence. If persists, check whether the affected sessions share a common tool call pattern (subagent delegation, tool failure, etc.).
|
||
|
||
## Re-verified 2026-11-12 03:3x AWST (stocktake — page was 19d old, inside the staleness window)
|
||
Both halves of this page re-checked against live state and **the fix holds; neither symptom has recurred**
|
||
in ~3.5 months and 106 days of logs:
|
||
- `grep -n "tools:" ~/.hermes/config.yaml` → the `memory.mnemosyne` block no longer carries `tools: None`
|
||
(the literal `None` key is absent), so the `Unknown Mnemosyne tool(s)… None` schema warning has no
|
||
trigger left.
|
||
- `grep -c "Unknown Mnemosyne tool" ~/.hermes/logs/gateway.log` and `.log.1`–`.log.3` → **0 in every
|
||
rotated file** (4 generations, 0 occurrences).
|
||
- `grep -ah "scope handle is not at the top" ~/.hermes/logs/gateway.log*` → **no occurrences**; the
|
||
relay-scope corruption has not repeated since 2026-07-29 08:10 UTC.
|
||
- Mnemosyne itself is healthy on the current build: `mnemosyne stats` → **2782 memories (1062 working /
|
||
1217 episodic)** and `mnemosyne recall "tailscale serve routes"` returns substantive ranked results, so
|
||
the relay/memory path is functioning.
|
||
|
||
The `REPORTED` action above still stands (no token to auto-create a NousResearch issue); the **MONITORING**
|
||
condition is now satisfied — no recurrence across four log generations. |