Files
agent-estate-wiki/incidents/2026-07-29-mnemosyne-relay.md
T
Hermes ab3652aa20 Stocktake 2026-11-12: tailscale live-vs-page drift corrected; sftpgo incident re-verified; pre-dispatch-filter signature found recurring on a second job/pin
- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json.
  mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping
  (CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online.
  omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout).
  9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge
  (running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table
  unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed.
- incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer
  Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared.
- incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on
  cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved
  2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded.
- Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits),
  work-outlook (himalaya still login not supported), omniroute-free-deepseek
  (good-free/free-stack both HTTP 200, non-deepseek resolution).
- Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md.
- wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
2026-11-12 03:19:12 +08:00

3.3 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Mnemosyne Relay Scope Corruption + `tools: None` — 2026-07-29 incident resolved 2026-07-29 2026-11-12 2026-11-12 high
mnemosyne
config
relay
incident
resolved
live-probe
config
session-20260729

mnemosyne-tools-none — 2026-07-29

FIXED: memory.mnemosyne.tools: None in config.yaml

Root cause: ~/.hermes/config.yaml had tools: None under the mnemosyne section (line 183). The memory manager interprets None as a literal tool name filter and fails schema loading with:

WARNING: Memory provider 'mnemosyne' get_tool_schemas() failed: Unknown Mnemosyne tool(s) in memory.mnemosyne.tools: None.

Fallback: the next line registers all 40 tools anyway — no data loss, just noise.

Fix applied: Removed the tools: None line. Config now defaults to all tools (correct behaviour).

Timeline: 86 occurrences logged between Jul 28–29, 19 unique sessions affected.


relay-scope-corruption — 2026-07-29

SYMPTOM

WARNING agent.relay_runtime: Hermes Relay turn finalization failed
RuntimeError: invalid argument: scope handle is not at the top of the stack
  File relay_runtime.py:644  end_turn → run_in_session → invoke
  File nemo_relay/scope.py:144  pop()

DETAILS

  • Hermes Agent 0.19.0, nemo_relay 0.6.0
  • 3 occurrences across 3 sessions within 33 min (07:37–08:10 UTC 2026-07-29)
  • Sessions: 20260723_024537_95c0e6f7, 20260728_233758_0c6b3c0d, 20260729_070054_b8452262
  • Exception is caught (WARNING, not ERROR). Turn finalization may be incomplete.
  • No recurrence after 08:10.

LIKELY CAUSE

Scope push/pop ordering mismatch. Something pushes a scope that isn't at the top when end_turn tries to pop it. Could be a recovery code path (tool failure → cleanup) pushing scopes in wrong order, or a subagent spawning pattern leaking a scope.

REPORTED

Filed (or should be filed) under NousResearch/hermes-agent issues. No token available to auto-create.

MONITORING

Watch for recurrence. If persists, check whether the affected sessions share a common tool call pattern (subagent delegation, tool failure, etc.).

Re-verified 2026-11-12 03:3x AWST (stocktake — page was 19d old, inside the staleness window)

Both halves of this page re-checked against live state and the fix holds; neither symptom has recurred in ~3.5 months and 106 days of logs:

  • grep -n "tools:" ~/.hermes/config.yaml → the memory.mnemosyne block no longer carries tools: None (the literal None key is absent), so the Unknown Mnemosyne tool(s)… None schema warning has no trigger left.
  • grep -c "Unknown Mnemosyne tool" ~/.hermes/logs/gateway.log and .log.1–.log.3 → 0 in every rotated file (4 generations, 0 occurrences).
  • grep -ah "scope handle is not at the top" ~/.hermes/logs/gateway.log* → no occurrences; the relay-scope corruption has not repeated since 2026-07-29 08:10 UTC.
  • Mnemosyne itself is healthy on the current build: mnemosyne stats → 2782 memories (1062 working / 1217 episodic) and mnemosyne recall "tailscale serve routes" returns substantive ranked results, so the relay/memory path is functioning.

The REPORTED action above still stands (no token to auto-create a NousResearch issue); the MONITORING condition is now satisfied — no recurrence across four log generations.