Files
agent-estate-wiki/incidents/2026-10-22-conduit-memory-start-timeout.md
T

5.3 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout) incident resolved 2026-10-24 2026-10-25 2026-10-25 2026-11-08 high
conduit
mnemosyne
timeout
plugin
voice
incident
resolved
live-log
plugin-tree
live-sqlite-conduit
git-diff

Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23

Window: 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences) Status: Resolved 2026-10-23 18:54 (patched and verified) Impact: A Conduit voice call that arrived before the Mnemosyne provider was warm lost its memory context — and, in the pre-fix code path, the call itself was abandoned by the plugin's memory layer.

Symptom

Three identical warnings in ~/.hermes/logs/agent.log (also in errors.log), one per call attempt that hit a cold memory provider:

2026-10-22 11:28:17,137 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
2026-10-22 13:16:37,867 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
2026-10-23 18:29:16,830 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
2026-10-22 13:16:37,875 WARNING hermes_dashboard_plugin_conduit_push: Memory context for Conduit failed: Memory start timed out

Root cause

dashboard/plugin_api.py in the conduit_push plugin hard-coded MEMORY_TIMEOUT_S = 10.0 — a budget for one request including the provider start. Mnemosyne's cold start for the conduit-voice session takes roughly 10–45 s, so any call landing on a cold provider blew the budget. The plugin then ran the call without memory context, and the failure was logged as Memory start timed out — the agent speaks to the user with no recall of prior sessions and no error surfaced to the caller.

This is not an OmniRoute/model fault, and not a Mnemosyne data fault: the store was healthy and the provider simply needed longer than the plugin allowed. → systems/mnemosyne

Fix applied (2026-10-23, live before 18:54:43)

A single-file change to ~/.hermes/plugins/conduit_push/dashboard/plugin_api.py, still uncommitted in the working tree:

-MEMORY_TIMEOUT_S = 10.0
+# Mnemosyne (the memory provider) can take ~10-45s to cold-start for the
+# conduit-voice session, so give it enough room; tunable via env.
+MEMORY_TIMEOUT_S = float(os.environ.get("CONDUIT_MEMORY_TIMEOUT_S", "60.0"))

and a hard-failure guard so a token/start error degrades instead of aborting:

-    entry = _memory_providers.get(key, name, deadline)
+    try:
+        entry = _memory_providers.get(key, name, deadline)
+    except TokenError:
+        logger.warning("Memory provider %r failed to start for Conduit; proceeding without recall", name)
+        entry = None

The new default is used by every deadline in the memory layer (6 call sites) plus the + 2.0 s backstop around each provider call. CONDUIT_MEMORY_TIMEOUT_S overrides it without touching the file.

Verification

  • Dashboard process on :9119 restarted 2026-10-23 18:54:43 — after the 18:29 abort and therefore running the patched code.
  • Zero abandoning the call / Memory context for Conduit failed lines exist after that restart (grep across ~/.hermes/logs/*.log, deduped).
  • Two Conduit call sessions closed cleanly after the restart — 2026-10-24 00:03:23 and 00:06:57, both outcome: done. Corrected 2026-10-25. Those two conduit-calls.json → ends entries are the cron sessions of that night (Mnemosyne Embedding Coverage Monitor 00:01, Health Monitor 00:04), not calls; ends records every turn-end that fires a watch (MAX_RECENT_ENDS = 64). history — real call records — is empty, so the fix has no positive call evidence and no call has been placed since it landed. The surviving evidence is the absence of abort lines after the 18:54:43 restart, and nothing has falsified it.
  • Caveat, stated plainly: the plugin logs only failures, so the absence of a warning is not positive proof that recall was populated on those calls. The proven part is that the call is no longer abandoned by the memory layer.

Follow-ups

  • ⚠ The fix is still unexercised by a call (see the correction above): conduit-calls.json → history is empty at 2026-10-25 00:2x, and the plugin is now five releases behind HEAD (0.64.0 → 0.69.0).
  • Follow-on failure (2026-10-24): three requests to be phoned were refused by the agent, which asserted the capability did not exist without calling the tool → incidents/2026-10-24-conduit-call-request-refused.
  • The patch is uncommitted. A plugin update that resets ~/.hermes/plugins/conduit_push reverts it to the 10 s budget. Commit it or send it upstream (kaishi00/hermes-conduit-notifier).
  • The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised. Consumers with tight timeouts elsewhere will hit the same wall. → systems/mnemosyne