Files
agent-estate-wiki/incidents/2026-10-22-conduit-memory-start-timeout.md
T

7.0 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout) incident degraded 2026-10-24 2026-11-15 2026-11-15 2026-12-15 high
conduit
mnemosyne
timeout
plugin
voice
incident
resolved
live-log
plugin-tree
live-sqlite-conduit
git-diff

Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23

Window: 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences) Status: Resolved 2026-10-23 18:54 (patched and verified) — but the fix was lost in the plugin's 0.168.x restructure; status: degraded since 2026-11-15 (see the regression section below) Impact: A Conduit voice call that arrived before the Mnemosyne provider was warm lost its memory context — and, in the pre-fix code path, the call itself was abandoned by the plugin's memory layer.

Symptom

Three identical warnings in ~/.hermes/logs/agent.log (also in errors.log), one per call attempt that hit a cold memory provider:

2026-10-22 11:28:17,137 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
2026-10-22 13:16:37,867 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
2026-10-23 18:29:16,830 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
2026-10-22 13:16:37,875 WARNING hermes_dashboard_plugin_conduit_push: Memory context for Conduit failed: Memory start timed out

Root cause

dashboard/plugin_api.py in the conduit_push plugin hard-coded MEMORY_TIMEOUT_S = 10.0 — a budget for one request including the provider start. Mnemosyne's cold start for the conduit-voice session takes roughly 10–45 s, so any call landing on a cold provider blew the budget. The plugin then ran the call without memory context, and the failure was logged as Memory start timed out — the agent speaks to the user with no recall of prior sessions and no error surfaced to the caller.

This is not an OmniRoute/model fault, and not a Mnemosyne data fault: the store was healthy and the provider simply needed longer than the plugin allowed. → systems/mnemosyne

Fix applied (2026-10-23, live before 18:54:43)

A single-file change to ~/.hermes/plugins/conduit_push/dashboard/plugin_api.py, still uncommitted in the working tree:

-MEMORY_TIMEOUT_S = 10.0
+# Mnemosyne (the memory provider) can take ~10-45s to cold-start for the
+# conduit-voice session, so give it enough room; tunable via env.
+MEMORY_TIMEOUT_S = float(os.environ.get("CONDUIT_MEMORY_TIMEOUT_S", "60.0"))

and a hard-failure guard so a token/start error degrades instead of aborting:

-    entry = _memory_providers.get(key, name, deadline)
+    try:
+        entry = _memory_providers.get(key, name, deadline)
+    except TokenError:
+        logger.warning("Memory provider %r failed to start for Conduit; proceeding without recall", name)
+        entry = None

The new default is used by every deadline in the memory layer (6 call sites) plus the + 2.0 s backstop around each provider call. CONDUIT_MEMORY_TIMEOUT_S overrides it without touching the file.

Verification

  • Dashboard process on :9119 restarted 2026-10-23 18:54:43 — after the 18:29 abort and therefore running the patched code.
  • Zero abandoning the call / Memory context for Conduit failed lines exist after that restart (grep across ~/.hermes/logs/*.log, deduped).
  • Two Conduit call sessions closed cleanly after the restart — 2026-10-24 00:03:23 and 00:06:57, both outcome: done. Corrected 2026-10-25. Those two conduit-calls.json → ends entries are the cron sessions of that night (Mnemosyne Embedding Coverage Monitor 00:01, Health Monitor 00:04), not calls; ends records every turn-end that fires a watch (MAX_RECENT_ENDS = 64). history — real call records — is empty, so the fix has no positive call evidence and no call has been placed since it landed. The surviving evidence is the absence of abort lines after the 18:54:43 restart, and nothing has falsified it.
  • Caveat, stated plainly: the plugin logs only failures, so the absence of a warning is not positive proof that recall was populated on those calls. The proven part is that the call is no longer abandoned by the memory layer.

⚠ Regression — the fix was lost in the plugin restructure (verified 2026-11-15)

The follow-up below warned that a plugin update would revert the patch. It happened. Live state at the 2026-11-15 03:00 stocktake:

  • The plugin tree is clean at upstream 0.168.2 (296c94b, committed 2026-11-12 US / 2026-11-13 AWST) — the patch is gone entirely, not just uncommitted.
  • The memory layer moved: dashboard/plugin_api.py (now a doc index) → dashboard/api/memory.py. There, MEMORY_TIMEOUT_S = 10.0 is hard-coded again (memory.py:48), with no CONDUIT_MEMORY_TIMEOUT_S env override, and the "abandoning the call" behaviour is intact (memory.py:256). The restructure landed in memory.py on 2026-10-29 AWST, so the 60 s budget has been gone for ~2.5 weeks.
  • The running dashboard (started 2026-11-10) loads the restructured code, so the exposure is live.
  • One real call has since occurred — 2026-11-14 08:57 AWST, the first entry in conduit-calls.json → history — with no abort lines in the logs afterwards. But that call met a warm provider; it does not exercise the cold-start path. The page's original caveat stands.
  • Not re-applied by the stocktake: porting the fix to the restructured code is a code change to a git-managed upstream plugin that the next update will revert again — it needs Anthony's call (re-port + commit locally, or upstream it).

The original incident window (2026-10-22 → 23) remains resolved; status: degraded records that the protection is gone, not that the October fault returned.

Follow-ups

  • ⚠ The fix is still unexercised by a call … Superseded 2026-11-15: one call has since occurred (2026-11-14 08:57 AWST) — but the fix itself was lost in the restructure (section above), so "unexercised" is now moot.
  • Follow-on failure (2026-10-24): three requests to be phoned were refused by the agent, which asserted the capability did not exist without calling the tool → incidents/2026-10-24-conduit-call-request-refused.
  • The patch is uncommitted. A plugin update that resets ~/.hermes/plugins/conduit_push reverts it to the 10 s budget. Confirmed 2026-11-15 — see the regression section above.
  • The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised. Consumers with tight timeouts elsewhere will hit the same wall. → systems/mnemosyne