7.0 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout) | incident | degraded | 2026-10-24 | 2026-11-15 | 2026-11-15 | 2026-12-15 | high |
|
|
Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
Window: 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
Status: Resolved 2026-10-23 18:54 (patched and verified) — but the fix was lost in the
plugin's 0.168.x restructure; status: degraded since 2026-11-15 (see the regression section below)
Impact: A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
memory context — and, in the pre-fix code path, the call itself was abandoned by the plugin's
memory layer.
Symptom
Three identical warnings in ~/.hermes/logs/agent.log (also in errors.log), one per call attempt
that hit a cold memory provider:
2026-10-22 11:28:17,137 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
2026-10-22 13:16:37,867 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
2026-10-23 18:29:16,830 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
2026-10-22 13:16:37,875 WARNING hermes_dashboard_plugin_conduit_push: Memory context for Conduit failed: Memory start timed out
Root cause
dashboard/plugin_api.py in the conduit_push plugin hard-coded MEMORY_TIMEOUT_S = 10.0 — a
budget for one request including the provider start. Mnemosyne's cold start for the
conduit-voice session takes roughly 10–45 s, so any call landing on a cold provider blew the
budget. The plugin then ran the call without memory context, and the failure was logged as
Memory start timed out — the agent speaks to the user with no recall of prior sessions and no
error surfaced to the caller.
This is not an OmniRoute/model fault, and not a Mnemosyne data fault: the store was healthy and the provider simply needed longer than the plugin allowed. → systems/mnemosyne
Fix applied (2026-10-23, live before 18:54:43)
A single-file change to ~/.hermes/plugins/conduit_push/dashboard/plugin_api.py, still
uncommitted in the working tree:
-MEMORY_TIMEOUT_S = 10.0
+# Mnemosyne (the memory provider) can take ~10-45s to cold-start for the
+# conduit-voice session, so give it enough room; tunable via env.
+MEMORY_TIMEOUT_S = float(os.environ.get("CONDUIT_MEMORY_TIMEOUT_S", "60.0"))
and a hard-failure guard so a token/start error degrades instead of aborting:
- entry = _memory_providers.get(key, name, deadline)
+ try:
+ entry = _memory_providers.get(key, name, deadline)
+ except TokenError:
+ logger.warning("Memory provider %r failed to start for Conduit; proceeding without recall", name)
+ entry = None
The new default is used by every deadline in the memory layer (6 call sites) plus the
+ 2.0 s backstop around each provider call. CONDUIT_MEMORY_TIMEOUT_S overrides it without
touching the file.
Verification
- Dashboard process on
:9119restarted 2026-10-23 18:54:43 — after the 18:29 abort and therefore running the patched code. - Zero
abandoning the call/Memory context for Conduit failedlines exist after that restart (grepacross~/.hermes/logs/*.log, deduped). Two Conduit call sessions closed cleanly after the restart — 2026-10-24 00:03:23 and 00:06:57, bothCorrected 2026-10-25. Those twooutcome: done.conduit-calls.json → endsentries are the cron sessions of that night (Mnemosyne Embedding Coverage Monitor 00:01, Health Monitor 00:04), not calls;endsrecords every turn-end that fires a watch (MAX_RECENT_ENDS = 64).history— real call records — is empty, so the fix has no positive call evidence and no call has been placed since it landed. The surviving evidence is the absence of abort lines after the 18:54:43 restart, and nothing has falsified it.- Caveat, stated plainly: the plugin logs only failures, so the absence of a warning is not positive proof that recall was populated on those calls. The proven part is that the call is no longer abandoned by the memory layer.
⚠ Regression — the fix was lost in the plugin restructure (verified 2026-11-15)
The follow-up below warned that a plugin update would revert the patch. It happened. Live state at the 2026-11-15 03:00 stocktake:
- The plugin tree is clean at upstream
0.168.2(296c94b, committed 2026-11-12 US / 2026-11-13 AWST) — the patch is gone entirely, not just uncommitted. - The memory layer moved:
dashboard/plugin_api.py(now a doc index) →dashboard/api/memory.py. There,MEMORY_TIMEOUT_S = 10.0is hard-coded again (memory.py:48), with noCONDUIT_MEMORY_TIMEOUT_Senv override, and the "abandoning the call" behaviour is intact (memory.py:256). The restructure landed inmemory.pyon 2026-10-29 AWST, so the 60 s budget has been gone for ~2.5 weeks. - The running dashboard (started 2026-11-10) loads the restructured code, so the exposure is live.
- One real call has since occurred — 2026-11-14 08:57 AWST, the first entry in
conduit-calls.json → history— with no abort lines in the logs afterwards. But that call met a warm provider; it does not exercise the cold-start path. The page's original caveat stands. - Not re-applied by the stocktake: porting the fix to the restructured code is a code change to a git-managed upstream plugin that the next update will revert again — it needs Anthony's call (re-port + commit locally, or upstream it).
The original incident window (2026-10-22 → 23) remains resolved; status: degraded records that
the protection is gone, not that the October fault returned.
Follow-ups
⚠ The fix is still unexercised by a call …Superseded 2026-11-15: one call has since occurred (2026-11-14 08:57 AWST) — but the fix itself was lost in the restructure (section above), so "unexercised" is now moot.- Follow-on failure (2026-10-24): three requests to be phoned were refused by the agent, which asserted the capability did not exist without calling the tool → incidents/2026-10-24-conduit-call-request-refused.
The patch is uncommitted. A plugin update that resetsConfirmed 2026-11-15 — see the regression section above.~/.hermes/plugins/conduit_pushreverts it to the 10 s budget.- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised. Consumers with tight timeouts elsewhere will hit the same wall. → systems/mnemosyne