Files
agent-estate-wiki/incidents/2026-10-22-conduit-memory-start-timeout.md

133 lines
7.0 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
type: incident
status: degraded
created: 2026-10-24
updated: 2026-11-15
verified_on: 2026-11-15
review_after: 2026-12-15
confidence: high
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
---
# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
**Status:** Resolved 2026-10-23 18:54 (patched and verified) — **but the fix was lost in the
plugin's 0.168.x restructure; `status: degraded` since 2026-11-15** (see the regression section below)
**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's
memory layer.
## Symptom
Three identical warnings in `~/.hermes/logs/agent.log` (also in `errors.log`), one per call attempt
that hit a cold memory provider:
```
2026-10-22 11:28:17,137 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
2026-10-22 13:16:37,867 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
2026-10-23 18:29:16,830 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
2026-10-22 13:16:37,875 WARNING hermes_dashboard_plugin_conduit_push: Memory context for Conduit failed: Memory start timed out
```
## Root cause
`dashboard/plugin_api.py` in the `conduit_push` plugin hard-coded `MEMORY_TIMEOUT_S = 10.0` — a
budget for **one request including the provider start**. Mnemosyne's cold start for the
`conduit-voice` session takes roughly **10–45 s**, so any call landing on a cold provider blew the
budget. The plugin then ran the call **without memory context**, and the failure was logged as
`Memory start timed out` — the agent speaks to the user with no recall of prior sessions and no
error surfaced to the caller.
This is not an OmniRoute/model fault, and not a Mnemosyne data fault: the store was healthy and
the provider simply needed longer than the plugin allowed. → [[systems/mnemosyne]]
## Fix applied (2026-10-23, live before 18:54:43)
A single-file change to `~/.hermes/plugins/conduit_push/dashboard/plugin_api.py`, **still
uncommitted in the working tree**:
```diff
-MEMORY_TIMEOUT_S = 10.0
+# Mnemosyne (the memory provider) can take ~10-45s to cold-start for the
+# conduit-voice session, so give it enough room; tunable via env.
+MEMORY_TIMEOUT_S = float(os.environ.get("CONDUIT_MEMORY_TIMEOUT_S", "60.0"))
```
and a hard-failure guard so a token/start error degrades instead of aborting:
```diff
- entry = _memory_providers.get(key, name, deadline)
+ try:
+ entry = _memory_providers.get(key, name, deadline)
+ except TokenError:
+ logger.warning("Memory provider %r failed to start for Conduit; proceeding without recall", name)
+ entry = None
```
The new default is used by every deadline in the memory layer (6 call sites) plus the
`+ 2.0 s` backstop around each provider call. `CONDUIT_MEMORY_TIMEOUT_S` overrides it without
touching the file.
## Verification
- Dashboard process on `:9119` restarted **2026-10-23 18:54:43** — after the 18:29 abort and
therefore running the patched code.
- **Zero** `abandoning the call` / `Memory context for Conduit failed` lines exist after that
restart (`grep` across `~/.hermes/logs/*.log`, deduped).
- ~~Two Conduit call sessions closed cleanly after the restart — **2026-10-24 00:03:23** and
**00:06:57**, both `outcome: done`.~~ **Corrected 2026-10-25.** Those two
`conduit-calls.json → ends` entries are the **cron** sessions of that night (Mnemosyne Embedding
Coverage Monitor 00:01, Health Monitor 00:04), not calls; `ends` records *every* turn-end that
fires a watch (`MAX_RECENT_ENDS = 64`). **`history` — real call records — is empty**, so the fix
has **no positive call evidence** and no call has been placed since it landed. The surviving
evidence is the absence of abort lines after the 18:54:43 restart, and nothing has falsified it.
- **Caveat, stated plainly:** the plugin logs only *failures*, so the absence of a warning is
not positive proof that recall was populated on those calls. The proven part is that the call
is no longer abandoned by the memory layer.
## ⚠ Regression — the fix was lost in the plugin restructure (verified 2026-11-15)
The follow-up below warned that a plugin update would revert the patch. **It happened.** Live
state at the 2026-11-15 03:00 stocktake:
- The plugin tree is **clean at upstream `0.168.2`** (`296c94b`, committed 2026-11-12 US /
2026-11-13 AWST) — the patch is gone entirely, not just uncommitted.
- The memory layer **moved**: `dashboard/plugin_api.py` (now a doc index) →
`dashboard/api/memory.py`. There, `MEMORY_TIMEOUT_S = 10.0` is hard-coded again
(`memory.py:48`), with **no** `CONDUIT_MEMORY_TIMEOUT_S` env override, and the "abandoning the
call" behaviour is intact (`memory.py:256`). The restructure landed in `memory.py` on
**2026-10-29 AWST**, so the 60 s budget has been gone for ~2.5 weeks.
- The running dashboard (started 2026-11-10) loads the restructured code, so the exposure is live.
- One real call has since occurred — 2026-11-14 08:57 AWST, the first entry in
`conduit-calls.json → history` — with no abort lines in the logs afterwards. But that call met a
warm provider; it does not exercise the cold-start path. The page's original caveat stands.
- **Not re-applied by the stocktake**: porting the fix to the restructured code is a code change to
a git-managed upstream plugin that the next update will revert again — it needs Anthony's call
(re-port + commit locally, or upstream it).
The original incident window (2026-10-22 → 23) remains resolved; `status: degraded` records that
the *protection* is gone, not that the October fault returned.
## Follow-ups
- ~~⚠ **The fix is still unexercised by a call** …~~ **Superseded 2026-11-15**: one call has since
occurred (2026-11-14 08:57 AWST) — but the fix itself was lost in the restructure (section above),
so "unexercised" is now moot.
- **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which
asserted the capability did not exist **without calling the tool** →
[[incidents/2026-10-24-conduit-call-request-refused]].
- ~~**The patch is uncommitted.** A plugin update that resets `~/.hermes/plugins/conduit_push`
reverts it to the 10 s budget.~~ **Confirmed 2026-11-15** — see the regression section above.
- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised.
Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]]
## Related
- [[incidents/2026-10-24-conduit-call-request-refused]]
- [[systems/conduit]]
- [[systems/mnemosyne]]
- [[incidents/2026-10-09-memory-recall-relevance]]