audit 2026-10-24: conduit page + memory-start-timeout incident; 10-09 storm resolved; window 169/0; frontmatter for 2 legacy incident pages
This commit is contained in:
@@ -0,0 +1,97 @@
|
||||
---
|
||||
title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-24
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
confidence: high
|
||||
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
|
||||
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
|
||||
---
|
||||
|
||||
# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
|
||||
|
||||
**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
|
||||
**Status:** Resolved 2026-10-23 18:54 (patched and verified)
|
||||
**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
|
||||
memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's
|
||||
memory layer.
|
||||
|
||||
## Symptom
|
||||
|
||||
Three identical warnings in `~/.hermes/logs/agent.log` (also in `errors.log`), one per call attempt
|
||||
that hit a cold memory provider:
|
||||
|
||||
```
|
||||
2026-10-22 11:28:17,137 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
|
||||
2026-10-22 13:16:37,867 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
|
||||
2026-10-23 18:29:16,830 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
|
||||
2026-10-22 13:16:37,875 WARNING hermes_dashboard_plugin_conduit_push: Memory context for Conduit failed: Memory start timed out
|
||||
```
|
||||
|
||||
## Root cause
|
||||
|
||||
`dashboard/plugin_api.py` in the `conduit_push` plugin hard-coded `MEMORY_TIMEOUT_S = 10.0` — a
|
||||
budget for **one request including the provider start**. Mnemosyne's cold start for the
|
||||
`conduit-voice` session takes roughly **10–45 s**, so any call landing on a cold provider blew the
|
||||
budget. The plugin then ran the call **without memory context**, and the failure was logged as
|
||||
`Memory start timed out` — the agent speaks to the user with no recall of prior sessions and no
|
||||
error surfaced to the caller.
|
||||
|
||||
This is not an OmniRoute/model fault, and not a Mnemosyne data fault: the store was healthy and
|
||||
the provider simply needed longer than the plugin allowed. → [[systems/mnemosyne]]
|
||||
|
||||
## Fix applied (2026-10-23, live before 18:54:43)
|
||||
|
||||
A single-file change to `~/.hermes/plugins/conduit_push/dashboard/plugin_api.py`, **still
|
||||
uncommitted in the working tree**:
|
||||
|
||||
```diff
|
||||
-MEMORY_TIMEOUT_S = 10.0
|
||||
+# Mnemosyne (the memory provider) can take ~10-45s to cold-start for the
|
||||
+# conduit-voice session, so give it enough room; tunable via env.
|
||||
+MEMORY_TIMEOUT_S = float(os.environ.get("CONDUIT_MEMORY_TIMEOUT_S", "60.0"))
|
||||
```
|
||||
|
||||
and a hard-failure guard so a token/start error degrades instead of aborting:
|
||||
|
||||
```diff
|
||||
- entry = _memory_providers.get(key, name, deadline)
|
||||
+ try:
|
||||
+ entry = _memory_providers.get(key, name, deadline)
|
||||
+ except TokenError:
|
||||
+ logger.warning("Memory provider %r failed to start for Conduit; proceeding without recall", name)
|
||||
+ entry = None
|
||||
```
|
||||
|
||||
The new default is used by every deadline in the memory layer (6 call sites) plus the
|
||||
`+ 2.0 s` backstop around each provider call. `CONDUIT_MEMORY_TIMEOUT_S` overrides it without
|
||||
touching the file.
|
||||
|
||||
## Verification
|
||||
|
||||
- Dashboard process on `:9119` restarted **2026-10-23 18:54:43** — after the 18:29 abort and
|
||||
therefore running the patched code.
|
||||
- **Zero** `abandoning the call` / `Memory context for Conduit failed` lines exist after that
|
||||
restart (`grep` across `~/.hermes/logs/*.log`, deduped).
|
||||
- Two Conduit call sessions closed cleanly after the restart — **2026-10-24 00:03:23** and
|
||||
**00:06:57**, both `outcome: done`.
|
||||
- **Caveat, stated plainly:** the plugin logs only *failures*, so the absence of a warning is
|
||||
not positive proof that recall was populated on those calls. The proven part is that the call
|
||||
is no longer abandoned by the memory layer.
|
||||
|
||||
## Follow-ups
|
||||
|
||||
- **The patch is uncommitted.** A plugin update that resets
|
||||
`~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream
|
||||
(`kaishi00/hermes-conduit-notifier`).
|
||||
- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised.
|
||||
Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]]
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/conduit]]
|
||||
- [[systems/mnemosyne]]
|
||||
- [[incidents/2026-10-09-memory-recall-relevance]]
|
||||
Reference in New Issue
Block a user