audit 2026-10-24: conduit page + memory-start-timeout incident; 10-09 storm resolved; window 169/0; frontmatter for 2 legacy incident pages
This commit is contained in:
@@ -1,3 +1,16 @@
|
||||
---
|
||||
title: Mnemosyne Relay Scope Corruption + `tools: None` — 2026-07-29
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-07-29
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
confidence: high
|
||||
tags: [mnemosyne, config, relay, incident, resolved]
|
||||
sources: [live-probe, config, session-20260729]
|
||||
---
|
||||
|
||||
# mnemosyne-tools-none — 2026-07-29
|
||||
|
||||
## FIXED: `memory.mnemosyne.tools: None` in config.yaml
|
||||
|
||||
@@ -1,3 +1,16 @@
|
||||
---
|
||||
title: OmniRoute free-DeepSeek Silent Stops (Empty-After-Tool) — 2026-08-04
|
||||
type: incident
|
||||
status: monitoring
|
||||
created: 2026-08-04
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
confidence: high
|
||||
tags: [omniroute, deepseek, free-tier, silent-failure, incident, mitigated]
|
||||
sources: [live-probe, omniroute-log, session-20260804]
|
||||
---
|
||||
|
||||
# [2026-08-04] OmniRoute free-DeepSeek silent stops (empty-after-tool)
|
||||
|
||||
## Status
|
||||
|
||||
@@ -1,22 +1,23 @@
|
||||
---
|
||||
title: Fleet-Wide Provider Timeout Storm — 2026-10-09
|
||||
type: incident
|
||||
status: unresolved
|
||||
status: resolved
|
||||
created: 2026-10-09
|
||||
updated: 2026-10-13
|
||||
verified_on: 2026-10-13
|
||||
review_after: 2026-10-16
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
confidence: high
|
||||
tags: [cron, omniroute, router, timeout, incident, unresolved]
|
||||
tags: [cron, omniroute, router, timeout, incident, resolved]
|
||||
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
|
||||
---
|
||||
|
||||
# Fleet-Wide Provider Timeout Storm — 2026-10-09
|
||||
|
||||
**Window:** 2026-10-09 00:00 → 15:00 AWST (last failure 15:00; every run from 18:00 onward completed)
|
||||
**Status:** Unresolved — the storm ended and **11 of its 12 incidents are now closed**; only
|
||||
`1b23c0_c35703f637b8` remains open, and it cannot clear before that job's next scheduled slot
|
||||
(**Fri 2026-10-16 08:00**). Nothing here is still symptomatic.
|
||||
**Status:** Resolved — **all 12 of its incident rows are closed.** The last hold-outs
|
||||
(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job)
|
||||
closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again.
|
||||
The storm contributes **no** unclosed rows. Nothing here is still symptomatic.
|
||||
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
|
||||
|
||||
## What happened
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-24
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
confidence: high
|
||||
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
|
||||
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
|
||||
---
|
||||
|
||||
# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
|
||||
|
||||
**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
|
||||
**Status:** Resolved 2026-10-23 18:54 (patched and verified)
|
||||
**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
|
||||
memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's
|
||||
memory layer.
|
||||
|
||||
## Symptom
|
||||
|
||||
Three identical warnings in `~/.hermes/logs/agent.log` (also in `errors.log`), one per call attempt
|
||||
that hit a cold memory provider:
|
||||
|
||||
```
|
||||
2026-10-22 11:28:17,137 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
|
||||
2026-10-22 13:16:37,867 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
|
||||
2026-10-23 18:29:16,830 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
|
||||
2026-10-22 13:16:37,875 WARNING hermes_dashboard_plugin_conduit_push: Memory context for Conduit failed: Memory start timed out
|
||||
```
|
||||
|
||||
## Root cause
|
||||
|
||||
`dashboard/plugin_api.py` in the `conduit_push` plugin hard-coded `MEMORY_TIMEOUT_S = 10.0` — a
|
||||
budget for **one request including the provider start**. Mnemosyne's cold start for the
|
||||
`conduit-voice` session takes roughly **10–45 s**, so any call landing on a cold provider blew the
|
||||
budget. The plugin then ran the call **without memory context**, and the failure was logged as
|
||||
`Memory start timed out` — the agent speaks to the user with no recall of prior sessions and no
|
||||
error surfaced to the caller.
|
||||
|
||||
This is not an OmniRoute/model fault, and not a Mnemosyne data fault: the store was healthy and
|
||||
the provider simply needed longer than the plugin allowed. → [[systems/mnemosyne]]
|
||||
|
||||
## Fix applied (2026-10-23, live before 18:54:43)
|
||||
|
||||
A single-file change to `~/.hermes/plugins/conduit_push/dashboard/plugin_api.py`, **still
|
||||
uncommitted in the working tree**:
|
||||
|
||||
```diff
|
||||
-MEMORY_TIMEOUT_S = 10.0
|
||||
+# Mnemosyne (the memory provider) can take ~10-45s to cold-start for the
|
||||
+# conduit-voice session, so give it enough room; tunable via env.
|
||||
+MEMORY_TIMEOUT_S = float(os.environ.get("CONDUIT_MEMORY_TIMEOUT_S", "60.0"))
|
||||
```
|
||||
|
||||
and a hard-failure guard so a token/start error degrades instead of aborting:
|
||||
|
||||
```diff
|
||||
- entry = _memory_providers.get(key, name, deadline)
|
||||
+ try:
|
||||
+ entry = _memory_providers.get(key, name, deadline)
|
||||
+ except TokenError:
|
||||
+ logger.warning("Memory provider %r failed to start for Conduit; proceeding without recall", name)
|
||||
+ entry = None
|
||||
```
|
||||
|
||||
The new default is used by every deadline in the memory layer (6 call sites) plus the
|
||||
`+ 2.0 s` backstop around each provider call. `CONDUIT_MEMORY_TIMEOUT_S` overrides it without
|
||||
touching the file.
|
||||
|
||||
## Verification
|
||||
|
||||
- Dashboard process on `:9119` restarted **2026-10-23 18:54:43** — after the 18:29 abort and
|
||||
therefore running the patched code.
|
||||
- **Zero** `abandoning the call` / `Memory context for Conduit failed` lines exist after that
|
||||
restart (`grep` across `~/.hermes/logs/*.log`, deduped).
|
||||
- Two Conduit call sessions closed cleanly after the restart — **2026-10-24 00:03:23** and
|
||||
**00:06:57**, both `outcome: done`.
|
||||
- **Caveat, stated plainly:** the plugin logs only *failures*, so the absence of a warning is
|
||||
not positive proof that recall was populated on those calls. The proven part is that the call
|
||||
is no longer abandoned by the memory layer.
|
||||
|
||||
## Follow-ups
|
||||
|
||||
- **The patch is uncommitted.** A plugin update that resets
|
||||
`~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream
|
||||
(`kaishi00/hermes-conduit-notifier`).
|
||||
- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised.
|
||||
Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]]
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/conduit]]
|
||||
- [[systems/mnemosyne]]
|
||||
- [[incidents/2026-10-09-memory-recall-relevance]]
|
||||
Reference in New Issue
Block a user