audit 2026-10-24: conduit page + memory-start-timeout incident; 10-09 storm resolved; window 169/0; frontmatter for 2 legacy incident pages

This commit is contained in:
Tony0410
2026-10-24 00:24:44 +08:00
parent 250c73530a
commit 573a4a2434
10 changed files with 384 additions and 23 deletions
+13
View File
@@ -1,3 +1,16 @@
---
title: Mnemosyne Relay Scope Corruption + `tools: None` — 2026-07-29
type: incident
status: resolved
created: 2026-07-29
updated: 2026-10-24
verified_on: 2026-10-24
review_after: 2026-11-07
confidence: high
tags: [mnemosyne, config, relay, incident, resolved]
sources: [live-probe, config, session-20260729]
---
# mnemosyne-tools-none — 2026-07-29
## FIXED: `memory.mnemosyne.tools: None` in config.yaml
@@ -1,3 +1,16 @@
---
title: OmniRoute free-DeepSeek Silent Stops (Empty-After-Tool) — 2026-08-04
type: incident
status: monitoring
created: 2026-08-04
updated: 2026-10-24
verified_on: 2026-10-24
review_after: 2026-11-07
confidence: high
tags: [omniroute, deepseek, free-tier, silent-failure, incident, mitigated]
sources: [live-probe, omniroute-log, session-20260804]
---
# [2026-08-04] OmniRoute free-DeepSeek silent stops (empty-after-tool)
## Status
@@ -1,22 +1,23 @@
---
title: Fleet-Wide Provider Timeout Storm — 2026-10-09
type: incident
status: unresolved
status: resolved
created: 2026-10-09
updated: 2026-10-13
verified_on: 2026-10-13
review_after: 2026-10-16
updated: 2026-10-24
verified_on: 2026-10-24
review_after: 2026-11-07
confidence: high
tags: [cron, omniroute, router, timeout, incident, unresolved]
tags: [cron, omniroute, router, timeout, incident, resolved]
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
---
# Fleet-Wide Provider Timeout Storm — 2026-10-09
**Window:** 2026-10-09 00:00 → 15:00 AWST (last failure 15:00; every run from 18:00 onward completed)
**Status:** Unresolved — the storm ended and **11 of its 12 incidents are now closed**; only
`1b23c0_c35703f637b8` remains open, and it cannot clear before that job's next scheduled slot
(**Fri 2026-10-16 08:00**). Nothing here is still symptomatic.
**Status:** Resolved — **all 12 of its incident rows are closed.** The last hold-outs
(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job)
closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again.
The storm contributes **no** unclosed rows. Nothing here is still symptomatic.
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
## What happened
@@ -0,0 +1,97 @@
---
title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
type: incident
status: resolved
created: 2026-10-24
updated: 2026-10-24
verified_on: 2026-10-24
review_after: 2026-11-07
confidence: high
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
---
# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
**Status:** Resolved 2026-10-23 18:54 (patched and verified)
**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's
memory layer.
## Symptom
Three identical warnings in `~/.hermes/logs/agent.log` (also in `errors.log`), one per call attempt
that hit a cold memory provider:
```
2026-10-22 11:28:17,137 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
2026-10-22 13:16:37,867 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
2026-10-23 18:29:16,830 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
2026-10-22 13:16:37,875 WARNING hermes_dashboard_plugin_conduit_push: Memory context for Conduit failed: Memory start timed out
```
## Root cause
`dashboard/plugin_api.py` in the `conduit_push` plugin hard-coded `MEMORY_TIMEOUT_S = 10.0` — a
budget for **one request including the provider start**. Mnemosyne's cold start for the
`conduit-voice` session takes roughly **10–45 s**, so any call landing on a cold provider blew the
budget. The plugin then ran the call **without memory context**, and the failure was logged as
`Memory start timed out` — the agent speaks to the user with no recall of prior sessions and no
error surfaced to the caller.
This is not an OmniRoute/model fault, and not a Mnemosyne data fault: the store was healthy and
the provider simply needed longer than the plugin allowed. → [[systems/mnemosyne]]
## Fix applied (2026-10-23, live before 18:54:43)
A single-file change to `~/.hermes/plugins/conduit_push/dashboard/plugin_api.py`, **still
uncommitted in the working tree**:
```diff
-MEMORY_TIMEOUT_S = 10.0
+# Mnemosyne (the memory provider) can take ~10-45s to cold-start for the
+# conduit-voice session, so give it enough room; tunable via env.
+MEMORY_TIMEOUT_S = float(os.environ.get("CONDUIT_MEMORY_TIMEOUT_S", "60.0"))
```
and a hard-failure guard so a token/start error degrades instead of aborting:
```diff
- entry = _memory_providers.get(key, name, deadline)
+ try:
+ entry = _memory_providers.get(key, name, deadline)
+ except TokenError:
+ logger.warning("Memory provider %r failed to start for Conduit; proceeding without recall", name)
+ entry = None
```
The new default is used by every deadline in the memory layer (6 call sites) plus the
`+ 2.0 s` backstop around each provider call. `CONDUIT_MEMORY_TIMEOUT_S` overrides it without
touching the file.
## Verification
- Dashboard process on `:9119` restarted **2026-10-23 18:54:43** — after the 18:29 abort and
therefore running the patched code.
- **Zero** `abandoning the call` / `Memory context for Conduit failed` lines exist after that
restart (`grep` across `~/.hermes/logs/*.log`, deduped).
- Two Conduit call sessions closed cleanly after the restart — **2026-10-24 00:03:23** and
**00:06:57**, both `outcome: done`.
- **Caveat, stated plainly:** the plugin logs only *failures*, so the absence of a warning is
not positive proof that recall was populated on those calls. The proven part is that the call
is no longer abandoned by the memory layer.
## Follow-ups
- **The patch is uncommitted.** A plugin update that resets
`~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream
(`kaishi00/hermes-conduit-notifier`).
- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised.
Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]]
## Related
- [[systems/conduit]]
- [[systems/mnemosyne]]
- [[incidents/2026-10-09-memory-recall-relevance]]