stocktake(2026-11-15): skills-index regenerated (299/67); storm incident re-verified (all 12 rows resolved, evidence-window note); conduit memory-timeout fix LOST in plugin 0.168.x restructure — page → degraded with regression section; 0 stale after fixes

This commit is contained in:
Tony0410
2026-11-15 03:25:18 +08:00
parent 646be98c31
commit 402d8546e8
6 changed files with 233 additions and 23 deletions
@@ -3,9 +3,9 @@ title: Fleet-Wide Provider Timeout Storm — 2026-10-09
type: incident
status: resolved
created: 2026-10-09
updated: 2026-10-24
verified_on: 2026-10-24
review_after: 2026-11-07
updated: 2026-11-15
verified_on: 2026-11-15
review_after: 2026-12-15
confidence: high
tags: [cron, omniroute, router, timeout, incident, resolved]
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
@@ -18,6 +18,15 @@ sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-2026100
(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job)
closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again.
The storm contributes **no** unclosed rows. Nothing here is still symptomatic.
> **Re-verified 2026-11-15** (3 a.m. stocktake): all 12 of the storm's `cron_incidents` rows still read
> `resolved` — none reopened. `1b23c042e17d` (the weekly Friday job, last hold-out) is healthy —
> `last_status: ok`, last run 2026-11-13 08:11 AWST. One incident is currently open fleet-wide
> (`ea79a2_601037516abd`, Mnemosyne Health Monitor, first seen 2026-11-15 00:03) but it is the
> capture-stall detection firing → [[incidents/2026-11-14-mnemosyne-capture-stall]], unrelated to
> this storm. ⚠ Evidence window: `executions.db` retention now reaches back only to 2026-11-09, so
> the 2026-10-09 execution counts below are **no longer re-derivable from the DB** — they stand on
> the 2026-10-24 read (the incident rows themselves persist, which is what this check re-read).
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
## What happened
@@ -1,11 +1,11 @@
---
title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
type: incident
status: resolved
status: degraded
created: 2026-10-24
updated: 2026-10-25
verified_on: 2026-10-25
review_after: 2026-11-08
updated: 2026-11-15
verified_on: 2026-11-15
review_after: 2026-12-15
confidence: high
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
@@ -14,7 +14,8 @@ sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
**Status:** Resolved 2026-10-23 18:54 (patched and verified)
**Status:** Resolved 2026-10-23 18:54 (patched and verified) — **but the fix was lost in the
plugin's 0.168.x restructure; `status: degraded` since 2026-11-15** (see the regression section below)
**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's
memory layer.
@@ -87,17 +88,39 @@ touching the file.
not positive proof that recall was populated on those calls. The proven part is that the call
is no longer abandoned by the memory layer.
## ⚠ Regression — the fix was lost in the plugin restructure (verified 2026-11-15)
The follow-up below warned that a plugin update would revert the patch. **It happened.** Live
state at the 2026-11-15 03:00 stocktake:
- The plugin tree is **clean at upstream `0.168.2`** (`296c94b`, committed 2026-11-12 US /
2026-11-13 AWST) — the patch is gone entirely, not just uncommitted.
- The memory layer **moved**: `dashboard/plugin_api.py` (now a doc index) →
`dashboard/api/memory.py`. There, `MEMORY_TIMEOUT_S = 10.0` is hard-coded again
(`memory.py:48`), with **no** `CONDUIT_MEMORY_TIMEOUT_S` env override, and the "abandoning the
call" behaviour is intact (`memory.py:256`). The restructure landed in `memory.py` on
**2026-10-29 AWST**, so the 60 s budget has been gone for ~2.5 weeks.
- The running dashboard (started 2026-11-10) loads the restructured code, so the exposure is live.
- One real call has since occurred — 2026-11-14 08:57 AWST, the first entry in
`conduit-calls.json → history` — with no abort lines in the logs afterwards. But that call met a
warm provider; it does not exercise the cold-start path. The page's original caveat stands.
- **Not re-applied by the stocktake**: porting the fix to the restructured code is a code change to
a git-managed upstream plugin that the next update will revert again — it needs Anthony's call
(re-port + commit locally, or upstream it).
The original incident window (2026-10-22 → 23) remains resolved; `status: degraded` records that
the *protection* is gone, not that the October fault returned.
## Follow-ups
- ⚠ **The fix is still unexercised by a call** (see the correction above): `conduit-calls.json →
history` is empty at 2026-10-25 00:2x, and the plugin is now **five releases behind HEAD**
(`0.64.0` → `0.69.0`).
- ~~⚠ **The fix is still unexercised by a call** …~~ **Superseded 2026-11-15**: one call has since
occurred (2026-11-14 08:57 AWST) — but the fix itself was lost in the restructure (section above),
so "unexercised" is now moot.
- **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which
asserted the capability did not exist **without calling the tool** →
[[incidents/2026-10-24-conduit-call-request-refused]].
- **The patch is uncommitted.** A plugin update that resets
`~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream
(`kaishi00/hermes-conduit-notifier`).
- ~~**The patch is uncommitted.** A plugin update that resets `~/.hermes/plugins/conduit_push`
reverts it to the 10 s budget.~~ **Confirmed 2026-11-15** — see the regression section above.
- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised.
Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]]