audit 2026-11-16: memory stall 62h54m (counts unchanged, cron_incidents 3->5 unclosed, consolidator zero-item streak 120); NEW incident — 11-15 plugin load-timeout storm (52 failed loads, 519 dropped registrations incl. conduit_push 474, 11 pre_tool_call + 2 pre_llm_call hook timeouts, 10 refusals) never recorded by the 11-15 audit; hook page Recurrence 2 (load average removed as predictor); fleet 29 jobs, 11-16 ledger 175 rows 173/2, deliveries 11/11

This commit is contained in:
hermes
2026-11-17 00:36:27 +08:00
parent 4eee02b9a2
commit e5612c8a8d
9 changed files with 299 additions and 22 deletions
+43
View File
@@ -1801,3 +1801,46 @@ Pages corrected: [[incidents/2026-10-09-cron-provider-timeout-storm]],
**Sessions:** 9 started on 11-15, **all `source: cron`** — no interactive session, so no human-side change the wiki missed. Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**.
Pages updated: [[incidents/2026-11-14-mnemosyne-capture-stall]] (status re-read, the `4.64 h` decode, the three-row table + fragmentation mechanism, the watchdog slot loss), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[current-state.md]], [[index.md]]. Root cause of the stall: **still not established** — the incident stays `active`, `closed_at: null`.
## 2026-11-17 00:2x — nightly audit (window 2026-11-16 00:00 – 23:59 AWST)
**Verdict: 1 new page + 7 page updates, nothing archived, no new system.**
**Window (11-16):** quiet by every measure except the one already open. Ledger **175 rows — 173
`completed` / 2 `failed`** (both `ea79a2107e36`, the Mnemosyne Health Monitor, now `failure_streak: 5`);
deliveries **11/11 `delivered`**, 2 `for_failure`; **16 cron sessions, zero interactive**; the 11-15
watchdog deps-guard 600 s timeout did **not** recur; `state.db` 2.09 GB / WAL 40 MB / `/` 88 % / load
6.19–7.48. **No new system, no new concept, no page worth archiving.**
**Prior-window correction — the substantive finding.** While backing up 11-15 (the window the previous
audit covered), exact greps of `errors.log` turned up an unrecorded **plugin load-timeout storm** on
**2026-11-15**: **52** `Failed to load plugin '…': load timed out after 10s`, **519** discarded
`register_*()` calls — `conduit_push` **474** of them across eight load events, i.e. Conduit's tools,
skills and hooks were silently absent for those processes — **11** `pre_tool_call` timeouts (5 in
01:07–01:26, 6 in 12:18–12:49), **2** `pre_llm_call` timeouts, **10** refused tool calls. 11-14 had 4 / 0
/ 1 / 1 / 4; **11-16 and 11-17 are 0 across the board.** No fix was applied and the 10 s loader budget is
a code default (there is no plugin-load-timeout key in `config.yaml`, only `hook_callback_timeout: 120`).
**Pages:**
- **New** [[incidents/2026-11-15-plugin-load-timeout-storm]] — counts by day and by plugin, the
load-failure/registration-drop distinction, the eight `conduit_push` events, clustering, and the
open question (fail-open loader; a registration health check is proposed, **not applied**).
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — 11-17 re-read: **62 h 54 m**, counts identical for
the fifth read, unclosed `alerted` **3 → 5** (143 total) with the two new incident ids, monitor streak
**5**, zero-item streak **120** (was 74); title raised 38+ → **62+ h**.
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — **Recurrence 2 (11-15)**: 11 timeouts in
two clusters, 10 refusals, call-level blocks to 120.63 s, plus the note that a hook cluster can fire
**without** a loader failure (12:18–12:49). Load average **removed** as a claimed predictor.
- [[systems/conduit]] — 474 dropped registrations and what they cost.
- [[systems/mnemosyne]] — OPEN paragraph carries the 11-17 numbers.
- [[systems/scheduled-tasks]] — 11-17 Count bullet (fleet, ledger, deliveries, streak, the 11-16 clean-up).
- [[current-state]] and [[index]] — 11-16 window sections; `Updated:` / `Last updated:` lines bumped.
**Freshness:** `reports/communities-wa-social-media-audit-2026` is the only page outside threshold
(23 d). It is a static deliverable with `review_after` 2026-11-26 — **left unbumped deliberately**, and
recorded here so the next audit does not re-litigate it.
**Still open from earlier windows:** the capture stall has no established cause and no fix applied
(`status: active`, `review_after` 2026-11-18); the incident-signature fragmentation fix remains
**proposed, not applied**; the plugin-load/hook-budget health check is **proposed, not applied**.