audit 2026-11-16: memory stall 62h54m (counts unchanged, cron_incidents 3->5 unclosed, consolidator zero-item streak 120); NEW incident — 11-15 plugin load-timeout storm (52 failed loads, 519 dropped registrations incl. conduit_push 474, 11 pre_tool_call + 2 pre_llm_call hook timeouts, 10 refusals) never recorded by the 11-15 audit; hook page Recurrence 2 (load average removed as predictor); fleet 29 jobs, 11-16 ledger 175 rows 173/2, deliveries 11/11

This commit is contained in:
hermes
2026-11-17 00:36:27 +08:00
parent 4eee02b9a2
commit e5612c8a8d
9 changed files with 299 additions and 22 deletions
+32 -4
View File
@@ -3,16 +3,44 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-11-16
verified_on: 2026-11-16
review_after: 2026-11-17
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-18
confidence: high
tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
---
# Current State
Updated: 2026-11-16
Updated: 2026-11-17
## Recent Changes (2026-11-16, nightly audit; live probes 2026-11-16 23:59 – 2026-11-17 00:27 AWST)
**The memory pipeline is still dead at 62 h, and the 11-15 record needed a correction: a plugin
load-timeout storm nobody had logged.**
- **Capture stall, third read — unchanged in every respect but the counters.** `working_memory` **1130**
/ `episodic_memory` **1249**, newest write `2026-11-14 09:32:59` AWST → **62 h 54 m**. Unclosed
`alerted` incidents **3 → 5** (143 rows total); monitor `failure_streak` **5**; consolidator zero-item
streak **120** cycles. No cause, no fix, `status: active` →
[[incidents/2026-11-14-mnemosyne-capture-stall]]
- ⚠ **New page — [[incidents/2026-11-15-plugin-load-timeout-storm]].** 11-15 carried **52** failed plugin
loads and **519** discarded registrations, `conduit_push` alone losing **474** (its tools and hooks were
absent for those processes), plus **11** `pre_tool_call` and **2** `pre_llm_call` hook timeouts and
**10** refused tool calls. The 11-15 audit recorded **none** of it. **Zero on 11-16 and 11-17** — it
cleared with no fix applied. The companion
[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] gained a Recurrence 2 section; load average
is now **explicitly not** a sufficient predictor (11-17 read 6.19 vs 12.08 on 11-14, no recurrence, and
a *larger* `state.db`).
- **Cron fleet, 11th window, flat at 29 jobs** (26 enabled / 3 disabled). 11-16 ledger **175 rows —
173 `completed` / 2 `failed`**, both the health monitor; deliveries **11/11 `delivered`**. 11-17 so far:
5 / 1 / 1. The 11-15 watchdog deps-guard 600 s timeout **did not recur**.
- **Quiet where it matters, growing where it doesn't.** 11-16 ran **16 cron sessions, zero interactive** —
nothing human-side to document. `state.db` **2,093,998,080 B (2.09 GB)**, WAL **40 MB**, `/` at **88 %**,
load **6.19 / 6.64 / 7.48**, uptime 6 d 2 h (host up since 2026-11-10).
- **Freshness:** 1 stale page by design — `reports/communities-wa-social-media-audit-2026` (23 d at
11-17); it is a static deliverable with `review_after` 2026-11-26, so it was left unbumped.
## Recent Changes (2026-11-15, nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST)