wiki audit 2026-11-20: three-window catch-up (11-17/11-18/11-19, both prior audit slots died)

- new: incidents/2026-11-18-wiki-audit-credit-outage (audit failed 11-18 + 11-19, deliveries suppressed, incident row alerted_at=None)
- new: incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm (115 consecutive slots, 59 unresolved conflicts vs threshold 50)
- corrected: incidents/2026-11-14-mnemosyne-capture-stall + systems/mnemosyne — freeze ENDED 2026-11-18 19:25 AWST (105h52m, no intervention); 'capture dead' reading was conversational silence (every interactive session captured)
- systems/scheduled-tasks: new Count bullet 11-17 173(171/2) / 11-18 172(153/19) / 11-19 176(78/98); fleet 29 (26/3); non-clean last_status 1->4; unclosed incidents 5->4; deliveries 24 (23 delivered / 1 failed)
- index.md feed + incidents + systems lines; log.md dated entry; current-state.md new section; frontmatter dates -> 2026-11-20
This commit is contained in:
Tony0410
2026-11-20 01:02:44 +08:00
parent 16f2c878c0
commit 226767f3ec
8 changed files with 310 additions and 22 deletions
+11
View File
@@ -1864,3 +1864,14 @@ recorded here so the next audit does not re-litigate it.
- `incidents/2026-10-26-planned-restart-orphaned-cron-run.md` — both `ea79a2_*` incident rows closed/resolved (c3ab2d784ba3 10-27, a4661554a3c2 closed 11-07 15:43), error string absent from retained executions window → status → resolved; drain-policy follow-ups 1/3/4 remain open design questions.
- `reports/communities-wa-social-media-audit-2026.md` — live wa.gov.au check: remit verbatim-identical, five Ministers unchanged, site updated 18 Nov 2026 with fresh announcements.
- Quartz rebuilt + deployed via second `wiki-maintain.sh` run (pattern 13).
## 2026-11-20 00:5x — nightly audit (three windows: 2026-11-17, 11-18 and 11-19 AWST)
**Why three windows:** the audit's own slots at **2026-11-18T00:00:08 → 00:01:43** and **2026-11-19T00:00:16 → 00:03:02** both closed `failed`, so **2026-11-17 and 2026-11-18 were never audited**. This run covers all three.
- **New page — [[incidents/2026-11-18-wiki-audit-credit-outage]]:** `cbfe434a3b5e` (Wiki Audit — Nightly) is at `failure_streak: 2`, both failures on a chained-provider **credit/auth** fault (last clean run 11-17). Both failed slots carry `delivery_outcome: suppressed`, so **no failure notice ever reached Anthony** — the estate's own auditor was the one thing nobody heard about. Its incident row `cbfe43_fa12cca1ad4b` (`failure_type: auth`, opened 2026-11-18 00:01:42) is still `alerted` with **`alerted_at: None`**, the same silent-skip shape as the 11-10 429 incident.
- **New page — [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]:** `96eb95ce12f2` (Mnemosyne Watchdog) has failed **every** 15-minute slot since **2026-11-18 20:01:06**, reaching **115 consecutive failures** at **2026-11-20 00:31:16** (`failure_streak: 115`, `last_status: error`), for one reason: **59 unresolved conflicts against its threshold of 50**. A single durable trip, not 115 faults — but each slot delivers a fresh *failure notice* (9 of the 24 deliveries since 11-18 carry `for_failure: 1`), so the noisy channel is the estate's only channel, and every slot re-opens incident `96eb95_98ade422d195`.
- **Corrected — [[incidents/2026-11-14-mnemosyne-capture-stall]] and [[systems/mnemosyne]]:** the freeze is **over**. `working_memory` **1130 → 1131**, `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST** → the stall ran **105 h 52 m** and ended **with no intervention** (no config edit, restart or repair on the capture path). ⚠ **And the "capture is dead" reading was wrong:** `state.db` lists only **three** non-cron sessions since 11-14 00:00 — two `desktop` on **11-14 08:35 / 08:52** (both *before* the 09:33 boundary) and the **11-18 19:18** `desktop` whose turn *is* the captured row — so **every interactive session in the window was captured**. `mnemosyne_freshness_gate.py` measures the age of the newest non-`cron`/`subagent` message, so **silence and failure read identically**; that ambiguity is the real defect, and the monitor is still failing on it (`failure_streak: 2`; 11-19 00:01 was the only clean slot). Title and body reframed accordingly.
- **[[systems/scheduled-tasks]] — new Count bullet (11-17 / 11-18 / 11-19):** ledger **173 rows (171 `completed` / 2 `failed`)** / **172 (153/19)** / **176 (78/98)** — the last is the worst day in the retained window; 11-20 so far **8 (3 / 4 / 1 `running`)**. Deliveries since 11-18: **24 — 23 `delivered`** (9 `for_failure: 1`) / **1 `failed`** (a *content* delivery, `live adapter send failed: Timed out`, `2026-11-19 08:34:13`). Fleet **29 jobs (26 enabled / 3 disabled)** — a **twelfth consecutive window** at the same count (11-18 and 11-19 went unrecorded, see above) — but non-clean `last_status` **1 → 4**: `cbfe434a3b5e` `error` (2), `96eb95ce12f2` `error` (115), `ea79a2107e36` `error` (2), `08eb86f02d37` `delivery_failed` (0). `cron_incidents` unclosed **5 → 4**. The 03:00 stocktake (`1a8e90ea2271`) ran `completed` on all three days — **11 m 35 s / 23 m 07 s / 24 m 38 s** — so the estate's structural maintenance kept running while both alarm paths were red.
- **[[current-state]]** — new Recent Changes section for the 2026-11-17 → 2026-11-20 window. Frontmatter `updated` / `verified_on` bumped to **2026-11-20** across `index.md`, `current-state.md`, `systems/mnemosyne.md`, `systems/scheduled-tasks.md` and the two touched incident pages.
- **Mnemosyne consolidation is still zero-item:** newest `consolidation_log` row **`2026-11-19T16:20:14+00:00`** (= 2026-11-20 00:20 AWST, ~31 min cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle — timestamp updated for watchdog"`; **199 of the last 200 cycles are zero-item** (was **120** on the 11-17 read). The zero-item-streak threshold remains **proposed, not applied**.