wiki audit 2026-11-21: session-scoped capture re-diagnosis (124 Pulse rows, ~57 uncaptured turns, gateway-restart hypothesis NOT proven); NEW page — first content_policy_blocked digest failures (gmail 08eb86f02d37 + freshrss ecc1fc13655a); watchdog streak 115->210 (single incident row); wiki-audit credit outage RESOLVED (two nights lost, 11-20 delivery suppressed); ledger corrections (4 unclosed health-monitor rows, not 5; audit incident closed 2026-11-20T01:19:13; 7 unclosed cron_incidents); skills-index regenerated 299 skills/67 cats; freshness 5 stale/132 fresh/3 no-date; Quartz rebuilt+deployed (879 files)
This commit is contained in:
@@ -1890,3 +1890,61 @@ recorded here so the next audit does not re-litigate it.
|
||||
- `incidents/2026-10-05-dead-model-pin-repin.md`: residual "four jobs still carry pre-repin error status" has **cleared** — all seven now `last_status: ok`, streak 0. `command-code/gpt-5.6-terra` error class drifted **again**: now 403 `error code: 1010` (Cloudflare) vs 401 on 10-29 vs 403 MODEL_NOT_IN_PLAN originally — pin still dead, rejection layer differs. Fleet now 29 jobs / 26 enabled (was 30/27).
|
||||
- **Claims that held:** Steel fully removed (CT201 + CT333: no containers/images/routes; `_archived-steel/` intact); `cliprox/terra` 200 on CT600 / 401 on CT101 (per-node credentials, unchanged since 10-29); zero jobs on `cliprox`; `free-stack`/`smart-route` 200 on both nodes; docker.sock permission-denied for `hermes` on CT460; CT201 homepage `:8085` still 502 (degradation persists).
|
||||
- Not re-probed: the five pages at exactly 21d (job-radar ×2, conduit-voice, m93p-nfs, homepage-dashboard) — they become the 11-21 stale list; homepage's core claim was incidentally re-confirmed (502 persists) but the page itself was left for its own cycle. Pre-existing dangling wikilinks in `agents/nanobot.md` (`systems/gitea`, `systems/chromadb`, `incidents/nanobot-degraded-2026-08-07`, `runbooks/nanobot-restart`) confirmed present before this edit — flagged, not introduced.
|
||||
|
||||
---
|
||||
|
||||
## 2026-11-21 00:0x–00:5x AWST — nightly wiki audit (window: 2026-11-20)
|
||||
|
||||
**Headline: the memory capture fault is *session-scoped*, and its boundary is exact — the 11-20
|
||||
"conversational silence" conclusion is withdrawn.**
|
||||
|
||||
The desktop DM session `20261031_190813_95fbde` owns **124 Pulse/automation rows** in `working_memory`,
|
||||
spanning **2026-10-31 11:09:16 → 2026-11-13 05:10:35 UTC (13:10:35 AWST)** and **nothing after** — while the
|
||||
session itself kept producing turns: **~57** since, last captured turn **13:06:05 AWST**, first uncaptured
|
||||
**13:21:00 AWST** (verified against `state.db`). The **11-13 13:14 gateway restart** sits inside that
|
||||
15-minute boundary; that is the **leading hypothesis, explicitly NOT proven** — the oldest surviving
|
||||
`agent.log` file begins **11-13 19:03:34**, so the window has rotated out of the log horizon. Capture is
|
||||
**not** globally dead: the single write since, `2026-11-18 19:25:03 AWST`, came from the new session
|
||||
`hermes_20261118_191808_5c9e08`. Global silence now **52 h 35 m**; counters frozen at `working_memory`
|
||||
**1,131** / `episodic_memory` **1,250**. The health monitor is already reporting this in its own words —
|
||||
`STALE MEMORY PIPELINE: capture missed: user message`, `failure_streak = 4`.
|
||||
|
||||
**New failure class: `content_policy_blocked`.** Two unrelated digests died on the identical provider string
|
||||
on 11-20 — Gmail Action Digest **08:31:06** (`08eb86f02d37`) and FreshRSS Midday News Briefing **12:01:14**
|
||||
(`ecc1fc13655a`), both `RuntimeError: content_policy_blocked: Output data may contain inappropriate content.`
|
||||
Both carry `failure_streak = 1` (**isolated**, not chronic), both **alerted** and both **unclosed**, both
|
||||
classified `failure_type: unknown` → [[incidents/2026-11-20-content-policy-blocked-digest-failures]].
|
||||
|
||||
**Watchdog: 210 consecutive 15-minute slots** (was 115 at the 11-20 read). Incident
|
||||
`96eb95_98ade422d195` held **one id** from 11-18 20:01:15 to 11-21 00:31:16 (00:16:15 at the first read) — **no fragmentation**, unlike the
|
||||
health monitor, which re-forms a fresh incident id every slot for a single standing condition. 59 conflicts
|
||||
against a threshold of 50. Consolidation still **0 items** (199 of the last 200 cycles).
|
||||
|
||||
**The wiki-audit outage is resolved.** `cbfe434a3b5e` now reads `last_status: ok` / `failure_streak: 0`, and
|
||||
the execution ledger dates the outage exactly: **11-18 00:00:08 failed**, **11-19 00:00:16 failed**, **11-20
|
||||
00:00:08 completed** (report not delivered — `delivery_outcome: suppressed`), and **11-21 00:00:08** is this
|
||||
run. So **two** nights were lost (windows 11-17 and 11-18); the 11-20 run absorbed both via its three-window
|
||||
catch-up, which read the cron ledger and `state.db` rather than conversation. **No re-run was forced.**
|
||||
→ [[incidents/2026-11-18-wiki-audit-credit-outage]] now `resolved`.
|
||||
|
||||
**Ledger confirmation read at the end of this run** (`~/.hermes/cron/executions.db`, `cron_incidents`, 2026-11-21 00:3x):
|
||||
its incident row `cbfe43_fa12cca1ad4b` (`auth`) was **never alerted** and is **closed** (`closed_at` **2026-11-20 01:19:13**) — it
|
||||
cleared itself on the first clean completion. Unclosed rows stand at **7**, and none of them belong to the audit:
|
||||
`08eb86_c0cca1a55b96` + `ecc1fc_5a10b70830d1` (the two `content_policy_blocked` digests, `unknown`),
|
||||
four `ea79a2_*` health-monitor rows (`auth`, one per slot: `f2be2db3bfce`, `aae4d914f3c9`, `51c605d08ea5`, `ace0f263e18a`),
|
||||
and `96eb95_98ade422d195` (`script`, the conflict-queue trip). The same read moved the watchdog's `last_seen_at`
|
||||
to **00:31:16** — the streak is still live, so every "hold one id" timestamp on these pages is a moving target;
|
||||
the pages now cite **00:31:16 (00:16:15 at the first read)**. Earlier standing claims corrected in the same pass:
|
||||
the health monitor's unclosed count is **four** (not five) — the capture-stall title now says so — and
|
||||
[[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]'s "estate's only open incident row" is overtaken.
|
||||
|
||||
**Freshness:** `python3 raw/stocktake/_freshness_audit.py . raw/stocktake/freshness-20261121.txt 21` →
|
||||
**5 stale / 131 fresh / 3 no-date of 139 pages**; the five at exactly 21 d are the predicted cohort
|
||||
(`job-radar` ×2, `conduit-voice`, `m93p-nfs`, `homepage-dashboard`).
|
||||
|
||||
**No job added, retired, disabled or repinned; no wiki page removed or archived; no decision reversed.**
|
||||
Pages touched: new `incidents/2026-11-20-content-policy-blocked-digest-failures.md`;
|
||||
`incidents/2026-11-14-mnemosyne-capture-stall.md` (title + fifth read, correction);
|
||||
`incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md` (115 → 210 slots);
|
||||
`incidents/2026-11-18-wiki-audit-credit-outage.md` (→ resolved); `systems/mnemosyne.md`;
|
||||
`systems/scheduled-tasks.md`; `current-state.md`; `index.md`.
|
||||
|
||||
Reference in New Issue
Block a user