audit(2026-10-14): 10-13 broke the clean streak — wiki-stocktake failed on a dead opencode credential (HTTP 404, incident 1a8e90_d11698bd7275); stranded skills-index + freshness-20261013 recovered; retention floor moved to 2026-10-08 07:11

This commit is contained in:
Tony0410
2026-10-14 00:17:31 +08:00
parent 1df564e152
commit 828913e23b
6 changed files with 244 additions and 10 deletions
+55
View File
@@ -598,3 +598,58 @@ history entries), `systems/mnemosyne.md` (12:02 coverage), `incidents/2026-10-09
(status + full closure table + timeline), `index.md` (date, four blurbs), `log.md`.
No secrets written; job/incident IDs, paths and tailnet addresses only.
## [2026-10-14] audit | nightly review of 2026-10-13 — the clean streak broke on a dead opencode credential
Window 2026-10-13 00:00 → 23:59 AWST (ran 2026-10-13 23:59 → 2026-10-14 00:2x AWST). Live probes only.
**Cron.** 2026-10-13 = **`167 completed` / 1 `failed`** (plus this audit's own row in flight → the day finalises
at 168). Tail: 10-10 `175 / 0`, 10-11 `172 / 0`, 10-12 `175 / 0` — **the first failed run since 2026-10-09**
(which recorded 15). Retention floor moved to **2026-10-08 07:11** (`MAX_TERMINAL_EXECUTIONS = 1000`; 1,001 rows
= 1,000 terminal + 1 in flight), so days above the floor are complete and older ones undercount (10-08 reads
`126 / 1`, partial).
**Finding 1 — `wiki-stocktake-maintenance` (`1a8e90ea2271`) died on a dead `opencode` credential.** 03:00:43 →
03:01:57, `RuntimeError: HTTP 404: No active credentials for provider: opencode.` Pinned
`custom:omni-101:free-stack`. New incident **`1a8e90_d11698bd7275`** (`state: detected`, `failure_type: agent`,
`alerted_at: NULL` — streak 1, under the 3-strike alert rule). Third failure of this job in six days, all three
on the opencode path (10-08 `HTTP 403: OpenCode's free tier can only be used from within OpenCode`; 10-09
timeout; 10-13 404). The first two resolved on 2026-10-10; this one clears only on a successful 2026-10-14 03:00
run. → [[incidents/2026-10-05-dead-model-pin-repin]]
**Finding 2 — the failure stranded its own generated artifacts (now committed).** Step 1 (skills-index) and
step 2 (freshness report) both completed, then the agent turn failed → no commit, no push. Left in the working
tree: `systems/skills-index.md` (modified, **date-only**: `updated`/`verified_on` 10-12 → 10-13; **346 skills /
70 categories unchanged**) and `raw/stocktake/freshness-20261013.txt` (untracked; **48 stale / 38 fresh /
7 no-date of 93 pages**, down from 54 stale / 32 fresh on 10-12). Both are committed by this audit.
Consequence: the stocktake published nothing on 10-13 — and because the *agent* row is what commits, even a
perfect script run leaves nothing behind when the LLM turn dies.
**Delivery.** `159 suppressed` / **9 delivered** / 1 no-record. Delivered: Daily Marketing Job Radar,
Mosquito Tracker ×3, Tech & AI Newsletter Digest, Daily Birthday Checker, Gmail Action Digest, FreshRSS Midday,
Evening Stand-Down Briefing. No unexpected delivery; the two dead-thread faults (`0e753e9720fd`,
`066bb5191280`, both Telegram `thread_id 34795`) are unchanged since 2026-10-01.
**Mnemosyne.** Six monitor runs, all silent: Health Monitor 00:02 + 12:02, Embedding Coverage Monitor 00:02 /
06:02 / 12:01 / 18:01 → no exception, so both coverages stayed at/above the 95 % floor and no `reindex` was
needed. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = **96.7 %**, episodic
1050/1056 = **99.4 %**). Wrinkle: the 12:02 health run emitted the silent token markdown-bolded rather than
bare; the delivery layer still suppressed it (`delivery_outcome: suppressed`) — the matcher tolerates wrapping
rather than requiring the literal token as a strict control. → [[systems/mnemosyne]]
**Sessions.** **15 in the window, all `cron`, zero interactive** — third consecutive window with no interactive
session.
**Fleet.** 29 jobs (26 enabled / 3 disabled), pins unchanged; **non-clean `last_status` 3 → 4** (cleared
`fad5a86979d4` 10-12 03:05 and `dca8482e4f76` 10-13 07:02; added `1a8e90ea2271`); **open incidents 2 → 3**
(1 `alerted`, 2 `detected`, 8 `closed`, 66 `resolved`; 77 rows).
**Pages touched:** `systems/scheduled-tasks.md` (frontmatter, retention floor, non-clean list,
`## Re-verified 2026-10-14`), `systems/skills-index.md` (committed stocktake artifact), `current-state.md`
(new Recent Changes + date), `index.md` (date + two blurbs), `log.md`.
**Vault mirror.** The 2026-10-13 audit was never mirrored to `/home/hermes/wiki` (last mirror `bb09581`,
2026-10-12), so this run mirrored both nights: the 10-13 audit's ten pages plus `skills-index.md`, whose vault
copy was still the 2026-10-08 version. The vault's own `raw/stocktake/` still holds only the 2026-10-03 report.
No secrets written; job/incident IDs, paths and tailnet addresses only.