audit(2026-10-14): 10-13 broke the clean streak — wiki-stocktake failed on a dead opencode credential (HTTP 404, incident 1a8e90_d11698bd7275); stranded skills-index + freshness-20261013 recovered; retention floor moved to 2026-10-08 07:11
This commit is contained in:
@@ -598,3 +598,58 @@ history entries), `systems/mnemosyne.md` (12:02 coverage), `incidents/2026-10-09
|
||||
(status + full closure table + timeline), `index.md` (date, four blurbs), `log.md`.
|
||||
|
||||
No secrets written; job/incident IDs, paths and tailnet addresses only.
|
||||
|
||||
## [2026-10-14] audit | nightly review of 2026-10-13 — the clean streak broke on a dead opencode credential
|
||||
|
||||
Window 2026-10-13 00:00 → 23:59 AWST (ran 2026-10-13 23:59 → 2026-10-14 00:2x AWST). Live probes only.
|
||||
|
||||
**Cron.** 2026-10-13 = **`167 completed` / 1 `failed`** (plus this audit's own row in flight → the day finalises
|
||||
at 168). Tail: 10-10 `175 / 0`, 10-11 `172 / 0`, 10-12 `175 / 0` — **the first failed run since 2026-10-09**
|
||||
(which recorded 15). Retention floor moved to **2026-10-08 07:11** (`MAX_TERMINAL_EXECUTIONS = 1000`; 1,001 rows
|
||||
= 1,000 terminal + 1 in flight), so days above the floor are complete and older ones undercount (10-08 reads
|
||||
`126 / 1`, partial).
|
||||
|
||||
**Finding 1 — `wiki-stocktake-maintenance` (`1a8e90ea2271`) died on a dead `opencode` credential.** 03:00:43 →
|
||||
03:01:57, `RuntimeError: HTTP 404: No active credentials for provider: opencode.` Pinned
|
||||
`custom:omni-101:free-stack`. New incident **`1a8e90_d11698bd7275`** (`state: detected`, `failure_type: agent`,
|
||||
`alerted_at: NULL` — streak 1, under the 3-strike alert rule). Third failure of this job in six days, all three
|
||||
on the opencode path (10-08 `HTTP 403: OpenCode's free tier can only be used from within OpenCode`; 10-09
|
||||
timeout; 10-13 404). The first two resolved on 2026-10-10; this one clears only on a successful 2026-10-14 03:00
|
||||
run. → [[incidents/2026-10-05-dead-model-pin-repin]]
|
||||
|
||||
**Finding 2 — the failure stranded its own generated artifacts (now committed).** Step 1 (skills-index) and
|
||||
step 2 (freshness report) both completed, then the agent turn failed → no commit, no push. Left in the working
|
||||
tree: `systems/skills-index.md` (modified, **date-only**: `updated`/`verified_on` 10-12 → 10-13; **346 skills /
|
||||
70 categories unchanged**) and `raw/stocktake/freshness-20261013.txt` (untracked; **48 stale / 38 fresh /
|
||||
7 no-date of 93 pages**, down from 54 stale / 32 fresh on 10-12). Both are committed by this audit.
|
||||
Consequence: the stocktake published nothing on 10-13 — and because the *agent* row is what commits, even a
|
||||
perfect script run leaves nothing behind when the LLM turn dies.
|
||||
|
||||
**Delivery.** `159 suppressed` / **9 delivered** / 1 no-record. Delivered: Daily Marketing Job Radar,
|
||||
Mosquito Tracker ×3, Tech & AI Newsletter Digest, Daily Birthday Checker, Gmail Action Digest, FreshRSS Midday,
|
||||
Evening Stand-Down Briefing. No unexpected delivery; the two dead-thread faults (`0e753e9720fd`,
|
||||
`066bb5191280`, both Telegram `thread_id 34795`) are unchanged since 2026-10-01.
|
||||
|
||||
**Mnemosyne.** Six monitor runs, all silent: Health Monitor 00:02 + 12:02, Embedding Coverage Monitor 00:02 /
|
||||
06:02 / 12:01 / 18:01 → no exception, so both coverages stayed at/above the 95 % floor and no `reindex` was
|
||||
needed. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = **96.7 %**, episodic
|
||||
1050/1056 = **99.4 %**). Wrinkle: the 12:02 health run emitted the silent token markdown-bolded rather than
|
||||
bare; the delivery layer still suppressed it (`delivery_outcome: suppressed`) — the matcher tolerates wrapping
|
||||
rather than requiring the literal token as a strict control. → [[systems/mnemosyne]]
|
||||
|
||||
**Sessions.** **15 in the window, all `cron`, zero interactive** — third consecutive window with no interactive
|
||||
session.
|
||||
|
||||
**Fleet.** 29 jobs (26 enabled / 3 disabled), pins unchanged; **non-clean `last_status` 3 → 4** (cleared
|
||||
`fad5a86979d4` 10-12 03:05 and `dca8482e4f76` 10-13 07:02; added `1a8e90ea2271`); **open incidents 2 → 3**
|
||||
(1 `alerted`, 2 `detected`, 8 `closed`, 66 `resolved`; 77 rows).
|
||||
|
||||
**Pages touched:** `systems/scheduled-tasks.md` (frontmatter, retention floor, non-clean list,
|
||||
`## Re-verified 2026-10-14`), `systems/skills-index.md` (committed stocktake artifact), `current-state.md`
|
||||
(new Recent Changes + date), `index.md` (date + two blurbs), `log.md`.
|
||||
|
||||
**Vault mirror.** The 2026-10-13 audit was never mirrored to `/home/hermes/wiki` (last mirror `bb09581`,
|
||||
2026-10-12), so this run mirrored both nights: the 10-13 audit's ten pages plus `skills-index.md`, whose vault
|
||||
copy was still the 2026-10-08 version. The vault's own `raw/stocktake/` still holds only the 2026-10-03 report.
|
||||
|
||||
No secrets written; job/incident IDs, paths and tailnet addresses only.
|
||||
|
||||
Reference in New Issue
Block a user