audit(2026-10-14): 10-13 broke the clean streak — wiki-stocktake failed on a dead opencode credential (HTTP 404, incident 1a8e90_d11698bd7275); stranded skills-index + freshness-20261013 recovered; retention floor moved to 2026-10-08 07:11
This commit is contained in:
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-13
|
||||
verified_on: 2026-10-13
|
||||
updated: 2026-10-14
|
||||
verified_on: 2026-10-14
|
||||
review_after: 2026-10-25
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
@@ -341,3 +341,68 @@ archived: the DHL monitor stays where the 2026-10-10 audit put it (redundant, in
|
||||
multi-tab workloads (new 2026-10-13); Quartz rebuild step inert; skills-index off-by-one; **3** non-clean
|
||||
`last_status` rows (`0e753e9720fd`, `1b23c042e17d`, `066bb5191280`); Mattermost platform unreachable;
|
||||
gateway runtime env outside the 15-minute dependency guard.
|
||||
|
||||
|
||||
## Re-verified 2026-10-14
|
||||
|
||||
Window: **2026-10-13 00:00 → 23:59 AWST**; live probes 2026-10-14 00:00–00:30 AWST.
|
||||
Fleet **29 jobs (26 enabled / 3 disabled)** — unchanged this cycle. **Open incidents 2 → 3**.
|
||||
|
||||
**Cron tally — the clean streak broke.** 2026-10-13 = **`167 completed` / 1 `failed`** (plus this audit's own row
|
||||
in flight at probe time, which finalises the day at 168). Tail: 10-10 `175 / 0`, 10-11 `172 / 0`, 10-12 `175 / 0`
|
||||
→ **the first failed cron run since 2026-10-09** (a day that recorded 15).
|
||||
|
||||
**Finding 1 — `wiki-stocktake-maintenance` (`1a8e90ea2271`) died on a dead `opencode` credential.**
|
||||
03:00:43 → 03:01:57, `RuntimeError: HTTP 404: No active credentials for provider: opencode.`
|
||||
Pinned `custom:omni-101:free-stack`. New incident **`1a8e90_d11698bd7275`** (`state: detected`,
|
||||
`failure_type: agent`, `alerted_at: NULL` — failure streak 1, so under the 3-strike alert rule).
|
||||
This is the job's **third failure in six days, all three on the opencode path**: 10-08
|
||||
`HTTP 403: OpenCode's free tier can only be used from within OpenCode` (`1a8e90_789224ad589c`), 10-09 timeout
|
||||
(`1a8e90_dc6b2d40ec44`), 10-13 404 no credentials (`1a8e90_d11698bd7275`). The first two resolved on
|
||||
2026-10-10; this one clears only when the 2026-10-14 03:00 run succeeds — i.e. **each incident clears only on
|
||||
the next successful run of its own job**. → [[incidents/2026-10-05-dead-model-pin-repin]]
|
||||
|
||||
**Finding 2 — the failure stranded its own generated artifacts; this audit committed them.**
|
||||
The script's step 1 (skills-index regeneration) and step 2 (freshness report) both completed and wrote to disk,
|
||||
then the agent/LLM turn that follows died → nothing committed, nothing pushed. Left in the working tree:
|
||||
`systems/skills-index.md` (modified — **date-only change: `updated`/`verified_on` 10-12 → 10-13; 346 skills /
|
||||
70 categories unchanged**) and `raw/stocktake/freshness-20261013.txt` (untracked — **48 stale / 38 fresh /
|
||||
7 no-date of 93 pages**, down from 54 stale / 32 fresh on 10-12). Both are in this audit's commit.
|
||||
Structural point: the *agent* row is what commits, so a perfectly healthy script run still publishes nothing
|
||||
when the LLM turn dies. There is no green-row alternative — the row went red.
|
||||
|
||||
**Retention floor moved.** `executions.db` now holds **1,001 rows** (1,000 terminal + 1 in flight);
|
||||
`MAX_TERMINAL_EXECUTIONS = 1000` in `cron/executions.py`. Oldest retained terminal row: **2026-10-08 07:11**.
|
||||
Days above the floor are complete; older ones undercount — 10-08 now reads `126 completed / 1 failed` where its
|
||||
finalised tally was higher. Run-level evidence older than the floor is pruned, not recoverable.
|
||||
|
||||
**Delivery.** 10-13: `159 suppressed` / **9 delivered** / 1 no-record. Delivered = Daily Marketing Job Radar
|
||||
(`dca8482e4f76`), Mosquito Tracker (`1b31187397a5` ×3), Tech & AI Newsletter Digest (`85edf6772bca`),
|
||||
Daily Birthday Checker (`a432898b55a4`), Gmail Action Digest (`08eb86f02d37`), FreshRSS Midday News Briefing
|
||||
(`ecc1fc13655a`), Evening Stand-Down Briefing (`e530c89f2be0`). No unexpected delivery; the two dead-thread
|
||||
delivery faults (`0e753e9720fd`, `066bb5191280`, both Telegram `thread_id 34795`) are unchanged since 10-01.
|
||||
|
||||
**Mnemosyne.** All six monitor runs returned the silent token — Health Monitor 00:02 and 12:02, Embedding
|
||||
Coverage Monitor 00:02 / 06:02 / 12:01 / 18:01 → no exception, i.e. both coverages at/above the 95 % floor and
|
||||
no `reindex`. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = **96.7 %**, episodic
|
||||
1050/1056 = **99.4 %**). Wrinkle: the 12:02 health run emitted the token **markdown-bolded** rather than bare;
|
||||
the delivery layer still suppressed it (`delivery_outcome: suppressed`), so the guard held — but the matcher is
|
||||
tolerant of wrapping rather than literal. → [[systems/mnemosyne]]
|
||||
|
||||
**Non-clean `last_status`: 3 → 4.** Cleared: `fad5a86979d4` (success 10-12 03:05), `dca8482e4f76` (success
|
||||
10-13 07:02). Added: `1a8e90ea2271`. Remaining: `0e753e9720fd` (10-01), `1b23c042e17d` (10-09),
|
||||
`066bb5191280` (10-01), `1a8e90ea2271` (10-13).
|
||||
|
||||
**Sessions.** **15 in the window, all `cron`, zero interactive** — third consecutive window with no interactive
|
||||
session.
|
||||
|
||||
**Pages touched:** `systems/scheduled-tasks.md` (this block, frontmatter, non-clean list, retention note),
|
||||
`systems/skills-index.md` (committed stocktake artifact), `current-state.md`, `index.md`, `log.md`.
|
||||
**Nothing added/removed:** no new system, no retired job, no page archived → `index.md` structure unchanged.
|
||||
|
||||
**Still open after this cycle:** `wiki-stocktake-maintenance` opencode credential (new 2026-10-13 — watch the
|
||||
10-14 03:00 run: streak 3 triggers an alert, and the durable fix is to repin the job off `free-stack`-opencode
|
||||
or repair OmniRoute's `opencode` provider credentials); GitHub Trending Radar token (2026-10-13, next run
|
||||
Mon 09:00); Camofox containment vs. multi-tab workloads (2026-10-13); Quartz rebuild step inert;
|
||||
skills-index off-by-one; **4** non-clean `last_status` rows (above); Mattermost platform unreachable;
|
||||
gateway runtime env outside the 15-minute dependency guard.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-12
|
||||
verified_on: 2026-10-12
|
||||
updated: 2026-10-13
|
||||
verified_on: 2026-10-13
|
||||
confidence: high
|
||||
tags: [skills, hermes]
|
||||
sources: []
|
||||
|
||||
Reference in New Issue
Block a user