audit(2026-11-11): the audit's own 429 killed the 11-10 window; stocktake + radar recover with output
Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed). NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2 RESOLVED with output (dated Recovery sections added): - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B) - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md (opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x), incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
This commit is contained in:
@@ -1,10 +1,10 @@
|
||||
---
|
||||
title: "2026-11-09 — Stocktake Completes With Zero Edits"
|
||||
type: incident
|
||||
status: active
|
||||
status: resolved
|
||||
created: 2026-11-10
|
||||
updated: 2026-11-10
|
||||
verified_on: 2026-11-10
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [cron, wiki, stocktake, tool-budget, silent-failure, omni-101, free-stack]
|
||||
sources: ["executions.db (1a8e90ea2271: 2026-11-09T03:00:43+08:00 → 2026-11-09T12:10:47+08:00, status completed, delivery_outcome suppressed)", "~/.hermes/cron/output/1a8e90ea2271/2026-11-09_12-10-46.md (8,305 bytes; RUN INCOMPLETE, \\u201ctool-iteration budget exhausted during verification\\u201d)", "cron_incidents (1a8e90_628d914f1f97, 1a8e90_2f370ef764f5, 1a8e90_a9cad99cb150 — all closed 2026-11-09T12:10:47.125205+08:00)", "wiki-stocktake-maintenance job 1a8e90ea2271 (provider custom:omni-101, model free-stack)"]
|
||||
@@ -57,6 +57,35 @@ Also recorded this window: the `free-stack` pin on `1a8e90ea2271` — the only d
|
||||
- **Alerting must fire on write progress, not completion.** This run produced 2 of 3 failure days silently and the third day produced a run that reported success while editing nothing. A check on `commit_count` / `edits_applied` for this job would have caught all three.
|
||||
- Don't read `status`/`failure_streak` as the stocktake's health. Read whether a commit landed.
|
||||
|
||||
## Recovery — 2026-11-11: the pass ran in 1 h 10 m and landed 36 pages
|
||||
|
||||
The next scheduled slot **worked**, and worked quickly:
|
||||
|
||||
| Field | 11-09 (this incident) | **11-11** |
|
||||
|---|---|---|
|
||||
| Window | 03:00:43 → 12:10:47 | **03:00:56.081504 → 04:11:38.724852** |
|
||||
| Duration | **9 h 10 m** | **1 h 10 m 42 s** |
|
||||
| Terminal state | `completed` / `suppressed` | `completed` / `suppressed` |
|
||||
| Edits landed | **0** | **36 stale pages re-verified → 0 stale** |
|
||||
| Commit | **none** | **`341dda9`** + **`c108eb1`** |
|
||||
| Freshness report | — | `raw/stocktake/freshness-20261111.txt` (**6,235 B**) |
|
||||
|
||||
- `341dda9` re-verified 30+ further pages live and de-flagged them, and applied real content
|
||||
corrections — the live fallback chain rewritten in [[runbooks/test-openrouter]] (primary back to
|
||||
`smart-route` / `custom:omni-101`), the CT333 `clawtest` naming claim corrected, and the nous
|
||||
**404 `insufficient_credits_for_paid_model`** signature replacing the old refresh-token reading in
|
||||
[[incidents/2026-10-22-nous-portal-refresh-token-invalid]].
|
||||
- `c108eb1` regenerated the freshness report after the log/index addendum, so the published Quartz build
|
||||
matches HEAD.
|
||||
- **Job state:** `last_status: ok`, `failure_streak` 0, `last_run` 2026-11-11T03:00:56.
|
||||
⚠ The job is **still pinned to `custom:omni-101` / `free-stack`** — the combo recorded broken on
|
||||
2026-08-14 — and this run succeeded on it anyway. That is an observation, not a repin.
|
||||
|
||||
**Status: `resolved`.** The 11-07/11-08/11-09 run of failures is closed by output, not by a cosmetic
|
||||
`failure_streak` reset. The page's own prevention rule stands: *alert on `commit_count` / `edits_applied`,
|
||||
never on `status`* — this recovery is exactly why it matters, because 11-09 and 11-11 both read
|
||||
`completed` / `suppressed` and only one of them did anything.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
|
||||
Reference in New Issue
Block a user