audit(2026-11-11): the audit's own 429 killed the 11-10 window; stocktake + radar recover with output
Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed). NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2 RESOLVED with output (dated Recovery sections added): - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B) - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md (opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x), incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
This commit is contained in:
@@ -0,0 +1,96 @@
|
||||
---
|
||||
title: Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-11-11
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
review_after: 2026-12-11
|
||||
confidence: high
|
||||
tags: [cron, incident, omniroute, rate-limit, audit]
|
||||
sources: [live-executions-db, live-cron-incidents, gateway-log, live-jobs-json]
|
||||
---
|
||||
|
||||
# Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited (2026-11-10 23:59 → 2026-11-11 00:12)
|
||||
|
||||
**This job is the estate's own observer, and on 2026-11-10 it became the estate's only failure.** Its
|
||||
23:59 slot never produced a report, so **no page, no log entry and no incident alert describes the day
|
||||
of 2026-11-10** — an unaudited day is invisible by construction.
|
||||
|
||||
## What happened
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Job | `cbfe434a3b5e` — **Wiki Audit — Nightly**, `custom:omni-101` / `deepseek-paid-smart` |
|
||||
| Slot | 2026-11-10 23:59:54.486572 +08:00 (cron `59 23 * * *`) |
|
||||
| Ended | **2026-11-11 00:12:29.646987 +08:00** — **12 m 35 s** |
|
||||
| Status | `failed` · `delivery_outcome: suppressed` |
|
||||
| Error | `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` |
|
||||
| Output | `~/.hermes/cron/output/cbfe434a3b5e/2026-11-11_00-12-28.md` |
|
||||
|
||||
The three log lines from the run's own session `local-1000c1a3…` show the shape of the failure:
|
||||
|
||||
ERROR agent.chat_completion_helpers: Streaming failed before delivery: [429]: service failure: tenant tn-5ud77abtl1 quota exceeded
|
||||
ERROR agent.chat_completion_helpers: Streaming failed before delivery: Error code: 429 -
|
||||
ERROR agent.conversation_loop: API call failed after 3 retries. HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters
|
||||
ERROR cron.scheduler: Job 'Wiki Audit — Nightly' failed: RuntimeError: HTTP 429: Service temporarily unavailable
|
||||
|
||||
- **The pool skipped every target.** Six candidates were evaluated and **`attempted 0`** — the run never
|
||||
reached a provider. The only live target's credit balance read **100 %** consumed on connection
|
||||
`conn:3021bab6`, which is why the router refused to dispatch rather than serving a degraded model.
|
||||
- **`reset after 6h 24m 31s`** puts the pool's recovery at **≈ 2026-11-11 06:36:57 AWST**. The 07:0x
|
||||
cron cohort ran clean the next morning, so the window closed on its own.
|
||||
- The **same tenant** (`tn-5ud77abtl1`) appears in the `quota exceeded` line, so this is an
|
||||
account-level quota fault on the router, not a single-model lockout.
|
||||
|
||||
## Incident row — raised, never alerted, still open
|
||||
|
||||
`cbfe43_ed01845310d8` — `job_id cbfe434a3b5e`, `failure_type: rate_limit`, state **`detected`**,
|
||||
`alerted_at: NULL`, `closed_at: NULL`, opened **2026-11-11T00:12:28.578435+08:00**.
|
||||
|
||||
- It is the **only open row** in `cron_incidents` (135 rows total) — the previous open item,
|
||||
`5fc292_773822455856`, closed `resolved` at **2026-11-11T08:03:47.644226**.
|
||||
- So the estate's single outstanding incident is **the audit's own failure**, and nothing was told
|
||||
about it.
|
||||
|
||||
## Impact
|
||||
|
||||
- **The 2026-11-10 window was never reviewed.** The nightly audit is the only producer of the per-day
|
||||
ledger/wiki record; its failure means 11-10's 157 ledger rows were never reconciled on the day.
|
||||
- **Fleet non-clean `last_status` 1 → 2.** `cbfe434a3b5e` now reads `last_status: error`,
|
||||
`failure_streak: 1`, joining `066bb5191280` (*Uber Eats Monthly Spend Report*, `delivery_failed`,
|
||||
streak 0).
|
||||
- **The review published on 2026-11-11 23:59 carries both windows** — it is the recovery, not a
|
||||
duplicate.
|
||||
|
||||
## Same signature, earlier appearances
|
||||
|
||||
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — the *identical* pre-dispatch-filter
|
||||
string killed the Daily Marketing Job Radar on 2026-10-21 (`attempted 0`, no 503 anywhere in the day).
|
||||
- 2026-11-05 — `deepseek-paid-smart` against the **CT600** endpoint returned `HTTP 503
|
||||
ALL_TARGETS_SKIPPED` (`poolSize 5`, `attempted 0`) while `omniroute-1` answered 200 — recorded on
|
||||
[[systems/scheduled-tasks]].
|
||||
- This is the **third distinct symptom** of the same router behaviour: `attempted 0` with a healthy node
|
||||
behind it is the thing to recognise, not the specific HTTP code.
|
||||
|
||||
## Fix Applied
|
||||
|
||||
**None — this is a detection-and-verification record.** No configuration changed this window. The run's
|
||||
own recovery was the pool reset at ≈ 06:36:57 on 2026-11-11.
|
||||
|
||||
## Prevention
|
||||
|
||||
- **The observer needs an observer.** A failed nightly audit raises no delivery (suppressed), no alert
|
||||
(`alerted_at: NULL`) and no page — every layer that would normally surface this is *inside the thing
|
||||
that failed*. A one-line watchdog on `Wiki Audit — Nightly`'s `last_status != ok` (or on "no audit
|
||||
commit in >24 h") is the only check that can catch it.
|
||||
- **Alert on the artefact.** Verifying that a dated `audit(YYYY-MM-DD)` commit exists in this repo would
|
||||
have caught both this failure and the stocktake's zero-edit days — the same lesson recorded on
|
||||
[[incidents/2026-11-09-stocktake-completes-with-zero-edits]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/omniroute]]
|
||||
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]
|
||||
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]
|
||||
Reference in New Issue
Block a user