Files
agent-estate-wiki/incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
Tony0410 e033a08f17 audit(2026-11-12): the estate's last incident closed itself; a past-0.9.42 hq-push install was refused by the plugin scan (fix parked in PR #2)
- 429 row cbfe43_ed01845310d8 closed resolved 2026-11-12T00:25:36 on the next clean slot; 0 of 137 unclosed
- 11-12: 176 rows / 175 completed / 0 failed / 1 running; 11-11: 172/172/0; deliveries 9/9 and 7/7 delivered
- NEW incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md (dangerous verdict, one critical in the
  plugin's own tile skill; --force cannot override; no per-verdict override exists)
- systems/hermes-hq-push.md corrected live (0.9.27 -> 0.9.42, tree mtime 2026-11-08 02:19, config line 13226)
- current-state / scheduled-tasks / index / log updated; skills-index regenerated; freshness 2026-11-13 (0 stale)
2026-11-13 00:41:15 +08:00

119 lines
7.0 KiB
Markdown

---
title: Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited
type: incident
status: resolved
created: 2026-11-11
updated: 2026-11-12
verified_on: 2026-11-12
review_after: 2026-12-11
confidence: high
tags: [cron, incident, omniroute, rate-limit, audit]
sources: [live-executions-db, live-cron-incidents, gateway-log, live-jobs-json]
---
# Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited (2026-11-10 23:59 → 2026-11-11 00:12)
**This job is the estate's own observer, and on 2026-11-10 it became the estate's only failure.** Its
23:59 slot never produced a report, so **no page, no log entry and no incident alert describes the day
of 2026-11-10** — an unaudited day is invisible by construction.
## What happened
| Field | Value |
|---|---|
| Job | `cbfe434a3b5e` — **Wiki Audit — Nightly**, `custom:omni-101` / `deepseek-paid-smart` |
| Slot | 2026-11-10 23:59:54.486572 +08:00 (cron `59 23 * * *`) |
| Ended | **2026-11-11 00:12:29.646987 +08:00** — **12 m 35 s** |
| Status | `failed` · `delivery_outcome: suppressed` |
| Error | `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` |
| Output | `~/.hermes/cron/output/cbfe434a3b5e/2026-11-11_00-12-28.md` |
The three log lines from the run's own session `local-1000c1a3…` show the shape of the failure:
ERROR agent.chat_completion_helpers: Streaming failed before delivery: [429]: service failure: tenant tn-5ud77abtl1 quota exceeded
ERROR agent.chat_completion_helpers: Streaming failed before delivery: Error code: 429 -
ERROR agent.conversation_loop: API call failed after 3 retries. HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters
ERROR cron.scheduler: Job 'Wiki Audit — Nightly' failed: RuntimeError: HTTP 429: Service temporarily unavailable
- **The pool skipped every target.** Six candidates were evaluated and **`attempted 0`** — the run never
reached a provider. The only live target's credit balance read **100 %** consumed on connection
`conn:3021bab6`, which is why the router refused to dispatch rather than serving a degraded model.
- **`reset after 6h 24m 31s`** puts the pool's recovery at **≈ 2026-11-11 06:36:57 AWST**. The 07:0x
cron cohort ran clean the next morning, so the window closed on its own.
- The **same tenant** (`tn-5ud77abtl1`) appears in the `quota exceeded` line, so this is an
account-level quota fault on the router, not a single-model lockout.
## Incident row — raised, never alerted, closed `resolved` 2026-11-12T00:25:36
`cbfe43_ed01845310d8` — `job_id cbfe434a3b5e`, `failure_type: rate_limit`, state **`detected`**,
`alerted_at: NULL`, `acked_at: NULL`, `closed_at: NULL`, opened **2026-11-11T00:12:28.578435+08:00**
(`first_seen_at` = `last_seen_at` = that instant — seen exactly once).
- It **was** the only open row in `cron_incidents` — the table held **135** rows at audit time and **137**
by the end of this run. `acked_at` is **NULL** as well as `alerted_at`, so nothing was told and
nothing was acknowledged. The previous open item,
`5fc292_773822455856`, closed `resolved` at **2026-11-11T08:03:47.644226**.
- So the estate's single outstanding incident **was** the audit's own failure, and nothing was told
about it. ⚠ **Superseded 2026-11-12** — the row closed `resolved` at **2026-11-12T00:25:36.133387+08:00**;
*cron_incidents* now holds **0 unclosed of 137**. See *Resolution* below.
## Resolution — 2026-11-12T00:25:36, self-cleared
The row closed **`resolved`** at **2026-11-12T00:25:36.133387+08:00**, at the end of the **next** slot of
the same job: `cbfe434a3b5e` ran **2026-11-11T23:59:54.486572 → 2026-11-12T00:25:36.133387 = 25 m 42 s**
and completed. Its delivery carried **both windows**, so 2026-11-10's day was reconciled a day late
rather than never (the recovery record is the `audit(2026-11-11)` commit series at 2026-11-12 00:1x-00:24).
Nothing was repaired to make this happen: the pool's own reset at ≈ **2026-11-11 06:36:57** restored the
dispatch path, exactly as the `reset after 6h 24m 31s` field predicted, and the 07:0x cohort ran clean.
`cron_incidents` now reads **0 unclosed of 137** rows — the estate is incident-clean as of 2026-11-12.
**The recurrence risk is unchanged.** The same *"all targets were skipped by pre-dispatch filters"*
signature was found again on a **second job and a second pin** by the 2026-11-12 03:00 stocktake
([[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]), and every model-calling job still rides
`custom:omni-101` with no failover for the free-tier cohort. Closing this row proves the guard
self-heals; it is not evidence the guard is gone.
## Impact
- **The 2026-11-10 window was never reviewed.** The nightly audit is the only producer of the per-day
ledger/wiki record; its failure means 11-10's 157 ledger rows were never reconciled on the day.
- **Fleet non-clean `last_status` 1 → 2.** `cbfe434a3b5e` now reads `last_status: error`,
`failure_streak: 1`, joining `066bb5191280` (*Uber Eats Monthly Spend Report*, `delivery_failed`,
streak 0).
- **The review published on 2026-11-11 23:59 carries both windows** — it is the recovery, not a
duplicate.
## Same signature, earlier appearances
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — the *identical* pre-dispatch-filter
string killed the Daily Marketing Job Radar on 2026-10-21 (`attempted 0`, no 503 anywhere in the day).
- 2026-11-05 — `deepseek-paid-smart` against the **CT600** endpoint returned `HTTP 503
ALL_TARGETS_SKIPPED` (`poolSize 5`, `attempted 0`) while `omniroute-1` answered 200 — recorded on
[[systems/scheduled-tasks]].
- This is the **third distinct symptom** of the same router behaviour: `attempted 0` with a healthy node
behind it is the thing to recognise, not the specific HTTP code.
## Fix Applied
**None — this is a detection-and-verification record.** No configuration changed this window. The run's
own recovery was the pool reset at ≈ 06:36:57 on 2026-11-11, and the incident row closed on the next
slot's clean completion (2026-11-12T00:25:36).
## Prevention
- **The observer needs an observer.** A failed nightly audit raises no delivery (suppressed), no alert
(`alerted_at: NULL`) and no page — every layer that would normally surface this is *inside the thing
that failed*. A one-line watchdog on `Wiki Audit — Nightly`'s `last_status != ok` (or on "no audit
commit in >24 h") is the only check that can catch it.
- **Alert on the artefact.** Verifying that a dated `audit(YYYY-MM-DD)` commit exists in this repo would
have caught both this failure and the stocktake's zero-edit days — the same lesson recorded on
[[incidents/2026-11-09-stocktake-completes-with-zero-edits]].
## Related
- [[systems/scheduled-tasks]]
- [[systems/omniroute]]
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]