Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed). NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2 RESOLVED with output (dated Recovery sections added): - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B) - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md (opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x), incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
5.2 KiB
5.2 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited | incident | open | 2026-11-11 | 2026-11-11 | 2026-11-11 | 2026-12-11 | high |
|
|
Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited (2026-11-10 23:59 → 2026-11-11 00:12)
This job is the estate's own observer, and on 2026-11-10 it became the estate's only failure. Its 23:59 slot never produced a report, so no page, no log entry and no incident alert describes the day of 2026-11-10 — an unaudited day is invisible by construction.
What happened
| Field | Value |
|---|---|
| Job | cbfe434a3b5e — Wiki Audit — Nightly, custom:omni-101 / deepseek-paid-smart |
| Slot | 2026-11-10 23:59:54.486572 +08:00 (cron 59 23 * * *) |
| Ended | 2026-11-11 00:12:29.646987 +08:00 — 12 m 35 s |
| Status | failed · delivery_outcome: suppressed |
| Error | RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s |
| Output | ~/.hermes/cron/output/cbfe434a3b5e/2026-11-11_00-12-28.md |
The three log lines from the run's own session local-1000c1a3… show the shape of the failure:
ERROR agent.chat_completion_helpers: Streaming failed before delivery: [429]: service failure: tenant tn-5ud77abtl1 quota exceeded
ERROR agent.chat_completion_helpers: Streaming failed before delivery: Error code: 429 -
ERROR agent.conversation_loop: API call failed after 3 retries. HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters
ERROR cron.scheduler: Job 'Wiki Audit — Nightly' failed: RuntimeError: HTTP 429: Service temporarily unavailable
- The pool skipped every target. Six candidates were evaluated and
attempted 0— the run never reached a provider. The only live target's credit balance read 100 % consumed on connectionconn:3021bab6, which is why the router refused to dispatch rather than serving a degraded model. reset after 6h 24m 31sputs the pool's recovery at ≈ 2026-11-11 06:36:57 AWST. The 07:0x cron cohort ran clean the next morning, so the window closed on its own.- The same tenant (
tn-5ud77abtl1) appears in thequota exceededline, so this is an account-level quota fault on the router, not a single-model lockout.
Incident row — raised, never alerted, still open
cbfe43_ed01845310d8 — job_id cbfe434a3b5e, failure_type: rate_limit, state detected,
alerted_at: NULL, closed_at: NULL, opened 2026-11-11T00:12:28.578435+08:00.
- It is the only open row in
cron_incidents(135 rows total) — the previous open item,5fc292_773822455856, closedresolvedat 2026-11-11T08:03:47.644226. - So the estate's single outstanding incident is the audit's own failure, and nothing was told about it.
Impact
- The 2026-11-10 window was never reviewed. The nightly audit is the only producer of the per-day ledger/wiki record; its failure means 11-10's 157 ledger rows were never reconciled on the day.
- Fleet non-clean
last_status1 → 2.cbfe434a3b5enow readslast_status: error,failure_streak: 1, joining066bb5191280(Uber Eats Monthly Spend Report,delivery_failed, streak 0). - The review published on 2026-11-11 23:59 carries both windows — it is the recovery, not a duplicate.
Same signature, earlier appearances
- incidents/2026-10-21-omniroute-pre-dispatch-filter-skip — the identical pre-dispatch-filter
string killed the Daily Marketing Job Radar on 2026-10-21 (
attempted 0, no 503 anywhere in the day). - 2026-11-05 —
deepseek-paid-smartagainst the CT600 endpoint returnedHTTP 503 ALL_TARGETS_SKIPPED(poolSize 5,attempted 0) whileomniroute-1answered 200 — recorded on systems/scheduled-tasks. - This is the third distinct symptom of the same router behaviour:
attempted 0with a healthy node behind it is the thing to recognise, not the specific HTTP code.
Fix Applied
None — this is a detection-and-verification record. No configuration changed this window. The run's own recovery was the pool reset at ≈ 06:36:57 on 2026-11-11.
Prevention
- The observer needs an observer. A failed nightly audit raises no delivery (suppressed), no alert
(
alerted_at: NULL) and no page — every layer that would normally surface this is inside the thing that failed. A one-line watchdog onWiki Audit — Nightly'slast_status != ok(or on "no audit commit in >24 h") is the only check that can catch it. - Alert on the artefact. Verifying that a dated
audit(YYYY-MM-DD)commit exists in this repo would have caught both this failure and the stocktake's zero-edit days — the same lesson recorded on incidents/2026-11-09-stocktake-completes-with-zero-edits.