Files
agent-estate-wiki/incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
T
Tony0410 e033a08f17 audit(2026-11-12): the estate's last incident closed itself; a past-0.9.42 hq-push install was refused by the plugin scan (fix parked in PR #2)
- 429 row cbfe43_ed01845310d8 closed resolved 2026-11-12T00:25:36 on the next clean slot; 0 of 137 unclosed
- 11-12: 176 rows / 175 completed / 0 failed / 1 running; 11-11: 172/172/0; deliveries 9/9 and 7/7 delivered
- NEW incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md (dangerous verdict, one critical in the
  plugin's own tile skill; --force cannot override; no per-verdict override exists)
- systems/hermes-hq-push.md corrected live (0.9.27 -> 0.9.42, tree mtime 2026-11-08 02:19, config line 13226)
- current-state / scheduled-tasks / index / log updated; skills-index regenerated; freshness 2026-11-13 (0 stale)
2026-11-13 00:41:15 +08:00

7.0 KiB

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited incident resolved 2026-11-11 2026-11-12 2026-11-12 2026-12-11 high
cron
incident
omniroute
rate-limit
audit
live-executions-db
live-cron-incidents
gateway-log
live-jobs-json

Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited (2026-11-10 23:59 → 2026-11-11 00:12)

This job is the estate's own observer, and on 2026-11-10 it became the estate's only failure. Its 23:59 slot never produced a report, so no page, no log entry and no incident alert describes the day of 2026-11-10 — an unaudited day is invisible by construction.

What happened

Field Value
Job cbfe434a3b5e — Wiki Audit — Nightly, custom:omni-101 / deepseek-paid-smart
Slot 2026-11-10 23:59:54.486572 +08:00 (cron 59 23 * * *)
Ended 2026-11-11 00:12:29.646987 +08:00 — 12 m 35 s
Status failed · delivery_outcome: suppressed
Error RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s
Output ~/.hermes/cron/output/cbfe434a3b5e/2026-11-11_00-12-28.md

The three log lines from the run's own session local-1000c1a3… show the shape of the failure:

ERROR agent.chat_completion_helpers: Streaming failed before delivery: [429]: service failure: tenant tn-5ud77abtl1 quota exceeded
ERROR agent.chat_completion_helpers: Streaming failed before delivery: Error code: 429 -
ERROR agent.conversation_loop: API call failed after 3 retries. HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters
ERROR cron.scheduler: Job 'Wiki Audit — Nightly' failed: RuntimeError: HTTP 429: Service temporarily unavailable
  • The pool skipped every target. Six candidates were evaluated and attempted 0 — the run never reached a provider. The only live target's credit balance read 100 % consumed on connection conn:3021bab6, which is why the router refused to dispatch rather than serving a degraded model.
  • reset after 6h 24m 31s puts the pool's recovery at ≈ 2026-11-11 06:36:57 AWST. The 07:0x cron cohort ran clean the next morning, so the window closed on its own.
  • The same tenant (tn-5ud77abtl1) appears in the quota exceeded line, so this is an account-level quota fault on the router, not a single-model lockout.

Incident row — raised, never alerted, closed resolved 2026-11-12T00:25:36

cbfe43_ed01845310d8 — job_id cbfe434a3b5e, failure_type: rate_limit, state detected, alerted_at: NULL, acked_at: NULL, closed_at: NULL, opened 2026-11-11T00:12:28.578435+08:00 (first_seen_at = last_seen_at = that instant — seen exactly once).

  • It was the only open row in cron_incidents — the table held 135 rows at audit time and 137 by the end of this run. acked_at is NULL as well as alerted_at, so nothing was told and nothing was acknowledged. The previous open item, 5fc292_773822455856, closed resolved at 2026-11-11T08:03:47.644226.
  • So the estate's single outstanding incident was the audit's own failure, and nothing was told about it. ⚠ Superseded 2026-11-12 — the row closed resolved at 2026-11-12T00:25:36.133387+08:00; cron_incidents now holds 0 unclosed of 137. See Resolution below.

Resolution — 2026-11-12T00:25:36, self-cleared

The row closed resolved at 2026-11-12T00:25:36.133387+08:00, at the end of the next slot of the same job: cbfe434a3b5e ran 2026-11-11T23:59:54.486572 → 2026-11-12T00:25:36.133387 = 25 m 42 s and completed. Its delivery carried both windows, so 2026-11-10's day was reconciled a day late rather than never (the recovery record is the audit(2026-11-11) commit series at 2026-11-12 00:1x-00:24).

Nothing was repaired to make this happen: the pool's own reset at ≈ 2026-11-11 06:36:57 restored the dispatch path, exactly as the reset after 6h 24m 31s field predicted, and the 07:0x cohort ran clean. cron_incidents now reads 0 unclosed of 137 rows — the estate is incident-clean as of 2026-11-12.

The recurrence risk is unchanged. The same "all targets were skipped by pre-dispatch filters" signature was found again on a second job and a second pin by the 2026-11-12 03:00 stocktake (incidents/2026-10-21-omniroute-pre-dispatch-filter-skip), and every model-calling job still rides custom:omni-101 with no failover for the free-tier cohort. Closing this row proves the guard self-heals; it is not evidence the guard is gone.

Impact

  • The 2026-11-10 window was never reviewed. The nightly audit is the only producer of the per-day ledger/wiki record; its failure means 11-10's 157 ledger rows were never reconciled on the day.
  • Fleet non-clean last_status 1 → 2. cbfe434a3b5e now reads last_status: error, failure_streak: 1, joining 066bb5191280 (Uber Eats Monthly Spend Report, delivery_failed, streak 0).
  • The review published on 2026-11-11 23:59 carries both windows — it is the recovery, not a duplicate.

Same signature, earlier appearances

  • incidents/2026-10-21-omniroute-pre-dispatch-filter-skip — the identical pre-dispatch-filter string killed the Daily Marketing Job Radar on 2026-10-21 (attempted 0, no 503 anywhere in the day).
  • 2026-11-05 — deepseek-paid-smart against the CT600 endpoint returned HTTP 503 ALL_TARGETS_SKIPPED (poolSize 5, attempted 0) while omniroute-1 answered 200 — recorded on systems/scheduled-tasks.
  • This is the third distinct symptom of the same router behaviour: attempted 0 with a healthy node behind it is the thing to recognise, not the specific HTTP code.

Fix Applied

None — this is a detection-and-verification record. No configuration changed this window. The run's own recovery was the pool reset at ≈ 06:36:57 on 2026-11-11, and the incident row closed on the next slot's clean completion (2026-11-12T00:25:36).

Prevention

  • The observer needs an observer. A failed nightly audit raises no delivery (suppressed), no alert (alerted_at: NULL) and no page — every layer that would normally surface this is inside the thing that failed. A one-line watchdog on Wiki Audit — Nightly's last_status != ok (or on "no audit commit in >24 h") is the only check that can catch it.
  • Alert on the artefact. Verifying that a dated audit(YYYY-MM-DD) commit exists in this repo would have caught both this failure and the stocktake's zero-edit days — the same lesson recorded on incidents/2026-11-09-stocktake-completes-with-zero-edits.