audit(2026-11-12): the estate's last incident closed itself; a past-0.9.42 hq-push install was refused by the plugin scan (fix parked in PR #2)
- 429 row cbfe43_ed01845310d8 closed resolved 2026-11-12T00:25:36 on the next clean slot; 0 of 137 unclosed - 11-12: 176 rows / 175 completed / 0 failed / 1 running; 11-11: 172/172/0; deliveries 9/9 and 7/7 delivered - NEW incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md (dangerous verdict, one critical in the plugin's own tile skill; --force cannot override; no per-verdict override exists) - systems/hermes-hq-push.md corrected live (0.9.27 -> 0.9.42, tree mtime 2026-11-08 02:19, config line 13226) - current-state / scheduled-tasks / index / log updated; skills-index regenerated; freshness 2026-11-13 (0 stale)
This commit is contained in:
@@ -1,10 +1,10 @@
|
||||
---
|
||||
title: Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited
|
||||
type: incident
|
||||
status: open
|
||||
status: resolved
|
||||
created: 2026-11-11
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
updated: 2026-11-12
|
||||
verified_on: 2026-11-12
|
||||
review_after: 2026-12-11
|
||||
confidence: high
|
||||
tags: [cron, incident, omniroute, rate-limit, audit]
|
||||
@@ -43,18 +43,36 @@ The three log lines from the run's own session `local-1000c1a3…` show the shap
|
||||
- The **same tenant** (`tn-5ud77abtl1`) appears in the `quota exceeded` line, so this is an
|
||||
account-level quota fault on the router, not a single-model lockout.
|
||||
|
||||
## Incident row — raised, never alerted, still open
|
||||
## Incident row — raised, never alerted, closed `resolved` 2026-11-12T00:25:36
|
||||
|
||||
`cbfe43_ed01845310d8` — `job_id cbfe434a3b5e`, `failure_type: rate_limit`, state **`detected`**,
|
||||
`alerted_at: NULL`, `acked_at: NULL`, `closed_at: NULL`, opened **2026-11-11T00:12:28.578435+08:00**
|
||||
(`first_seen_at` = `last_seen_at` = that instant — seen exactly once).
|
||||
|
||||
- It is the **only open row** in `cron_incidents` — the table held **135** rows at audit time and **137**
|
||||
- It **was** the only open row in `cron_incidents` — the table held **135** rows at audit time and **137**
|
||||
by the end of this run. `acked_at` is **NULL** as well as `alerted_at`, so nothing was told and
|
||||
nothing was acknowledged. The previous open item,
|
||||
`5fc292_773822455856`, closed `resolved` at **2026-11-11T08:03:47.644226**.
|
||||
- So the estate's single outstanding incident is **the audit's own failure**, and nothing was told
|
||||
about it.
|
||||
- So the estate's single outstanding incident **was** the audit's own failure, and nothing was told
|
||||
about it. ⚠ **Superseded 2026-11-12** — the row closed `resolved` at **2026-11-12T00:25:36.133387+08:00**;
|
||||
*cron_incidents* now holds **0 unclosed of 137**. See *Resolution* below.
|
||||
|
||||
## Resolution — 2026-11-12T00:25:36, self-cleared
|
||||
|
||||
The row closed **`resolved`** at **2026-11-12T00:25:36.133387+08:00**, at the end of the **next** slot of
|
||||
the same job: `cbfe434a3b5e` ran **2026-11-11T23:59:54.486572 → 2026-11-12T00:25:36.133387 = 25 m 42 s**
|
||||
and completed. Its delivery carried **both windows**, so 2026-11-10's day was reconciled a day late
|
||||
rather than never (the recovery record is the `audit(2026-11-11)` commit series at 2026-11-12 00:1x-00:24).
|
||||
|
||||
Nothing was repaired to make this happen: the pool's own reset at ≈ **2026-11-11 06:36:57** restored the
|
||||
dispatch path, exactly as the `reset after 6h 24m 31s` field predicted, and the 07:0x cohort ran clean.
|
||||
`cron_incidents` now reads **0 unclosed of 137** rows — the estate is incident-clean as of 2026-11-12.
|
||||
|
||||
**The recurrence risk is unchanged.** The same *"all targets were skipped by pre-dispatch filters"*
|
||||
signature was found again on a **second job and a second pin** by the 2026-11-12 03:00 stocktake
|
||||
([[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]), and every model-calling job still rides
|
||||
`custom:omni-101` with no failover for the free-tier cohort. Closing this row proves the guard
|
||||
self-heals; it is not evidence the guard is gone.
|
||||
|
||||
## Impact
|
||||
|
||||
@@ -79,7 +97,8 @@ The three log lines from the run's own session `local-1000c1a3…` show the shap
|
||||
## Fix Applied
|
||||
|
||||
**None — this is a detection-and-verification record.** No configuration changed this window. The run's
|
||||
own recovery was the pool reset at ≈ 06:36:57 on 2026-11-11.
|
||||
own recovery was the pool reset at ≈ 06:36:57 on 2026-11-11, and the incident row closed on the next
|
||||
slot's clean completion (2026-11-12T00:25:36).
|
||||
|
||||
## Prevention
|
||||
|
||||
|
||||
Reference in New Issue
Block a user