audit(2026-11-11): the audit's own 429 killed the 11-10 window; stocktake + radar recover with output
Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed). NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2 RESOLVED with output (dated Recovery sections added): - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B) - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md (opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x), incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
This commit is contained in:
@@ -2467,3 +2467,13 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
|
||||
- **Estate shrank again (probed, not inferred):** `qm list` on `pve` now holds only VMs **100, 101, 110, 282** (VM **310** gone; the wiki had listed it) and `pct list` holds 17 CTs with **CT 601 gone**. VM 403/VM 401 and the `open-claw` Tailscale peer remain absent on both hypervisors → [[agents/open-claw]], [[agents/index]], `CONTRIBUTING.md`.
|
||||
- **Delivery verified at the destination, not assumed:** before the commit, `http://100.118.5.51:9120/` and the changed pages (incl. `runbooks/test-openrouter`, `concepts/model-routing`, `incidents/2026-08-29-gmail-read-state`) all returned **200**, as did the external route `https://hermes.kangaroo-eel.ts.net/wiki/`. `CONTRIBUTING.md`'s access table was re-verified the same way and its `agents/open-claw/` scope row struck through ([[CONTRIBUTING]]).
|
||||
- **No page deleted, no decision reversed, no secrets written.** No credential value appears in any page (the Gitea remote's embedded token is deliberately not reproduced). One pre-existing broken wikilink left as-is: `systems/guanaco.md` → `[[systems/ollama]]` (present in HEAD, page already qualifies it "if exists").
|
||||
|
||||
## 2026-11-12 00:1x AWST - Nightly wiki audit (window 2026-11-10 23:59 -> 2026-11-11 23:59)
|
||||
|
||||
- **The audit's own failure is the headline.** The 2026-11-10 23:59 slot (`cbfe434a3b5e`) ran **12 m 35 s** (23:59:54.486572 -> 2026-11-11 00:12:29.646987) and died on the router's pre-dispatch filter: `HTTP 429 ... all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` — **6 candidates, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded, delivery `suppressed`. **Consequence: the 2026-11-10 window was never reviewed.** Recovery was implicit (pool reset ≈ **06:36:57**; the 07:0x cohort ran clean), and this audit deliberately covers **both** windows. New page: [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]; new index entry.
|
||||
- **The raised row was never alerted.** `cbfe43_ed01845310d8` (`rate_limit`) sits at `detected` with **`alerted_at: NULL`, `closed_at: NULL`** — the **only open row** in `cron_incidents` (the previous one, `5fc292_773822455856`, closed `resolved` at 08:03:47). Fleet non-clean `last_status` **1 -> 2** (`066bb5191280` Uber Eats `delivery_failed` streak 0, plus `cbfe434a3b5e` `error` streak 1).
|
||||
- **Both long-running incidents recovered, with output.** Stocktake `1a8e90ea2271` **03:00:56.081504 -> 04:11:38.724852 = 1 h 10 m 42 s**, `completed`, **36 stale pages re-verified -> 0 stale**, commits **`341dda9`** + **`c108eb1`** (freshness report `raw/stocktake/freshness-20261111.txt`, 6,235 B) — closing [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]. Job Radar Stage 1 `dca8482e4f76` **07:01:08.358004 -> 07:37:34.076224 = 36 m 26 s**, writing the **first handoff since 2026-11-06** (`radar-ref/handoff/2026-11-11.json`, 22,682 B, mtime 07:59); Stage 2 `5fc292ff0f3b` recovery fire 07:35:56.632032 -> 08:03:47.816811 `delivered`, then 11:30:56 -> 11:35:18 and 14:30:55 -> 14:34:56 `delivered`, auto-closing the 11-10 row `resolved` — closing [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]. Both incident pages set to `status: resolved` with a dated Recovery section.
|
||||
- **Ledger, both days re-parsed (times stored as ISO TEXT, grouped by `substr(started_at,1,10)`).** 2026-11-09 **174 completed / 0 failed**; **2026-11-10: 157 rows — 155 `completed` / 2 `failed`** (Radar S2 scheduled 11:30:35.267155 -> 22:18:49.814215 = **10 h 48 m 14 s** `interrupted` with `for_failure: 1`; the audit's own 12 m 35 s 429); **2026-11-11: 172 rows — 171 `completed` / 1 `running` / 0 `failed`**, 15 distinct jobs, **first zero-failure day since 11-09**. Longest 11-11 run **1 h 10 m 42 s** — **nothing over 71 minutes**, against 9 h 09 m (11-09) and 10 h 48 m (11-10). Fleet **29 (26 enabled / 3 disabled)** for a sixth window.
|
||||
- **Deliveries checked at the destination:** 2026-11-11 = **7 rows, 7 / 7 `delivered`, 0 `for_failure`** (07:11:42, 07:31:01, 08:03:47, 08:34:36, 12:19:38, 14:34:55, 18:45:58); 2026-11-10's late row (22:18:48) is the one `for_failure: 1`. 2026-11-12 so far: 5 rows, 4 `completed` / 1 `running`.
|
||||
- **Providers unchanged; symptoms worse.** `config.yaml` mtime still **2026-11-09 22:58:07** — nothing repaired this window. The expired `opencode-zen` key at `fallback_providers[1]` produced **7 x HTTP 401 `api_key_expired`** on 11-11 (07:01:13 -> 19:19:39), each **immediately after** nous returned **14 x 404 `insufficient_credits_for_paid_model`** (07:01:11 -> last 19:19:38, all from the still-open desktop session `20261031_190813_95fbde`). **Two consecutive dead hops on one chain** — [[systems/opencode-go]] updated from "latent risk" to observed failure; [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] gained the 11-11 escalation block. `auxiliary.pulse_memory` remains the **only slot with no `fallback_chain`** and logged **19 lines / 11 refusals** on 11-11.
|
||||
- **Pages written:** 1 new incident; 2 incidents -> `resolved`; 2 system pages + `current-state.md` + `index.md` updated; `systems/scheduled-tasks.md` gained the 2026-11-11 count/state normalisation. `~/.hermes/logs/errors.log` is now **6,179 lines** through **2026-11-12 00:10:11**. **No page deleted, no decision reversed, no secrets written** — no credential value appears in any page; the opencode key is recorded by *state*, never by value.
|
||||
|
||||
Reference in New Issue
Block a user