audit(2026-11-11): the audit's own 429 killed the 11-10 window; stocktake + radar recover with output

Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed).

NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
 - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed
 - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6)
   reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded
 - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row
 - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2

RESOLVED with output (dated Recovery sections added):
 - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0
 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B)
 - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved

UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md
(opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x),
incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
This commit is contained in:
Hermes Wiki Audit
2026-11-12 00:16:39 +08:00
parent c108eb1012
commit 729bb597cf
11 changed files with 463 additions and 20 deletions
+62 -4
View File
@@ -3,16 +3,74 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-11-10
verified_on: 2026-11-10
review_after: 2026-11-10
updated: 2026-11-11
verified_on: 2026-11-11
review_after: 2026-11-12
confidence: high
tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
---
# Current State
Updated: 2026-11-10
Updated: 2026-11-11
## Recent Changes (2026-11-11, nightly audit; live probes 2026-11-11 23:59 – 2026-11-12 00:1x AWST)
**Recovery night — plus one failure, and it was the auditor's own.**
→ [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]
- **The 2026-11-10 audit died on its own 429.** `cbfe434a3b5e` ran 2026-11-10T23:59:54.486572 →
**2026-11-11T00:12:29.646987 = 12 m 35 s**, `failed`, delivery `suppressed`:
`RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch
filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` — **6 candidates, `attempted 0`**, tenant
`tn-5ud77abtl1` quota exceeded. Row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 but
**`alerted_at: NULL`, `closed_at: NULL` — it is the estate's only open incident row**. Pool reset
≈ **06:36:57**; the 07:0x cohort ran clean.
⇒ **The 2026-11-10 window was never audited.** This audit covers two windows.
- **State reads:** fleet **29 (26 enabled / 3 disabled)**; **non-clean `last_status` 2** (was 1) —
`066bb5191280` Uber Eats (`delivery_failed`, streak 0) and `cbfe434a3b5e` (`error`, streak 1).
Ledger — **2026-11-09: 174 completed / 0 failed**; **2026-11-10: 157 rows, 155 completed / 2 failed**;
**2026-11-11: 172 rows, 171 completed / 1 running / 0 failed** (15 distinct jobs) — first zero-failure
day since 11-09. **Deliveries 11-11: 7 rows, 7 / 7 `delivered`, 0 `for_failure`** (last at 18:45:58).
- **Stocktake recovered with output.** `1a8e90ea2271` 03:00:56.081504 → **04:11:38.724852 = 1 h 10 m
42 s**, `completed`, **36 stale pages re-verified → 0 stale**, **commits `341dda9` + `c108eb1`**,
freshness report `raw/stocktake/freshness-20261111.txt` (6,235 B). `341dda9` also rewrote the live
fallback chain in [[runbooks/test-openrouter]] (primary back to `smart-route` / `custom:omni-101`),
corrected the CT333 `clawtest` naming claim, and replaced the nous refresh-token reading with the
404-credits signature. Closes [[incidents/2026-11-09-stocktake-completes-with-zero-edits]].
⚠ The job is **still pinned to `free-stack`**, the combo recorded broken on 2026-08-14, and succeeded.
- **Job Radar recovered with output.** Stage 1 `dca8482e4f76` 07:01:08.358004 → **07:37:34.076224 =
36 m 26 s**, `completed`, writing **`radar-ref/handoff/2026-11-11.json` (22,682 B, mtime 07:59)** — the
first handoff since **2026-11-06**. Stage 2 `5fc292ff0f3b`: recovery fire 07:35:56.632032 →
08:03:47.816811 (`delivered`), scheduled 11:30:56 → 11:35:18, 14:30:55 → 14:34:56 (`delivered`). Row
`5fc292_773822455856` auto-closed **`resolved` 2026-11-11T08:03:47.644226**, closing
[[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]].
- **Longest runs.** 11-11: stocktake **1 h 10 m 42 s**, Radar S1 **36 m 26 s**, Radar S2 **27 m 51 s**
(recovery fire), Newsletter Digest `ecc1fc13655a` 12:01:02.571781 → 12:19:38.965387 (**18 m 36 s**),
`85edf6772bca` 07:01:08.287627 → 07:11:42.858722 (**10 m 34 s**). **Nothing over 71 minutes.**
11-10: Radar S2 scheduled slot 11:30:35.267155 → 22:18:49.814215 = **10 h 48 m 14 s**, `failed` /
`interrupted`, delivery `delivered` with `for_failure: 1`.
- **Provider state (unchanged config; worsening symptoms).** `config.yaml` mtime still **2026-11-09
22:58:07** — nothing repaired. Top model `custom:omni-101` / `smart-route`
(`http://100.78.63.20:20128`). `fallback_providers` (6): nous `meituan/longcat-2.5-preview:free` →
**opencode-zen `deepseek-v4.1-flash` (EXPIRED key)** → `custom:guacano` → `custom:litellm`
`kilo-reflection-beam-free` → kilocode `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`)
`or-inclusionai_ling-2_6-1t`. **The expired opencode-zen key is no longer latent** — **7 × HTTP 401
`api_key_expired`** on 11-11 (07:01:13 … 19:19:39), each immediately after nous returned **14 × 404
`insufficient_credits_for_paid_model`** (07:01:11 … last 19:19:38, all from session
`20261031_190813_95fbde`, source `desktop`, opened 2026-10-31, **still open**, pinned to that paid
model). **Two consecutive dead hops on one chain.** → [[systems/opencode-go]]
- **Auxiliary slots:** nine slots plus `free_only: true`. Eight chain
`no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` on `custom:omni-101`.
**`pulse_memory` remains the sole exception with no `fallback_chain`**
(`{"provider": "nous", "model": "anthropic/claude-haiku-4.5"}`) — 19 aux log lines on 11-11 across
06:0x / 08:0x / 13:0x / 18:0x, including **11 ×** *"main provider `custom:omni-101` is unavailable and
no fallback_chain / fallback_providers is configured — refusing to guess another logged-in provider"*.
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
- **Error log:** `~/.hermes/logs/errors.log` = **6,179 lines** through **2026-11-12 00:10:11**.
- **Still open, untouched this window:** Uber Eats Monthly Spend Report `delivery_failed` streak 0 (no
execution row on 11-11 — monthly cadence); disabled trio unchanged (Important-mail monitor, Rhino
AgentMail Inbox Watcher, AgentMail Rhino Inbox Poll); the three inherited items below.
## Recent Changes (2026-11-10, nightly audit; live probes 2026-11-09 23:5x – 2026-11-10 01:0x AWST)