audit(2026-11-11): the audit's own 429 killed the 11-10 window; stocktake + radar recover with output

Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed).

NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
 - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed
 - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6)
   reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded
 - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row
 - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2

RESOLVED with output (dated Recovery sections added):
 - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0
 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B)
 - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved

UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md
(opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x),
incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
This commit is contained in:
Hermes Wiki Audit
2026-11-12 00:16:39 +08:00
parent c108eb1012
commit 729bb597cf
11 changed files with 463 additions and 20 deletions
@@ -66,7 +66,31 @@ sources: [gateway-log, live-config, live-probe]
> currently erroring), and/or add credits for paid nous models. The original re-authenticate item
> is **superseded** — auth works now.
> **Escalated 2026-11-11 (nightly audit).** The single-slot exposure described above became the
> busiest error path on the host, and it now has a **downstream casualty**.
>
> - `anthropic/claude-haiku-4.5` on `nous` returned **404 `insufficient_credits_for_paid_model`**
> **14 times** on 2026-11-11 (07:01:11 … last **19:19:38**) — up from 3 in the whole preceding
> window. All from session `20261031_190813_95fbde` (**source `desktop`, started 2026-10-31
> 19:08:13, still open**), whose pinned model is that same paid nous model.
> - The **404 is immediately followed by a 401.** The session's fallback chain moves the turn on to
> `opencode-zen` / `deepseek-v4.1-flash`, whose key expired on 2026-11-08 — **7 × HTTP 401
> `api_key_expired`** on 11-11 (07:01:13, 07:19:30, 08:03:40, 13:03:28, 13:19:33, 18:03:39,
> 19:19:39). So **both hops of this chain are dead**, and the pair repeats every time the session is
> poked. → [[systems/opencode-go]]
> - `auxiliary.pulse_memory` logged **19 lines** on 11-11 (06:0x, 08:0x, 13:0x, 18:0x), including
> **11 ×** *"main provider `custom:omni-101` is unavailable and no fallback_chain /
> fallback_providers is configured — refusing to guess another logged-in provider."* Live config
> re-confirmed at audit time: `pulse_memory` is the **only** slot carrying **no `fallback_chain`**
> (`{"provider": "nous", "model": "anthropic/claude-haiku-4.5"}`), while the other eight all chain
> `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` on `custom:omni-101`.
> - **Net effect:** pulse/voice memory retrieval failed on every attempt this window. `config.yaml`
> mtime is unchanged since **2026-11-09 22:58:07** — nothing was fixed.
>
> **Open item, unchanged and now evidenced:** move `auxiliary.pulse_memory` onto `custom:omni-101`
> like the other eight slots and give it a `fallback_chain`, and re-point or retire the stale desktop
> session pinned to a paid nous model. No credentials were minted or modified.
## Related
- [[systems/model-providers]]
- [[systems/current-profile]]
- [[systems/hermes-agent]]
@@ -1,10 +1,10 @@
---
title: "2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op"
type: incident
status: active
status: resolved
created: 2026-11-10
updated: 2026-11-10
verified_on: 2026-11-10
updated: 2026-11-11
verified_on: 2026-11-11
confidence: high
tags: [cron, job-radar, stall, silent-failure, handoff, read-loop]
sources: ["executions.db (dca8482e4f76: 2026-11-09T07:00:48+08:00 → 2026-11-09T16:10:12+08:00, status completed, delivery_outcome suppressed; 5fc292ff0f3b: 11:30:38 → 12:10:07 and 14:30:34 → 14:37:27, both completed/suppressed)", "~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md (ends INCOMPLETE: tool budget exhausted; last words \\u201cI stopped retrying because I kept running read_file 5 times without making progress.\\u201d)", "~/.hermes/radar-ref/handoff/ (newest handoff on disk: 2026-11-06; no 2026-11-07/08/09 files)", "cron_incidents (no row raised for this run)"]
@@ -57,6 +57,31 @@ Two compounding faults, both previously documented for this job:
- Add a **read-progress guard** to Stage 1: when the same file has been read N times without a state change, stop and emit a partial handoff instead of burning the budget to `INCOMPLETE`.
- Note the pattern across incidents: **completion-time-only alerting hides every failure in this family** (stocktake 11-09, radar 11-09, radar 11-02/03).
## Recovery — 2026-11-11: the handoff landed, and Stage 2 delivered
| Field | 11-09 (this incident) | **11-11** |
|---|---|---|
| Stage 1 | 07:00:48 → 16:10:12 (**9 h 09 m 24 s**), no handoff | **07:01:08 → 07:37:34 = 36 m 26 s**, `completed` |
| Handoff on disk | newest was **2026-11-06** | **`radar-ref/handoff/2026-11-11.json` — 22,682 B, mtime 07:59** |
| Stage 2 (11:30 slot) | fired early, declined silently | **11:30:56 → 11:35:18**, `completed` |
| Stage 2 (14:30 slot) | declined silently | **14:30:55 → 14:34:56**, `completed` / **`delivered`** |
| Incident row | **none raised** | `5fc292_773822455856` closed **`resolved` 2026-11-11T08:03:47.644226** |
- **The first handoff in five days** (`2026-11-06` → `2026-11-11`) and the fastest Stage-1 cycle since
the two-stage split: **36 m 26 s** against 3 h 15 m (10-29, the last full success), 9 h 01 m (11-02),
6 h 28 m (11-03) and 9 h 09 m (11-09).
- **Stage 2's recovery fire** (`source: direct`, 07:35:56 → 08:03:47) consumed the handoff and delivered
at **08:03:47**; its completion is what auto-closed the row opened by 11-10's `interrupted` slot.
- ⚠ **Carried forward from 11-10:** the *scheduled* Stage-2 slot that day ran
**11:30:35 → 22:18:49 = 10 h 48 m 14 s** and closed **`failed` / `interrupted`** (delivery
`delivered`, `for_failure: 1`) — the longest run of either window, and itself a new data point for the
"completion-time-only alerting" family on this page.
- **Job state:** both radar jobs read `last_status: ok`, `failure_streak` 0.
**Status: `resolved`.** Both prevention rules on this page still stand and remain unimplemented: Stage 2
should hard-fail on a missing `handoff/<date>.json` rather than suppressing, and Stage 1 needs a
read-progress guard. Neither was needed on 11-11 because the run finished inside budget.
## Related
- [[systems/scheduled-tasks]]
@@ -1,10 +1,10 @@
---
title: "2026-11-09 — Stocktake Completes With Zero Edits"
type: incident
status: active
status: resolved
created: 2026-11-10
updated: 2026-11-10
verified_on: 2026-11-10
updated: 2026-11-11
verified_on: 2026-11-11
confidence: high
tags: [cron, wiki, stocktake, tool-budget, silent-failure, omni-101, free-stack]
sources: ["executions.db (1a8e90ea2271: 2026-11-09T03:00:43+08:00 → 2026-11-09T12:10:47+08:00, status completed, delivery_outcome suppressed)", "~/.hermes/cron/output/1a8e90ea2271/2026-11-09_12-10-46.md (8,305 bytes; RUN INCOMPLETE, \\u201ctool-iteration budget exhausted during verification\\u201d)", "cron_incidents (1a8e90_628d914f1f97, 1a8e90_2f370ef764f5, 1a8e90_a9cad99cb150 — all closed 2026-11-09T12:10:47.125205+08:00)", "wiki-stocktake-maintenance job 1a8e90ea2271 (provider custom:omni-101, model free-stack)"]
@@ -57,6 +57,35 @@ Also recorded this window: the `free-stack` pin on `1a8e90ea2271` — the only d
- **Alerting must fire on write progress, not completion.** This run produced 2 of 3 failure days silently and the third day produced a run that reported success while editing nothing. A check on `commit_count` / `edits_applied` for this job would have caught all three.
- Don't read `status`/`failure_streak` as the stocktake's health. Read whether a commit landed.
## Recovery — 2026-11-11: the pass ran in 1 h 10 m and landed 36 pages
The next scheduled slot **worked**, and worked quickly:
| Field | 11-09 (this incident) | **11-11** |
|---|---|---|
| Window | 03:00:43 → 12:10:47 | **03:00:56.081504 → 04:11:38.724852** |
| Duration | **9 h 10 m** | **1 h 10 m 42 s** |
| Terminal state | `completed` / `suppressed` | `completed` / `suppressed` |
| Edits landed | **0** | **36 stale pages re-verified → 0 stale** |
| Commit | **none** | **`341dda9`** + **`c108eb1`** |
| Freshness report | — | `raw/stocktake/freshness-20261111.txt` (**6,235 B**) |
- `341dda9` re-verified 30+ further pages live and de-flagged them, and applied real content
corrections — the live fallback chain rewritten in [[runbooks/test-openrouter]] (primary back to
`smart-route` / `custom:omni-101`), the CT333 `clawtest` naming claim corrected, and the nous
**404 `insufficient_credits_for_paid_model`** signature replacing the old refresh-token reading in
[[incidents/2026-10-22-nous-portal-refresh-token-invalid]].
- `c108eb1` regenerated the freshness report after the log/index addendum, so the published Quartz build
matches HEAD.
- **Job state:** `last_status: ok`, `failure_streak` 0, `last_run` 2026-11-11T03:00:56.
⚠ The job is **still pinned to `custom:omni-101` / `free-stack`** — the combo recorded broken on
2026-08-14 — and this run succeeded on it anyway. That is an observation, not a repin.
**Status: `resolved`.** The 11-07/11-08/11-09 run of failures is closed by output, not by a cosmetic
`failure_streak` reset. The page's own prevention rule stands: *alert on `commit_count` / `edits_applied`,
never on `status`* — this recovery is exactly why it matters, because 11-09 and 11-11 both read
`completed` / `suppressed` and only one of them did anything.
## Related
- [[systems/scheduled-tasks]]
@@ -0,0 +1,96 @@
---
title: Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited
type: incident
status: open
created: 2026-11-11
updated: 2026-11-11
verified_on: 2026-11-11
review_after: 2026-12-11
confidence: high
tags: [cron, incident, omniroute, rate-limit, audit]
sources: [live-executions-db, live-cron-incidents, gateway-log, live-jobs-json]
---
# Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited (2026-11-10 23:59 → 2026-11-11 00:12)
**This job is the estate's own observer, and on 2026-11-10 it became the estate's only failure.** Its
23:59 slot never produced a report, so **no page, no log entry and no incident alert describes the day
of 2026-11-10** — an unaudited day is invisible by construction.
## What happened
| Field | Value |
|---|---|
| Job | `cbfe434a3b5e` — **Wiki Audit — Nightly**, `custom:omni-101` / `deepseek-paid-smart` |
| Slot | 2026-11-10 23:59:54.486572 +08:00 (cron `59 23 * * *`) |
| Ended | **2026-11-11 00:12:29.646987 +08:00** — **12 m 35 s** |
| Status | `failed` · `delivery_outcome: suppressed` |
| Error | `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` |
| Output | `~/.hermes/cron/output/cbfe434a3b5e/2026-11-11_00-12-28.md` |
The three log lines from the run's own session `local-1000c1a3…` show the shape of the failure:
ERROR agent.chat_completion_helpers: Streaming failed before delivery: [429]: service failure: tenant tn-5ud77abtl1 quota exceeded
ERROR agent.chat_completion_helpers: Streaming failed before delivery: Error code: 429 -
ERROR agent.conversation_loop: API call failed after 3 retries. HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters
ERROR cron.scheduler: Job 'Wiki Audit — Nightly' failed: RuntimeError: HTTP 429: Service temporarily unavailable
- **The pool skipped every target.** Six candidates were evaluated and **`attempted 0`** — the run never
reached a provider. The only live target's credit balance read **100 %** consumed on connection
`conn:3021bab6`, which is why the router refused to dispatch rather than serving a degraded model.
- **`reset after 6h 24m 31s`** puts the pool's recovery at **≈ 2026-11-11 06:36:57 AWST**. The 07:0x
cron cohort ran clean the next morning, so the window closed on its own.
- The **same tenant** (`tn-5ud77abtl1`) appears in the `quota exceeded` line, so this is an
account-level quota fault on the router, not a single-model lockout.
## Incident row — raised, never alerted, still open
`cbfe43_ed01845310d8` — `job_id cbfe434a3b5e`, `failure_type: rate_limit`, state **`detected`**,
`alerted_at: NULL`, `closed_at: NULL`, opened **2026-11-11T00:12:28.578435+08:00**.
- It is the **only open row** in `cron_incidents` (135 rows total) — the previous open item,
`5fc292_773822455856`, closed `resolved` at **2026-11-11T08:03:47.644226**.
- So the estate's single outstanding incident is **the audit's own failure**, and nothing was told
about it.
## Impact
- **The 2026-11-10 window was never reviewed.** The nightly audit is the only producer of the per-day
ledger/wiki record; its failure means 11-10's 157 ledger rows were never reconciled on the day.
- **Fleet non-clean `last_status` 1 → 2.** `cbfe434a3b5e` now reads `last_status: error`,
`failure_streak: 1`, joining `066bb5191280` (*Uber Eats Monthly Spend Report*, `delivery_failed`,
streak 0).
- **The review published on 2026-11-11 23:59 carries both windows** — it is the recovery, not a
duplicate.
## Same signature, earlier appearances
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — the *identical* pre-dispatch-filter
string killed the Daily Marketing Job Radar on 2026-10-21 (`attempted 0`, no 503 anywhere in the day).
- 2026-11-05 — `deepseek-paid-smart` against the **CT600** endpoint returned `HTTP 503
ALL_TARGETS_SKIPPED` (`poolSize 5`, `attempted 0`) while `omniroute-1` answered 200 — recorded on
[[systems/scheduled-tasks]].
- This is the **third distinct symptom** of the same router behaviour: `attempted 0` with a healthy node
behind it is the thing to recognise, not the specific HTTP code.
## Fix Applied
**None — this is a detection-and-verification record.** No configuration changed this window. The run's
own recovery was the pool reset at ≈ 06:36:57 on 2026-11-11.
## Prevention
- **The observer needs an observer.** A failed nightly audit raises no delivery (suppressed), no alert
(`alerted_at: NULL`) and no page — every layer that would normally surface this is *inside the thing
that failed*. A one-line watchdog on `Wiki Audit — Nightly`'s `last_status != ok` (or on "no audit
commit in >24 h") is the only check that can catch it.
- **Alert on the artefact.** Verifying that a dated `audit(YYYY-MM-DD)` commit exists in this repo would
have caught both this failure and the stocktake's zero-edit days — the same lesson recorded on
[[incidents/2026-11-09-stocktake-completes-with-zero-edits]].
## Related
- [[systems/scheduled-tasks]]
- [[systems/omniroute]]
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]