audit(2026-11-11): the audit's own 429 killed the 11-10 window; stocktake + radar recover with output
Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed). NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2 RESOLVED with output (dated Recovery sections added): - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B) - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md (opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x), incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
This commit is contained in:
@@ -66,7 +66,31 @@ sources: [gateway-log, live-config, live-probe]
|
||||
> currently erroring), and/or add credits for paid nous models. The original re-authenticate item
|
||||
> is **superseded** — auth works now.
|
||||
|
||||
> **Escalated 2026-11-11 (nightly audit).** The single-slot exposure described above became the
|
||||
> busiest error path on the host, and it now has a **downstream casualty**.
|
||||
>
|
||||
> - `anthropic/claude-haiku-4.5` on `nous` returned **404 `insufficient_credits_for_paid_model`**
|
||||
> **14 times** on 2026-11-11 (07:01:11 … last **19:19:38**) — up from 3 in the whole preceding
|
||||
> window. All from session `20261031_190813_95fbde` (**source `desktop`, started 2026-10-31
|
||||
> 19:08:13, still open**), whose pinned model is that same paid nous model.
|
||||
> - The **404 is immediately followed by a 401.** The session's fallback chain moves the turn on to
|
||||
> `opencode-zen` / `deepseek-v4.1-flash`, whose key expired on 2026-11-08 — **7 × HTTP 401
|
||||
> `api_key_expired`** on 11-11 (07:01:13, 07:19:30, 08:03:40, 13:03:28, 13:19:33, 18:03:39,
|
||||
> 19:19:39). So **both hops of this chain are dead**, and the pair repeats every time the session is
|
||||
> poked. → [[systems/opencode-go]]
|
||||
> - `auxiliary.pulse_memory` logged **19 lines** on 11-11 (06:0x, 08:0x, 13:0x, 18:0x), including
|
||||
> **11 ×** *"main provider `custom:omni-101` is unavailable and no fallback_chain /
|
||||
> fallback_providers is configured — refusing to guess another logged-in provider."* Live config
|
||||
> re-confirmed at audit time: `pulse_memory` is the **only** slot carrying **no `fallback_chain`**
|
||||
> (`{"provider": "nous", "model": "anthropic/claude-haiku-4.5"}`), while the other eight all chain
|
||||
> `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` on `custom:omni-101`.
|
||||
> - **Net effect:** pulse/voice memory retrieval failed on every attempt this window. `config.yaml`
|
||||
> mtime is unchanged since **2026-11-09 22:58:07** — nothing was fixed.
|
||||
>
|
||||
> **Open item, unchanged and now evidenced:** move `auxiliary.pulse_memory` onto `custom:omni-101`
|
||||
> like the other eight slots and give it a `fallback_chain`, and re-point or retire the stale desktop
|
||||
> session pinned to a paid nous model. No credentials were minted or modified.
|
||||
|
||||
## Related
|
||||
- [[systems/model-providers]]
|
||||
- [[systems/current-profile]]
|
||||
- [[systems/hermes-agent]]
|
||||
|
||||
Reference in New Issue
Block a user