audit(2026-11-11): the audit's own 429 killed the 11-10 window; stocktake + radar recover with output

Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed).

NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
 - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed
 - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6)
   reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded
 - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row
 - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2

RESOLVED with output (dated Recovery sections added):
 - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0
 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B)
 - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved

UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md
(opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x),
incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
This commit is contained in:
Hermes Wiki Audit
2026-11-12 00:16:39 +08:00
parent c108eb1012
commit 729bb597cf
11 changed files with 463 additions and 20 deletions
+35 -3
View File
@@ -3,8 +3,8 @@ title: OpenCode Go
type: system
status: active
created: 2026-07-29
updated: 2026-11-08
verified_on: 2026-11-08
updated: 2026-11-11
verified_on: 2026-11-11
confidence: high
tags: [opencode, provider, models, subscription, limits]
sources: [https://opencode.ai/docs/go/]
@@ -160,12 +160,44 @@ both returned **HTTP 401** on the Go endpoint:
use.
- **To restore:** mint a new key in the OpenCode console, then replace it in **both** stores — the Hermes
provider entry and `~/.local/share/opencode/auth.json` — and re-probe `…/zen/go/v1/models` for a 200.
(Keys live in the vault; this page records only their state — [[references/hermes-api-vault]].)
(Keys live in the vault; this page records only their state — the `hermes-api-vault` skill (vault-backed; no value is stored in this wiki).)
**Config format:** `opencode-go/<model-id>` (e.g. `opencode-go/kimi-k3`)
**Model list API:** `https://opencode.ai/zen/go/v1/models`
## 2026-11-11 — the expired credential is no longer latent: it is in the live fallback path
The 2026-11-08 record above closed with *"**No execution failed on an auth error** — nothing in the fleet
was actually consuming this provider in-window. The risk is latent: anything pinned to it fails on first
use."* **That latent risk has now fired, repeatedly.**
- `fallback_providers[1]` is `opencode-zen` / `deepseek-v4.1-flash` (`https://opencode.ai/zen/v1`,
`api_mode: chat_completions`) — i.e. the **second hop of every session fallback chain**.
- On **2026-11-11** it produced **7 × HTTP 401 `api_key_expired`**, each one *immediately* after the
hop above it (nous `anthropic/claude-haiku-4.5`) returned **404 `insufficient_credits_for_paid_model`**:
| # | nous 404 | opencode-zen 401 |
|---|---|---|
| 1 | 07:01:11 | 07:01:13 |
| 2 | 07:19:28 | 07:19:30 |
| 3 | 08:03:38 | 08:03:40 |
| 4 | 13:03:27 | 13:03:28 |
| 5 | 13:19:31 | 13:19:33 |
| 6 | 18:03:38 | 18:03:39 |
| 7 | 19:19:37 | 19:19:39 |
- **Two consecutive dead hops** on one chain: a paid model with no credit, then an expired key. Every
affected turn has to fall through to `custom:guacano` / `custom:litellm` / `kilocode` to succeed.
- ⚠ Nothing in the fleet is *pinned* to `opencode-zen` — no cron job names it — so this never shows up as
a job failure. It shows up only as `agent.chat_completion_helpers: Streaming failed before delivery:
Error code: 401` in `~/.hermes/logs/errors.log`. → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
**To restore:** mint a new key in the OpenCode console and replace it in both stores (the Hermes provider
entry and `~/.local/share/opencode/auth.json`), then re-probe `…/zen/go/v1/models` for a 200 — or
**remove `opencode-zen` from `fallback_providers`**, since a dead second hop costs a round-trip on every
retry. Keys live in the vault; this page records only their state — the `hermes-api-vault` skill (vault-backed; no value is stored in this wiki).
## Privacy
Zero-retention policy from providers — your data is not used for model training.