Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed). NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2 RESOLVED with output (dated Recovery sections added): - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B) - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md (opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x), incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
5.7 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Nous Portal Refresh Token Invalid — Context Summaries Unavailable | incident | open | 2026-10-26 | 2026-11-11 | 2026-11-11 | 2026-12-11 | high |
|
|
Nous Portal Refresh Token Invalid — Context Summaries Unavailable (2026-10-22 →)
Nous Portal runtime credentials unavailable: Invalid refresh token. Run hermes model to re-authenticate.
Occurrences (gateway logs, counted 2026-10-26)
| Day | Lines | Notable |
|---|---|---|
| 2026-10-22 | 1 | 11:32:20 tui_gateway.server: llm.oneshot failed |
| 2026-10-24 | 3 | — |
| 2026-10-25 | 20 | last at 14:53:04 — agent.context_compressor: Failed to generate context summary |
| 2026-10-26 | 0 (to 00:1x) | none since the 19:27:13 restart |
Effect
- Context compression failed outright. Sessions could not summarise, so working context grew unchecked — one cron fire reached 148 k tokens on 10-25 and ran 1 h 55 m. → incidents/2026-10-25-gmail-digest-runaway-and-refire
- Every auxiliary LLM path pointed at
nousfailed with it.
What changed — and what did not
- 2026-10-25,
config.yamlwritten 19:13 AWST: all eightauxiliary:slots moved offnoustono-think/freeapi/autooncustom:omni-101, each with the chainno-think/freeapi/deepseek-v4-flash→no-think/kg/kilo-auto/free,free_only: true(compression timeout 90 s, vision timeout 180 s). → systems/model-providers - The credential is still invalid. The fault is masked, not fixed — no
hermes modelre-authentication has been run, andnousis still the first entry infallback_providers. - Zero
invalid_grantlines appear after the 2026-10-25 19:27:13 gateway start, so the masking is holding.
Open
- Re-authenticate the Nous Portal credential, or remove
nousfromfallback_providers. - Until then, a failover event will attempt a dead credential silently.
Re-verified 2026-11-11 — the refresh-token failure is gone from the logs, and the exposure has changed shape rather than closed.
- Zero
invalid_grant/ refresh-token lines now appear in~/.hermes/logs/errors.log(which covers the window through 2026-11-10 19:19) — so the credential is no longer failing at auth.~/.hermes/.envcarries noNOUS_API_KEY;~/.hermes/auth.jsonis present and live.- What appears instead is 404
insufficient_credits_for_paid_modelforanthropic/claude-haiku-4.5— 3 occurrences, most recent 2026-11-10 19:19:33, from session20261031_190813_95fbde.- A ninth auxiliary slot now points back at
nous. The 2026-10-25 remediation moved all eight slots offnousontocustom:omni-101/no-think/freeapi/auto, butauxiliary.pulse_memory(provider: nous,model: anthropic/claude-haiku-4.5) was added afterwards and is the slot producing those 404s. So the masking is intact for the original eight slots but a new, single-slot nous dependency exists and is failing on credits, not on auth.nousis still the first entry infallback_providers(meituan/longcat-2.5-preview:free) — that is a free model and has not been observed failing, so the failover risk described below is now "may burn an attempt", not "guaranteed dead".Revised open item: decide whether
auxiliary.pulse_memoryshould be moved ontocustom:omni-101like the other eight slots (it is the only active nous path left, and it is currently erroring), and/or add credits for paid nous models. The original re-authenticate item is superseded — auth works now.
Escalated 2026-11-11 (nightly audit). The single-slot exposure described above became the busiest error path on the host, and it now has a downstream casualty.
anthropic/claude-haiku-4.5onnousreturned 404insufficient_credits_for_paid_model14 times on 2026-11-11 (07:01:11 … last 19:19:38) — up from 3 in the whole preceding window. All from session20261031_190813_95fbde(sourcedesktop, started 2026-10-31 19:08:13, still open), whose pinned model is that same paid nous model.- The 404 is immediately followed by a 401. The session's fallback chain moves the turn on to
opencode-zen/deepseek-v4.1-flash, whose key expired on 2026-11-08 — 7 × HTTP 401api_key_expiredon 11-11 (07:01:13, 07:19:30, 08:03:40, 13:03:28, 13:19:33, 18:03:39, 19:19:39). So both hops of this chain are dead, and the pair repeats every time the session is poked. → systems/opencode-goauxiliary.pulse_memorylogged 19 lines on 11-11 (06:0x, 08:0x, 13:0x, 18:0x), including 11 × "main providercustom:omni-101is unavailable and no fallback_chain / fallback_providers is configured — refusing to guess another logged-in provider." Live config re-confirmed at audit time:pulse_memoryis the only slot carrying nofallback_chain({"provider": "nous", "model": "anthropic/claude-haiku-4.5"}), while the other eight all chainno-think/freeapi/deepseek-v4-flash→no-think/kg/kilo-auto/freeoncustom:omni-101.- Net effect: pulse/voice memory retrieval failed on every attempt this window.
config.yamlmtime is unchanged since 2026-11-09 22:58:07 — nothing was fixed.Open item, unchanged and now evidenced: move
auxiliary.pulse_memoryontocustom:omni-101like the other eight slots and give it afallback_chain, and re-point or retire the stale desktop session pinned to a paid nous model. No credentials were minted or modified.