Files
agent-estate-wiki/incidents/2026-10-22-nous-portal-refresh-token-invalid.md
Hermes Wiki Audit 729bb597cf audit(2026-11-11): the audit's own 429 killed the 11-10 window; stocktake + radar recover with output
Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed).

NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
 - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed
 - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6)
   reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded
 - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row
 - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2

RESOLVED with output (dated Recovery sections added):
 - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0
 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B)
 - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved

UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md
(opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x),
incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
2026-11-12 00:16:39 +08:00

5.7 KiB
Raw Permalink Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Nous Portal Refresh Token Invalid — Context Summaries Unavailable incident open 2026-10-26 2026-11-11 2026-11-11 2026-12-11 high
hermes
router
memory
incident
unresolved
gateway-log
live-config
live-probe

Nous Portal Refresh Token Invalid — Context Summaries Unavailable (2026-10-22 →)

Nous Portal runtime credentials unavailable: Invalid refresh token. Run hermes model to re-authenticate.

Occurrences (gateway logs, counted 2026-10-26)

Day Lines Notable
2026-10-22 1 11:32:20 tui_gateway.server: llm.oneshot failed
2026-10-24 3 —
2026-10-25 20 last at 14:53:04 — agent.context_compressor: Failed to generate context summary
2026-10-26 0 (to 00:1x) none since the 19:27:13 restart

Effect

  • Context compression failed outright. Sessions could not summarise, so working context grew unchecked — one cron fire reached 148 k tokens on 10-25 and ran 1 h 55 m. → incidents/2026-10-25-gmail-digest-runaway-and-refire
  • Every auxiliary LLM path pointed at nous failed with it.

What changed — and what did not

  • 2026-10-25, config.yaml written 19:13 AWST: all eight auxiliary: slots moved off nous to no-think/freeapi/auto on custom:omni-101, each with the chain no-think/freeapi/deepseek-v4-flash → no-think/kg/kilo-auto/free, free_only: true (compression timeout 90 s, vision timeout 180 s). → systems/model-providers
  • The credential is still invalid. The fault is masked, not fixed — no hermes model re-authentication has been run, and nous is still the first entry in fallback_providers.
  • Zero invalid_grant lines appear after the 2026-10-25 19:27:13 gateway start, so the masking is holding.

Open

  • Re-authenticate the Nous Portal credential, or remove nous from fallback_providers.
  • Until then, a failover event will attempt a dead credential silently.

Re-verified 2026-11-11 — the refresh-token failure is gone from the logs, and the exposure has changed shape rather than closed.

  • Zero invalid_grant / refresh-token lines now appear in ~/.hermes/logs/errors.log (which covers the window through 2026-11-10 19:19) — so the credential is no longer failing at auth. ~/.hermes/.env carries no NOUS_API_KEY; ~/.hermes/auth.json is present and live.
  • What appears instead is 404 insufficient_credits_for_paid_model for anthropic/claude-haiku-4.5 — 3 occurrences, most recent 2026-11-10 19:19:33, from session 20261031_190813_95fbde.
  • A ninth auxiliary slot now points back at nous. The 2026-10-25 remediation moved all eight slots off nous onto custom:omni-101 / no-think/freeapi/auto, but auxiliary.pulse_memory (provider: nous, model: anthropic/claude-haiku-4.5) was added afterwards and is the slot producing those 404s. So the masking is intact for the original eight slots but a new, single-slot nous dependency exists and is failing on credits, not on auth.
  • nous is still the first entry in fallback_providers (meituan/longcat-2.5-preview:free) — that is a free model and has not been observed failing, so the failover risk described below is now "may burn an attempt", not "guaranteed dead".

Revised open item: decide whether auxiliary.pulse_memory should be moved onto custom:omni-101 like the other eight slots (it is the only active nous path left, and it is currently erroring), and/or add credits for paid nous models. The original re-authenticate item is superseded — auth works now.

Escalated 2026-11-11 (nightly audit). The single-slot exposure described above became the busiest error path on the host, and it now has a downstream casualty.

  • anthropic/claude-haiku-4.5 on nous returned 404 insufficient_credits_for_paid_model 14 times on 2026-11-11 (07:01:11 … last 19:19:38) — up from 3 in the whole preceding window. All from session 20261031_190813_95fbde (source desktop, started 2026-10-31 19:08:13, still open), whose pinned model is that same paid nous model.
  • The 404 is immediately followed by a 401. The session's fallback chain moves the turn on to opencode-zen / deepseek-v4.1-flash, whose key expired on 2026-11-08 — 7 × HTTP 401 api_key_expired on 11-11 (07:01:13, 07:19:30, 08:03:40, 13:03:28, 13:19:33, 18:03:39, 19:19:39). So both hops of this chain are dead, and the pair repeats every time the session is poked. → systems/opencode-go
  • auxiliary.pulse_memory logged 19 lines on 11-11 (06:0x, 08:0x, 13:0x, 18:0x), including 11 × "main provider custom:omni-101 is unavailable and no fallback_chain / fallback_providers is configured — refusing to guess another logged-in provider." Live config re-confirmed at audit time: pulse_memory is the only slot carrying no fallback_chain ({"provider": "nous", "model": "anthropic/claude-haiku-4.5"}), while the other eight all chain no-think/freeapi/deepseek-v4-flash → no-think/kg/kilo-auto/free on custom:omni-101.
  • Net effect: pulse/voice memory retrieval failed on every attempt this window. config.yaml mtime is unchanged since 2026-11-09 22:58:07 — nothing was fixed.

Open item, unchanged and now evidenced: move auxiliary.pulse_memory onto custom:omni-101 like the other eight slots and give it a fallback_chain, and re-point or retire the stale desktop session pinned to a paid nous model. No credentials were minted or modified.