Files
agent-estate-wiki/incidents/2026-10-22-nous-portal-refresh-token-invalid.md
T
Hermes Wiki Audit 729bb597cf audit(2026-11-11): the audit's own 429 killed the 11-10 window; stocktake + radar recover with output
Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed).

NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
 - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed
 - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6)
   reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded
 - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row
 - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2

RESOLVED with output (dated Recovery sections added):
 - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0
 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B)
 - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved

UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md
(opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x),
incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
2026-11-12 00:16:39 +08:00

97 lines
5.7 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Nous Portal Refresh Token Invalid — Context Summaries Unavailable
type: incident
status: open
created: 2026-10-26
updated: 2026-11-11
verified_on: 2026-11-11
review_after: 2026-12-11
confidence: high
tags: [hermes, router, memory, incident, unresolved]
sources: [gateway-log, live-config, live-probe]
---
# Nous Portal Refresh Token Invalid — Context Summaries Unavailable (2026-10-22 →)
`Nous Portal runtime credentials unavailable: Invalid refresh token. Run `hermes model` to re-authenticate.`
## Occurrences (gateway logs, counted 2026-10-26)
| Day | Lines | Notable |
|---|---|---|
| 2026-10-22 | 1 | 11:32:20 `tui_gateway.server: llm.oneshot failed` |
| 2026-10-24 | 3 | — |
| 2026-10-25 | **20** | last at 14:53:04 — `agent.context_compressor: Failed to generate context summary` |
| 2026-10-26 | 0 (to 00:1x) | none since the 19:27:13 restart |
## Effect
- **Context compression failed outright.** Sessions could not summarise, so working context grew unchecked —
one cron fire reached **148 k tokens** on 10-25 and ran 1 h 55 m.
→ [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]
- Every auxiliary LLM path pointed at `nous` failed with it.
## What changed — and what did not
- **2026-10-25, `config.yaml` written 19:13 AWST:** all **eight** `auxiliary:` slots moved off `nous` to
**`no-think/freeapi/auto` on `custom:omni-101`**, each with the chain
`no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, `free_only: true`
(compression timeout 90 s, vision timeout 180 s). → [[systems/model-providers]]
- **The credential is still invalid.** The fault is **masked, not fixed** — no `hermes model`
re-authentication has been run, and `nous` is still the **first** entry in `fallback_providers`.
- Zero `invalid_grant` lines appear after the 2026-10-25 19:27:13 gateway start, so the masking is holding.
## Open
- Re-authenticate the Nous Portal credential, or remove `nous` from `fallback_providers`.
- Until then, a failover event will attempt a dead credential silently.
> **Re-verified 2026-11-11 — the refresh-token failure is gone from the logs, and the exposure has
> changed shape rather than closed.**
>
> - **Zero `invalid_grant` / refresh-token lines** now appear in `~/.hermes/logs/errors.log` (which
> covers the window through 2026-11-10 19:19) — so the credential is no longer failing at *auth*.
> `~/.hermes/.env` carries no `NOUS_API_KEY`; `~/.hermes/auth.json` is present and live.
> - What appears instead is **404 `insufficient_credits_for_paid_model`** for
> `anthropic/claude-haiku-4.5` — 3 occurrences, most recent **2026-11-10 19:19:33**, from session
> `20261031_190813_95fbde`.
> - **A ninth auxiliary slot now points back at `nous`.** The 2026-10-25 remediation moved all
> *eight* slots off `nous` onto `custom:omni-101` / `no-think/freeapi/auto`, but
> `auxiliary.pulse_memory` (`provider: nous`, `model: anthropic/claude-haiku-4.5`) was added
> afterwards and **is** the slot producing those 404s. So the masking is intact for the original
> eight slots but a new, single-slot nous dependency exists and is **failing on credits**, not on
> auth.
> - `nous` is **still the first entry** in `fallback_providers`
> (`meituan/longcat-2.5-preview:free`) — that is a *free* model and has not been observed failing,
> so the failover risk described below is now "may burn an attempt", not "guaranteed dead".
>
> **Revised open item:** decide whether `auxiliary.pulse_memory` should be moved onto
> `custom:omni-101` like the other eight slots (it is the only active nous path left, and it is
> currently erroring), and/or add credits for paid nous models. The original re-authenticate item
> is **superseded** — auth works now.
> **Escalated 2026-11-11 (nightly audit).** The single-slot exposure described above became the
> busiest error path on the host, and it now has a **downstream casualty**.
>
> - `anthropic/claude-haiku-4.5` on `nous` returned **404 `insufficient_credits_for_paid_model`**
> **14 times** on 2026-11-11 (07:01:11 … last **19:19:38**) — up from 3 in the whole preceding
> window. All from session `20261031_190813_95fbde` (**source `desktop`, started 2026-10-31
> 19:08:13, still open**), whose pinned model is that same paid nous model.
> - The **404 is immediately followed by a 401.** The session's fallback chain moves the turn on to
> `opencode-zen` / `deepseek-v4.1-flash`, whose key expired on 2026-11-08 — **7 × HTTP 401
> `api_key_expired`** on 11-11 (07:01:13, 07:19:30, 08:03:40, 13:03:28, 13:19:33, 18:03:39,
> 19:19:39). So **both hops of this chain are dead**, and the pair repeats every time the session is
> poked. → [[systems/opencode-go]]
> - `auxiliary.pulse_memory` logged **19 lines** on 11-11 (06:0x, 08:0x, 13:0x, 18:0x), including
> **11 ×** *"main provider `custom:omni-101` is unavailable and no fallback_chain /
> fallback_providers is configured — refusing to guess another logged-in provider."* Live config
> re-confirmed at audit time: `pulse_memory` is the **only** slot carrying **no `fallback_chain`**
> (`{"provider": "nous", "model": "anthropic/claude-haiku-4.5"}`), while the other eight all chain
> `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` on `custom:omni-101`.
> - **Net effect:** pulse/voice memory retrieval failed on every attempt this window. `config.yaml`
> mtime is unchanged since **2026-11-09 22:58:07** — nothing was fixed.
>
> **Open item, unchanged and now evidenced:** move `auxiliary.pulse_memory` onto `custom:omni-101`
> like the other eight slots and give it a `fallback_chain`, and re-point or retire the stale desktop
> session pinned to a paid nous model. No credentials were minted or modified.
## Related
- [[systems/current-profile]]
- [[systems/hermes-agent]]