Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed). NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2 RESOLVED with output (dated Recovery sections added): - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B) - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md (opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x), incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
97 lines
5.7 KiB
Markdown
97 lines
5.7 KiB
Markdown
---
|
||
title: Nous Portal Refresh Token Invalid — Context Summaries Unavailable
|
||
type: incident
|
||
status: open
|
||
created: 2026-10-26
|
||
updated: 2026-11-11
|
||
verified_on: 2026-11-11
|
||
review_after: 2026-12-11
|
||
confidence: high
|
||
tags: [hermes, router, memory, incident, unresolved]
|
||
sources: [gateway-log, live-config, live-probe]
|
||
---
|
||
|
||
# Nous Portal Refresh Token Invalid — Context Summaries Unavailable (2026-10-22 →)
|
||
|
||
`Nous Portal runtime credentials unavailable: Invalid refresh token. Run `hermes model` to re-authenticate.`
|
||
|
||
## Occurrences (gateway logs, counted 2026-10-26)
|
||
| Day | Lines | Notable |
|
||
|---|---|---|
|
||
| 2026-10-22 | 1 | 11:32:20 `tui_gateway.server: llm.oneshot failed` |
|
||
| 2026-10-24 | 3 | — |
|
||
| 2026-10-25 | **20** | last at 14:53:04 — `agent.context_compressor: Failed to generate context summary` |
|
||
| 2026-10-26 | 0 (to 00:1x) | none since the 19:27:13 restart |
|
||
|
||
## Effect
|
||
- **Context compression failed outright.** Sessions could not summarise, so working context grew unchecked —
|
||
one cron fire reached **148 k tokens** on 10-25 and ran 1 h 55 m.
|
||
→ [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]
|
||
- Every auxiliary LLM path pointed at `nous` failed with it.
|
||
|
||
## What changed — and what did not
|
||
- **2026-10-25, `config.yaml` written 19:13 AWST:** all **eight** `auxiliary:` slots moved off `nous` to
|
||
**`no-think/freeapi/auto` on `custom:omni-101`**, each with the chain
|
||
`no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, `free_only: true`
|
||
(compression timeout 90 s, vision timeout 180 s). → [[systems/model-providers]]
|
||
- **The credential is still invalid.** The fault is **masked, not fixed** — no `hermes model`
|
||
re-authentication has been run, and `nous` is still the **first** entry in `fallback_providers`.
|
||
- Zero `invalid_grant` lines appear after the 2026-10-25 19:27:13 gateway start, so the masking is holding.
|
||
|
||
## Open
|
||
- Re-authenticate the Nous Portal credential, or remove `nous` from `fallback_providers`.
|
||
- Until then, a failover event will attempt a dead credential silently.
|
||
|
||
> **Re-verified 2026-11-11 — the refresh-token failure is gone from the logs, and the exposure has
|
||
> changed shape rather than closed.**
|
||
>
|
||
> - **Zero `invalid_grant` / refresh-token lines** now appear in `~/.hermes/logs/errors.log` (which
|
||
> covers the window through 2026-11-10 19:19) — so the credential is no longer failing at *auth*.
|
||
> `~/.hermes/.env` carries no `NOUS_API_KEY`; `~/.hermes/auth.json` is present and live.
|
||
> - What appears instead is **404 `insufficient_credits_for_paid_model`** for
|
||
> `anthropic/claude-haiku-4.5` — 3 occurrences, most recent **2026-11-10 19:19:33**, from session
|
||
> `20261031_190813_95fbde`.
|
||
> - **A ninth auxiliary slot now points back at `nous`.** The 2026-10-25 remediation moved all
|
||
> *eight* slots off `nous` onto `custom:omni-101` / `no-think/freeapi/auto`, but
|
||
> `auxiliary.pulse_memory` (`provider: nous`, `model: anthropic/claude-haiku-4.5`) was added
|
||
> afterwards and **is** the slot producing those 404s. So the masking is intact for the original
|
||
> eight slots but a new, single-slot nous dependency exists and is **failing on credits**, not on
|
||
> auth.
|
||
> - `nous` is **still the first entry** in `fallback_providers`
|
||
> (`meituan/longcat-2.5-preview:free`) — that is a *free* model and has not been observed failing,
|
||
> so the failover risk described below is now "may burn an attempt", not "guaranteed dead".
|
||
>
|
||
> **Revised open item:** decide whether `auxiliary.pulse_memory` should be moved onto
|
||
> `custom:omni-101` like the other eight slots (it is the only active nous path left, and it is
|
||
> currently erroring), and/or add credits for paid nous models. The original re-authenticate item
|
||
> is **superseded** — auth works now.
|
||
|
||
> **Escalated 2026-11-11 (nightly audit).** The single-slot exposure described above became the
|
||
> busiest error path on the host, and it now has a **downstream casualty**.
|
||
>
|
||
> - `anthropic/claude-haiku-4.5` on `nous` returned **404 `insufficient_credits_for_paid_model`**
|
||
> **14 times** on 2026-11-11 (07:01:11 … last **19:19:38**) — up from 3 in the whole preceding
|
||
> window. All from session `20261031_190813_95fbde` (**source `desktop`, started 2026-10-31
|
||
> 19:08:13, still open**), whose pinned model is that same paid nous model.
|
||
> - The **404 is immediately followed by a 401.** The session's fallback chain moves the turn on to
|
||
> `opencode-zen` / `deepseek-v4.1-flash`, whose key expired on 2026-11-08 — **7 × HTTP 401
|
||
> `api_key_expired`** on 11-11 (07:01:13, 07:19:30, 08:03:40, 13:03:28, 13:19:33, 18:03:39,
|
||
> 19:19:39). So **both hops of this chain are dead**, and the pair repeats every time the session is
|
||
> poked. → [[systems/opencode-go]]
|
||
> - `auxiliary.pulse_memory` logged **19 lines** on 11-11 (06:0x, 08:0x, 13:0x, 18:0x), including
|
||
> **11 ×** *"main provider `custom:omni-101` is unavailable and no fallback_chain /
|
||
> fallback_providers is configured — refusing to guess another logged-in provider."* Live config
|
||
> re-confirmed at audit time: `pulse_memory` is the **only** slot carrying **no `fallback_chain`**
|
||
> (`{"provider": "nous", "model": "anthropic/claude-haiku-4.5"}`), while the other eight all chain
|
||
> `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` on `custom:omni-101`.
|
||
> - **Net effect:** pulse/voice memory retrieval failed on every attempt this window. `config.yaml`
|
||
> mtime is unchanged since **2026-11-09 22:58:07** — nothing was fixed.
|
||
>
|
||
> **Open item, unchanged and now evidenced:** move `auxiliary.pulse_memory` onto `custom:omni-101`
|
||
> like the other eight slots and give it a `fallback_chain`, and re-point or retire the stale desktop
|
||
> session pinned to a paid nous model. No credentials were minted or modified.
|
||
|
||
## Related
|
||
- [[systems/current-profile]]
|
||
- [[systems/hermes-agent]]
|