audit(2026-10-26): runaway Gmail digest + dead Nous credential; model default reverted to OmniRoute; conduit 0.69.2 live; freshness floor still 2026-10-08
This commit is contained in:
@@ -0,0 +1,48 @@
|
||||
---
|
||||
title: Nous Portal Refresh Token Invalid — Context Summaries Unavailable
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-26
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
review_after: 2026-11-09
|
||||
confidence: high
|
||||
tags: [hermes, router, memory, incident, unresolved]
|
||||
sources: [gateway-log, live-config, live-probe]
|
||||
---
|
||||
|
||||
# Nous Portal Refresh Token Invalid — Context Summaries Unavailable (2026-10-22 →)
|
||||
|
||||
`Nous Portal runtime credentials unavailable: Invalid refresh token. Run `hermes model` to re-authenticate.`
|
||||
|
||||
## Occurrences (gateway logs, counted 2026-10-26)
|
||||
| Day | Lines | Notable |
|
||||
|---|---|---|
|
||||
| 2026-10-22 | 1 | 11:32:20 `tui_gateway.server: llm.oneshot failed` |
|
||||
| 2026-10-24 | 3 | — |
|
||||
| 2026-10-25 | **20** | last at 14:53:04 — `agent.context_compressor: Failed to generate context summary` |
|
||||
| 2026-10-26 | 0 (to 00:1x) | none since the 19:27:13 restart |
|
||||
|
||||
## Effect
|
||||
- **Context compression failed outright.** Sessions could not summarise, so working context grew unchecked —
|
||||
one cron fire reached **148 k tokens** on 10-25 and ran 1 h 55 m.
|
||||
→ [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]
|
||||
- Every auxiliary LLM path pointed at `nous` failed with it.
|
||||
|
||||
## What changed — and what did not
|
||||
- **2026-10-25, `config.yaml` written 19:13 AWST:** all **eight** `auxiliary:` slots moved off `nous` to
|
||||
**`no-think/freeapi/auto` on `custom:omni-101`**, each with the chain
|
||||
`no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, `free_only: true`
|
||||
(compression timeout 90 s, vision timeout 180 s). → [[systems/model-providers]]
|
||||
- **The credential is still invalid.** The fault is **masked, not fixed** — no `hermes model`
|
||||
re-authentication has been run, and `nous` is still the **first** entry in `fallback_providers`.
|
||||
- Zero `invalid_grant` lines appear after the 2026-10-25 19:27:13 gateway start, so the masking is holding.
|
||||
|
||||
## Open
|
||||
- Re-authenticate the Nous Portal credential, or remove `nous` from `fallback_providers`.
|
||||
- Until then, a failover event will attempt a dead credential silently.
|
||||
|
||||
## Related
|
||||
- [[systems/model-providers]]
|
||||
- [[systems/current-profile]]
|
||||
- [[systems/hermes-agent]]
|
||||
@@ -3,7 +3,7 @@ title: Conduit Call Requests Refused by the Agent
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-25
|
||||
updated: 2026-10-25
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-25
|
||||
review_after: 2026-11-08
|
||||
confidence: high
|
||||
@@ -86,3 +86,10 @@ into [[incidents/2026-10-22-conduit-memory-start-timeout]].
|
||||
- [[systems/conduit]]
|
||||
- [[incidents/2026-10-22-conduit-memory-start-timeout]]
|
||||
- [[concepts/tool-calling]]
|
||||
|
||||
## Status update — 2026-10-26 00:1x AWST
|
||||
- **No further refusals in-window** (2026-10-24 12:56 → 2026-10-26 00:1x). The signature has not recurred.
|
||||
- `conduit-calls.json` → `history` is still **empty (0 entries)**, so the call path has gone unused, not fixed. The file *does* carry `watches`, `outcomes`, `ends` and `told` state.
|
||||
- The plugin **is** working for cards: `conduit_show` ×1 on 2026-10-25 14:37, delivered live.
|
||||
- Plugin advanced to **`0.69.2`** and the dashboard gateway restarted at **18:42:33**, so upstream commits are live — see [[systems/conduit]].
|
||||
- **Status stays open**: the request path is untested, not confirmed good.
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
---
|
||||
title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-26
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
review_after: 2026-11-09
|
||||
confidence: medium
|
||||
tags: [cron, automation, hermes, incident, unresolved]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json]
|
||||
---
|
||||
|
||||
# Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop (2026-10-25)
|
||||
|
||||
**Windows:** 2026-10-25 08:31:14 → 10:26:37 AWST (runaway) and 17:19:36 → 19:51:59 AWST (five extra fires)
|
||||
**Job:** `08eb86f02d37` — *Gmail Action Digest*, daily `30 8 * * *`, pinned `custom:omni-101`
|
||||
|
||||
## What happened
|
||||
- The 10-25 **08:30 slot** claimed at 08:30:57, started **08:31:14** and did not finish until **10:26:37 —
|
||||
a 1 h 55 m 23 s run**, against **91 s** (10-23) and **104 s** (10-24) in the two preceding days. It closed
|
||||
`failed` on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed).`
|
||||
- The same job then fired **five more times** on 10-25 — 17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53 —
|
||||
for **six fires in the calendar day** against one configured daily slot.
|
||||
- `19:18:41 → 19:28:04` closed `failed` on a signature this ledger had not seen before:
|
||||
**`Cron admission missing: the owner never admitted this fire`** (`source: direct`, `pid 2473801`,
|
||||
`scheduled_instant` empty). The 2026-10-20 orphan was `Unclean gateway exit` — a second, distinct variant.
|
||||
- Every fire, including both failures, carries `delivery_outcome: delivered`, so **up to six Gmail Action
|
||||
Digests reached Anthony on 2026-10-25**. The 19:30:31 fire ran 21 m 28 s.
|
||||
- Restart context (`gateway.log`): `18:29:22` SIGTERM as a planned stop under systemd → `18:30:04` start;
|
||||
`19:26:16` *Stopping gateway for restart* → `19:27:13` start. Three starts on 10-25 (00:32:20, 18:30:04,
|
||||
19:27:13). The five extra fires bracket the evening pair. → [[systems/hermes-agent]]
|
||||
|
||||
## Root cause — hypothesised, not confirmed
|
||||
The 2026-10-25 14:17 interactive investigation found that the 08:31 fire's worker had made **70 `terminal`
|
||||
calls on unrelated Proxmox VM101 `/dev/sdb` backup work**, with run context growing **66 k → 148 k tokens**,
|
||||
and that **context compression was unavailable all day** (`invalid_grant`). Working hypothesis: the scheduled
|
||||
fire **adopted and continued an unrelated long-lived session** instead of starting its own. *Why* the
|
||||
scheduler admitted an unrelated session is unexplained and is the first thing to check.
|
||||
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
|
||||
## Impact
|
||||
- Six delivered digests in one day (duplicate content) — the user-visible symptom.
|
||||
- **Neither failure raised an `cron_incidents` row**; the two open rows are older (10-01, 10-19). A failed
|
||||
execution therefore does not reliably produce an incident.
|
||||
- The job now reads `last_status: delivery_queued` with an **empty** `last_error` (2026-10-26 00:1x) — a state
|
||||
that did not resolve to `ok`.
|
||||
- 2026-10-25 closed **172 `completed` / 2 `failed`**, both failures belonging to this job.
|
||||
|
||||
## Open questions / next actions
|
||||
- Why did a scheduled fire adopt an unrelated session? Check session-reuse across fires.
|
||||
- Record `Cron admission missing: the owner never admitted this fire` alongside the 2026-10-20 orphan
|
||||
signature and give both a recovery path on gateway restart.
|
||||
- Give scheduled runs a **wall-clock budget** — a 113× runtime creep was invisible anywhere except the
|
||||
execution row.
|
||||
- Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire.
|
||||
|
||||
## Related
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
- [[systems/hermes-agent]]
|
||||
Reference in New Issue
Block a user