audit(2026-10-26): runaway Gmail digest + dead Nous credential; model default reverted to OmniRoute; conduit 0.69.2 live; freshness floor still 2026-10-08

This commit is contained in:
Tony0410
2026-10-26 00:19:37 +08:00
parent 2df1bce5ea
commit 123a74cb0b
16 changed files with 433 additions and 59 deletions
@@ -0,0 +1,61 @@
---
title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop
type: incident
status: open
created: 2026-10-26
updated: 2026-10-26
verified_on: 2026-10-26
review_after: 2026-11-09
confidence: medium
tags: [cron, automation, hermes, incident, unresolved]
sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json]
---
# Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop (2026-10-25)
**Windows:** 2026-10-25 08:31:14 → 10:26:37 AWST (runaway) and 17:19:36 → 19:51:59 AWST (five extra fires)
**Job:** `08eb86f02d37` — *Gmail Action Digest*, daily `30 8 * * *`, pinned `custom:omni-101`
## What happened
- The 10-25 **08:30 slot** claimed at 08:30:57, started **08:31:14** and did not finish until **10:26:37 —
a 1 h 55 m 23 s run**, against **91 s** (10-23) and **104 s** (10-24) in the two preceding days. It closed
`failed` on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed).`
- The same job then fired **five more times** on 10-25 — 17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53 —
for **six fires in the calendar day** against one configured daily slot.
- `19:18:41 → 19:28:04` closed `failed` on a signature this ledger had not seen before:
**`Cron admission missing: the owner never admitted this fire`** (`source: direct`, `pid 2473801`,
`scheduled_instant` empty). The 2026-10-20 orphan was `Unclean gateway exit` — a second, distinct variant.
- Every fire, including both failures, carries `delivery_outcome: delivered`, so **up to six Gmail Action
Digests reached Anthony on 2026-10-25**. The 19:30:31 fire ran 21 m 28 s.
- Restart context (`gateway.log`): `18:29:22` SIGTERM as a planned stop under systemd → `18:30:04` start;
`19:26:16` *Stopping gateway for restart* → `19:27:13` start. Three starts on 10-25 (00:32:20, 18:30:04,
19:27:13). The five extra fires bracket the evening pair. → [[systems/hermes-agent]]
## Root cause — hypothesised, not confirmed
The 2026-10-25 14:17 interactive investigation found that the 08:31 fire's worker had made **70 `terminal`
calls on unrelated Proxmox VM101 `/dev/sdb` backup work**, with run context growing **66 k → 148 k tokens**,
and that **context compression was unavailable all day** (`invalid_grant`). Working hypothesis: the scheduled
fire **adopted and continued an unrelated long-lived session** instead of starting its own. *Why* the
scheduler admitted an unrelated session is unexplained and is the first thing to check.
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
## Impact
- Six delivered digests in one day (duplicate content) — the user-visible symptom.
- **Neither failure raised an `cron_incidents` row**; the two open rows are older (10-01, 10-19). A failed
execution therefore does not reliably produce an incident.
- The job now reads `last_status: delivery_queued` with an **empty** `last_error` (2026-10-26 00:1x) — a state
that did not resolve to `ok`.
- 2026-10-25 closed **172 `completed` / 2 `failed`**, both failures belonging to this job.
## Open questions / next actions
- Why did a scheduled fire adopt an unrelated session? Check session-reuse across fires.
- Record `Cron admission missing: the owner never admitted this fire` alongside the 2026-10-20 orphan
signature and give both a recovery path on gateway restart.
- Give scheduled runs a **wall-clock budget** — a 113× runtime creep was invisible anywhere except the
execution row.
- Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire.
## Related
- [[systems/scheduled-tasks]]
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
- [[systems/hermes-agent]]