--- title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop type: incident status: open created: 2026-10-26 updated: 2026-10-26 verified_on: 2026-10-26 review_after: 2026-11-09 confidence: medium tags: [cron, automation, hermes, incident, unresolved] sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json] --- # Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop (2026-10-25) **Windows:** 2026-10-25 08:31:14 → 10:26:37 AWST (runaway) and 17:19:36 → 19:51:59 AWST (five extra fires) **Job:** `08eb86f02d37` — *Gmail Action Digest*, daily `30 8 * * *`, pinned `custom:omni-101` ## What happened - The 10-25 **08:30 slot** claimed at 08:30:57, started **08:31:14** and did not finish until **10:26:37 — a 1 h 55 m 23 s run**, against **91 s** (10-23) and **104 s** (10-24) in the two preceding days. It closed `failed` on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed).` - The same job then fired **five more times** on 10-25 — 17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53 — for **six fires in the calendar day** against one configured daily slot. - `19:18:41 → 19:28:04` closed `failed` on a signature this ledger had not seen before: **`Cron admission missing: the owner never admitted this fire`** (`source: direct`, `pid 2473801`, `scheduled_instant` empty). The 2026-10-20 orphan was `Unclean gateway exit` — a second, distinct variant. - Every fire, including both failures, carries `delivery_outcome: delivered`, so **up to six Gmail Action Digests reached Anthony on 2026-10-25**. The 19:30:31 fire ran 21 m 28 s. - Restart context (`gateway.log`): `18:29:22` SIGTERM as a planned stop under systemd → `18:30:04` start; `19:26:16` *Stopping gateway for restart* → `19:27:13` start. Three starts on 10-25 (00:32:20, 18:30:04, 19:27:13). The five extra fires bracket the evening pair. → [[systems/hermes-agent]] ## Root cause — hypothesised, not confirmed The 2026-10-25 14:17 interactive investigation found that the 08:31 fire's worker had made **70 `terminal` calls on unrelated Proxmox VM101 `/dev/sdb` backup work**, with run context growing **66 k → 148 k tokens**, and that **context compression was unavailable all day** (`invalid_grant`). Working hypothesis: the scheduled fire **adopted and continued an unrelated long-lived session** instead of starting its own. *Why* the scheduler admitted an unrelated session is unexplained and is the first thing to check. → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] ## Impact - Six delivered digests in one day (duplicate content) — the user-visible symptom. - **Neither failure raised an `cron_incidents` row**; the two open rows are older (10-01, 10-19). A failed execution therefore does not reliably produce an incident. - The job now reads `last_status: delivery_queued` with an **empty** `last_error` (2026-10-26 00:1x) — a state that did not resolve to `ok`. - 2026-10-25 closed **172 `completed` / 2 `failed`**, both failures belonging to this job. ## Open questions / next actions - Why did a scheduled fire adopt an unrelated session? Check session-reuse across fires. - Record `Cron admission missing: the owner never admitted this fire` alongside the 2026-10-20 orphan signature and give both a recovery path on gateway restart. - Give scheduled runs a **wall-clock budget** — a 113× runtime creep was invisible anywhere except the execution row. - Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire. ## Related - [[systems/scheduled-tasks]] - [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] - [[systems/hermes-agent]]