Files
agent-estate-wiki/incidents/2026-10-25-gmail-digest-runaway-and-refire.md

65 lines
4.4 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop
type: incident
status: resolved
created: 2026-10-26
updated: 2026-11-18
verified_on: 2026-11-18
review_after: 2026-11-09
confidence: medium
tags: [cron, automation, hermes, incident]
sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json]
---
# Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop (2026-10-25)
**Windows:** 2026-10-25 08:31:14 → 10:26:37 AWST (runaway) and 17:19:36 → 19:51:59 AWST (five extra fires)
**Job:** `08eb86f02d37` — *Gmail Action Digest*, daily `30 8 * * *`, pinned `custom:omni-101`
## What happened
- The 10-25 **08:30 slot** claimed at 08:30:57, started **08:31:14** and did not finish until **10:26:37 —
a 1 h 55 m 23 s run**, against **91 s** (10-23) and **104 s** (10-24) in the two preceding days. It closed
`failed` on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed).`
- The same job then fired **five more times** on 10-25 — 17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53 —
for **six fires in the calendar day** against one configured daily slot.
- `19:18:41 → 19:28:04` closed `failed` on a signature this ledger had not seen before:
**`Cron admission missing: the owner never admitted this fire`** (`source: direct`, `pid 2473801`,
`scheduled_instant` empty). The 2026-10-20 orphan was `Unclean gateway exit` — a second, distinct variant.
- Every fire, including both failures, carries `delivery_outcome: delivered`, so **up to six Gmail Action
Digests reached Anthony on 2026-10-25**. The 19:30:31 fire ran 21 m 28 s.
- Restart context (`gateway.log`): `18:29:22` SIGTERM as a planned stop under systemd → `18:30:04` start;
`19:26:16` *Stopping gateway for restart* → `19:27:13` start. Three starts on 10-25 (00:32:20, 18:30:04,
19:27:13). The five extra fires bracket the evening pair. → [[systems/hermes-agent]]
## Root cause — hypothesised, not confirmed
The 2026-10-25 14:17 interactive investigation found that the 08:31 fire's worker had made **70 `terminal`
calls on unrelated Proxmox VM101 `/dev/sdb` backup work**, with run context growing **66 k → 148 k tokens**,
and that **context compression was unavailable all day** (`invalid_grant`). Working hypothesis: the scheduled
fire **adopted and continued an unrelated long-lived session** instead of starting its own. *Why* the
scheduler admitted an unrelated session is unexplained and is the first thing to check.
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
## Impact
- Six delivered digests in one day (duplicate content) — the user-visible symptom.
- **Neither failure raised an `cron_incidents` row**; the two open rows are older (10-01, 10-19). A failed
execution therefore does not reliably produce an incident.
- The job now reads `last_status: delivery_queued` with an **empty** `last_error` (2026-10-26 00:1x) — a state
that did not resolve to `ok`.
- 2026-10-25 closed **172 `completed` / 2 `failed`**, both failures belonging to this job.
## Open questions / next actions
- Why did a scheduled fire adopt an unrelated session? Check session-reuse across fires.
- Record `Cron admission missing: the owner never admitted this fire` alongside the 2026-10-20 orphan
signature and give both a recovery path on gateway restart.
- Give scheduled runs a **wall-clock budget** — a 113× runtime creep was invisible anywhere except the
execution row.
- Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire.
## Resolution (verified 2026-11-18)
**Resolved — no recurrence in 23 days.** The `delivery_queued` state cleared (the open question above is answered: it did **not** persist past the 2026-10-26 08:30 slot). Job `08eb86f02d37` now reads `enabled`, `last_status: ok`, `last_error: null`, `failure_streak: 0`, last run 2026-11-17 08:33. In the retained `executions.db` window (2026-11-12 → 11-17; DB retention floor ~18 d means earlier slots have aged out) it ran **exactly once per day, `completed` every time, zero failed rows** — no refires since the 10-25 loop. Root cause of the 10-25 session-adoption remains hypothesised, not confirmed; the wall-clock-budget follow-up was never implemented, so the *exposure* (a runaway can recur silently) is unchanged even though the incident itself has not recurred.
## Related
- [[systems/scheduled-tasks]]
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
- [[systems/hermes-agent]]