65 lines
4.4 KiB
Markdown
65 lines
4.4 KiB
Markdown
---
|
||
title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop
|
||
type: incident
|
||
status: resolved
|
||
created: 2026-10-26
|
||
updated: 2026-11-18
|
||
verified_on: 2026-11-18
|
||
review_after: 2026-11-09
|
||
confidence: medium
|
||
tags: [cron, automation, hermes, incident]
|
||
sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json]
|
||
---
|
||
|
||
# Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop (2026-10-25)
|
||
|
||
**Windows:** 2026-10-25 08:31:14 → 10:26:37 AWST (runaway) and 17:19:36 → 19:51:59 AWST (five extra fires)
|
||
**Job:** `08eb86f02d37` — *Gmail Action Digest*, daily `30 8 * * *`, pinned `custom:omni-101`
|
||
|
||
## What happened
|
||
- The 10-25 **08:30 slot** claimed at 08:30:57, started **08:31:14** and did not finish until **10:26:37 —
|
||
a 1 h 55 m 23 s run**, against **91 s** (10-23) and **104 s** (10-24) in the two preceding days. It closed
|
||
`failed` on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed).`
|
||
- The same job then fired **five more times** on 10-25 — 17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53 —
|
||
for **six fires in the calendar day** against one configured daily slot.
|
||
- `19:18:41 → 19:28:04` closed `failed` on a signature this ledger had not seen before:
|
||
**`Cron admission missing: the owner never admitted this fire`** (`source: direct`, `pid 2473801`,
|
||
`scheduled_instant` empty). The 2026-10-20 orphan was `Unclean gateway exit` — a second, distinct variant.
|
||
- Every fire, including both failures, carries `delivery_outcome: delivered`, so **up to six Gmail Action
|
||
Digests reached Anthony on 2026-10-25**. The 19:30:31 fire ran 21 m 28 s.
|
||
- Restart context (`gateway.log`): `18:29:22` SIGTERM as a planned stop under systemd → `18:30:04` start;
|
||
`19:26:16` *Stopping gateway for restart* → `19:27:13` start. Three starts on 10-25 (00:32:20, 18:30:04,
|
||
19:27:13). The five extra fires bracket the evening pair. → [[systems/hermes-agent]]
|
||
|
||
## Root cause — hypothesised, not confirmed
|
||
The 2026-10-25 14:17 interactive investigation found that the 08:31 fire's worker had made **70 `terminal`
|
||
calls on unrelated Proxmox VM101 `/dev/sdb` backup work**, with run context growing **66 k → 148 k tokens**,
|
||
and that **context compression was unavailable all day** (`invalid_grant`). Working hypothesis: the scheduled
|
||
fire **adopted and continued an unrelated long-lived session** instead of starting its own. *Why* the
|
||
scheduler admitted an unrelated session is unexplained and is the first thing to check.
|
||
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||
|
||
## Impact
|
||
- Six delivered digests in one day (duplicate content) — the user-visible symptom.
|
||
- **Neither failure raised an `cron_incidents` row**; the two open rows are older (10-01, 10-19). A failed
|
||
execution therefore does not reliably produce an incident.
|
||
- The job now reads `last_status: delivery_queued` with an **empty** `last_error` (2026-10-26 00:1x) — a state
|
||
that did not resolve to `ok`.
|
||
- 2026-10-25 closed **172 `completed` / 2 `failed`**, both failures belonging to this job.
|
||
|
||
## Open questions / next actions
|
||
- Why did a scheduled fire adopt an unrelated session? Check session-reuse across fires.
|
||
- Record `Cron admission missing: the owner never admitted this fire` alongside the 2026-10-20 orphan
|
||
signature and give both a recovery path on gateway restart.
|
||
- Give scheduled runs a **wall-clock budget** — a 113× runtime creep was invisible anywhere except the
|
||
execution row.
|
||
- Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire.
|
||
|
||
## Resolution (verified 2026-11-18)
|
||
**Resolved — no recurrence in 23 days.** The `delivery_queued` state cleared (the open question above is answered: it did **not** persist past the 2026-10-26 08:30 slot). Job `08eb86f02d37` now reads `enabled`, `last_status: ok`, `last_error: null`, `failure_streak: 0`, last run 2026-11-17 08:33. In the retained `executions.db` window (2026-11-12 → 11-17; DB retention floor ~18 d means earlier slots have aged out) it ran **exactly once per day, `completed` every time, zero failed rows** — no refires since the 10-25 loop. Root cause of the 10-25 session-adoption remains hypothesised, not confirmed; the wall-clock-budget follow-up was never implemented, so the *exposure* (a runaway can recur silently) is unchanged even though the incident itself has not recurred.
|
||
|
||
## Related
|
||
- [[systems/scheduled-tasks]]
|
||
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||
- [[systems/hermes-agent]]
|