Files
agent-estate-wiki/incidents/2026-10-25-gmail-digest-runaway-and-refire.md
T

3.6 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop incident open 2026-10-26 2026-10-26 2026-10-26 2026-11-09 medium
cron
automation
hermes
incident
unresolved
live-sqlite-cron-executions
live-sqlite-sessions
gateway-log
jobs-json

Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop (2026-10-25)

Windows: 2026-10-25 08:31:14 → 10:26:37 AWST (runaway) and 17:19:36 → 19:51:59 AWST (five extra fires) Job: 08eb86f02d37 — Gmail Action Digest, daily 30 8 * * *, pinned custom:omni-101

What happened

  • The 10-25 08:30 slot claimed at 08:30:57, started 08:31:14 and did not finish until 10:26:37 — a 1 h 55 m 23 s run, against 91 s (10-23) and 104 s (10-24) in the two preceding days. It closed failed on RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed).
  • The same job then fired five more times on 10-25 — 17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53 — for six fires in the calendar day against one configured daily slot.
    • 19:18:41 → 19:28:04 closed failed on a signature this ledger had not seen before: Cron admission missing: the owner never admitted this fire (source: direct, pid 2473801, scheduled_instant empty). The 2026-10-20 orphan was Unclean gateway exit — a second, distinct variant.
    • Every fire, including both failures, carries delivery_outcome: delivered, so up to six Gmail Action Digests reached Anthony on 2026-10-25. The 19:30:31 fire ran 21 m 28 s.
  • Restart context (gateway.log): 18:29:22 SIGTERM as a planned stop under systemd → 18:30:04 start; 19:26:16 Stopping gateway for restart → 19:27:13 start. Three starts on 10-25 (00:32:20, 18:30:04, 19:27:13). The five extra fires bracket the evening pair. → systems/hermes-agent

Root cause — hypothesised, not confirmed

The 2026-10-25 14:17 interactive investigation found that the 08:31 fire's worker had made 70 terminal calls on unrelated Proxmox VM101 /dev/sdb backup work, with run context growing 66 k → 148 k tokens, and that context compression was unavailable all day (invalid_grant). Working hypothesis: the scheduled fire adopted and continued an unrelated long-lived session instead of starting its own. Why the scheduler admitted an unrelated session is unexplained and is the first thing to check. → incidents/2026-10-22-nous-portal-refresh-token-invalid

Impact

  • Six delivered digests in one day (duplicate content) — the user-visible symptom.
  • Neither failure raised an cron_incidents row; the two open rows are older (10-01, 10-19). A failed execution therefore does not reliably produce an incident.
  • The job now reads last_status: delivery_queued with an empty last_error (2026-10-26 00:1x) — a state that did not resolve to ok.
  • 2026-10-25 closed 172 completed / 2 failed, both failures belonging to this job.

Open questions / next actions

  • Why did a scheduled fire adopt an unrelated session? Check session-reuse across fires.
  • Record Cron admission missing: the owner never admitted this fire alongside the 2026-10-20 orphan signature and give both a recovery path on gateway restart.
  • Give scheduled runs a wall-clock budget — a 113× runtime creep was invisible anywhere except the execution row.
  • Does delivery_queued clear on the 2026-10-26 08:30 slot? Check after that fire.