3.6 KiB
3.6 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop | incident | open | 2026-10-26 | 2026-10-26 | 2026-10-26 | 2026-11-09 | medium |
|
|
Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop (2026-10-25)
Windows: 2026-10-25 08:31:14 → 10:26:37 AWST (runaway) and 17:19:36 → 19:51:59 AWST (five extra fires)
Job: 08eb86f02d37 — Gmail Action Digest, daily 30 8 * * *, pinned custom:omni-101
What happened
- The 10-25 08:30 slot claimed at 08:30:57, started 08:31:14 and did not finish until 10:26:37 —
a 1 h 55 m 23 s run, against 91 s (10-23) and 104 s (10-24) in the two preceding days. It closed
failedonRuntimeError: Operation interrupted: waiting for model response (347.2s elapsed). - The same job then fired five more times on 10-25 — 17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53 —
for six fires in the calendar day against one configured daily slot.
19:18:41 → 19:28:04closedfailedon a signature this ledger had not seen before:Cron admission missing: the owner never admitted this fire(source: direct,pid 2473801,scheduled_instantempty). The 2026-10-20 orphan wasUnclean gateway exit— a second, distinct variant.- Every fire, including both failures, carries
delivery_outcome: delivered, so up to six Gmail Action Digests reached Anthony on 2026-10-25. The 19:30:31 fire ran 21 m 28 s.
- Restart context (
gateway.log):18:29:22SIGTERM as a planned stop under systemd →18:30:04start;19:26:16Stopping gateway for restart →19:27:13start. Three starts on 10-25 (00:32:20, 18:30:04, 19:27:13). The five extra fires bracket the evening pair. → systems/hermes-agent
Root cause — hypothesised, not confirmed
The 2026-10-25 14:17 interactive investigation found that the 08:31 fire's worker had made 70 terminal
calls on unrelated Proxmox VM101 /dev/sdb backup work, with run context growing 66 k → 148 k tokens,
and that context compression was unavailable all day (invalid_grant). Working hypothesis: the scheduled
fire adopted and continued an unrelated long-lived session instead of starting its own. Why the
scheduler admitted an unrelated session is unexplained and is the first thing to check.
→ incidents/2026-10-22-nous-portal-refresh-token-invalid
Impact
- Six delivered digests in one day (duplicate content) — the user-visible symptom.
- Neither failure raised an
cron_incidentsrow; the two open rows are older (10-01, 10-19). A failed execution therefore does not reliably produce an incident. - The job now reads
last_status: delivery_queuedwith an emptylast_error(2026-10-26 00:1x) — a state that did not resolve took. - 2026-10-25 closed 172
completed/ 2failed, both failures belonging to this job.
Open questions / next actions
- Why did a scheduled fire adopt an unrelated session? Check session-reuse across fires.
- Record
Cron admission missing: the owner never admitted this firealongside the 2026-10-20 orphan signature and give both a recovery path on gateway restart. - Give scheduled runs a wall-clock budget — a 113× runtime creep was invisible anywhere except the execution row.
- Does
delivery_queuedclear on the 2026-10-26 08:30 slot? Check after that fire.