audit(2026-10-26): runaway Gmail digest + dead Nous credential; model default reverted to OmniRoute; conduit 0.69.2 live; freshness floor still 2026-10-08
This commit is contained in:
@@ -0,0 +1,61 @@
|
||||
---
|
||||
title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-26
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
review_after: 2026-11-09
|
||||
confidence: medium
|
||||
tags: [cron, automation, hermes, incident, unresolved]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json]
|
||||
---
|
||||
|
||||
# Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop (2026-10-25)
|
||||
|
||||
**Windows:** 2026-10-25 08:31:14 → 10:26:37 AWST (runaway) and 17:19:36 → 19:51:59 AWST (five extra fires)
|
||||
**Job:** `08eb86f02d37` — *Gmail Action Digest*, daily `30 8 * * *`, pinned `custom:omni-101`
|
||||
|
||||
## What happened
|
||||
- The 10-25 **08:30 slot** claimed at 08:30:57, started **08:31:14** and did not finish until **10:26:37 —
|
||||
a 1 h 55 m 23 s run**, against **91 s** (10-23) and **104 s** (10-24) in the two preceding days. It closed
|
||||
`failed` on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed).`
|
||||
- The same job then fired **five more times** on 10-25 — 17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53 —
|
||||
for **six fires in the calendar day** against one configured daily slot.
|
||||
- `19:18:41 → 19:28:04` closed `failed` on a signature this ledger had not seen before:
|
||||
**`Cron admission missing: the owner never admitted this fire`** (`source: direct`, `pid 2473801`,
|
||||
`scheduled_instant` empty). The 2026-10-20 orphan was `Unclean gateway exit` — a second, distinct variant.
|
||||
- Every fire, including both failures, carries `delivery_outcome: delivered`, so **up to six Gmail Action
|
||||
Digests reached Anthony on 2026-10-25**. The 19:30:31 fire ran 21 m 28 s.
|
||||
- Restart context (`gateway.log`): `18:29:22` SIGTERM as a planned stop under systemd → `18:30:04` start;
|
||||
`19:26:16` *Stopping gateway for restart* → `19:27:13` start. Three starts on 10-25 (00:32:20, 18:30:04,
|
||||
19:27:13). The five extra fires bracket the evening pair. → [[systems/hermes-agent]]
|
||||
|
||||
## Root cause — hypothesised, not confirmed
|
||||
The 2026-10-25 14:17 interactive investigation found that the 08:31 fire's worker had made **70 `terminal`
|
||||
calls on unrelated Proxmox VM101 `/dev/sdb` backup work**, with run context growing **66 k → 148 k tokens**,
|
||||
and that **context compression was unavailable all day** (`invalid_grant`). Working hypothesis: the scheduled
|
||||
fire **adopted and continued an unrelated long-lived session** instead of starting its own. *Why* the
|
||||
scheduler admitted an unrelated session is unexplained and is the first thing to check.
|
||||
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
|
||||
## Impact
|
||||
- Six delivered digests in one day (duplicate content) — the user-visible symptom.
|
||||
- **Neither failure raised an `cron_incidents` row**; the two open rows are older (10-01, 10-19). A failed
|
||||
execution therefore does not reliably produce an incident.
|
||||
- The job now reads `last_status: delivery_queued` with an **empty** `last_error` (2026-10-26 00:1x) — a state
|
||||
that did not resolve to `ok`.
|
||||
- 2026-10-25 closed **172 `completed` / 2 `failed`**, both failures belonging to this job.
|
||||
|
||||
## Open questions / next actions
|
||||
- Why did a scheduled fire adopt an unrelated session? Check session-reuse across fires.
|
||||
- Record `Cron admission missing: the owner never admitted this fire` alongside the 2026-10-20 orphan
|
||||
signature and give both a recovery path on gateway restart.
|
||||
- Give scheduled runs a **wall-clock budget** — a 113× runtime creep was invisible anywhere except the
|
||||
execution row.
|
||||
- Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire.
|
||||
|
||||
## Related
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
- [[systems/hermes-agent]]
|
||||
Reference in New Issue
Block a user