6.4 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status `unknown`), No Alert | incident | active | 2026-10-21 | 2026-11-18 | 2026-11-18 | 2026-11-03 | high |
|
|
Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned, Nothing Alerted
Headline
On 2026-10-20 the Hermes gateway process died without running any exit path. The scheduler came back,
found a cron execution whose owner had vanished mid-run, and closed it as unknown with the note:
Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.
No incident row was created, no notification was sent, and the affected job's daily digest has no delivery row. This is the first recorded instance of the orphaned-execution failure mode, and the first unclean gateway exit since 2026-09-21.
Timeline (2026-10-20, AWST)
| Time | Event |
|---|---|
| 08:30 | slot for 08eb86f02d37 — Gmail Action Digest, 30 8 * * *, model: smart-route |
| 08:31:03 | execution claimed (status: running) |
| 08:50:28 | last heartbeat of the previous gateway life (pid 2496376, started 2026-10-17 05:14:08 AWST, rss 394 MB, mem_available 12.2 GB, swap_used 3.5 GB) |
| 09:09:01 | ledger records that life as exited UNCLEANLY — “no exit path ran — SIGKILL / OOM / VM death, or a process kill issued by the agent or one of its descendants”; suspected_oom=False |
| 09:09:16 | log line Starting Hermes Gateway... (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second) |
| ~09:10 | the orphaned execution row closes as unknown |
Re-verified 2026-11-18 — no recurrence in the retained window. Zero
status: unknownexecutions exist in the retainedexecutions.db(retention floor ~18 d, so the 10-20 row itself has aged out — its facts below stand on the 10-27 read). Retainedgateway.log*files carry no UNCLEANLY line newer than 2026-08-31 (log rotation has since dropped the 10-20 entry the ledger once showed; per that rotation,0means not in the retained window, not never happened). Zero unclosedcron_incidentsrows. Status unchanged — the silent-orphan failure mode still has no alerting of its own.
Impact
- The 2026-10-20 Gmail Action Digest has no delivery row in
deliveries.dband its execution carriesdelivery_outcome: NULL. The ledger therefore offers no evidence the digest was delivered — and no evidence it ran to completion either. Both questions are open by design of theunknownstatus. - The failure is invisible from the job table.
08eb86f02d37showslast_status: errorwith a 503 from its 2026-10-19 slot — i.e. the row that the operator sees is the previous day's failure, not this one. - The incident ledger only ingests execution errors it can classify, and the orphan note exists only in the
execution row: the string
exited before a durable terminal statedoes not appear anywhere in~/.hermes/logs/*.log*. Log-only forensics would miss this entirely.
How often does this happen?
- 20 unclean exits are recorded in the retained gateway ledger, spanning 2026-08-01 → 2026-10-20 — but they cluster hard: 5 in 24 h on 2026-08-14/15 (the worst cluster, during the 5 GB era), 3 in the first week of August, 4 across 08-19 → 08-24. Since 2026-09-01 there have been only four (09-09 ×2, 09-21, 10-20).
- The 2026-10-20 exit ended a ~27-day clean run. The preceding unclean life (
pid 249364) started 2026-09-21 22:11 AWST and last heartbeated 2026-09-23 13:07 AWST. suspected_oom=Falseon every entry, including this one — at 12.2 GB free and 394 MB RSS this was not memory pressure. The ledger's own candidate list (SIGKILL / VM death / apkillissued by the agent or a descendant) points at an operator- or agent-initiated kill, not a crash.- Only one execution has ever been orphaned by it. Across the same ledger window the other 19 exits either happened between slots or were survived by their runs — this one landed 38 minutes into a live run.
What would make this recoverable
- Orphan the run loudly. An
unknownexecution that had a delivery target and produced no delivery row is a probable non-delivery — that should raise an incident row like any other job failure. - Record the orphan marker in
agent.logas well as the execution row, so a log-only audit can see it. - Re-run on restart where the job is idempotent (a digest is), instead of leaving the slot empty.
- Reconcile
last_statuswith the orphaned execution — the row currently shows the previous slot's error, which is actively misleading.
None of the above is applied; the fix belongs with whoever owns the scheduler's restart path.
Verification commands
# unclean gateway exits, newest first
grep -ah "lifecycle_ledger" ~/.hermes/logs/gateway.log*
# orphaned executions
sqlite3 ~/.hermes/.hermes/cron/executions.db \
"SELECT job_id,started_at,status,error FROM executions WHERE status='unknown' ORDER BY started_at DESC LIMIT 5"
# was the slot delivered?
sqlite3 ~/.hermes/.hermes/cron/deliveries.db \
"SELECT * FROM deliveries WHERE created_at LIKE '2026-10-20%'"
Related
- incidents/2026-10-26-planned-restart-orphaned-cron-run — recurrence on 2026-10-26, and the loud-orphan gap here is now partly closed. This one was a planned
SIGTERM(not an unclean exit) and the drain still timed out after 1.35 s, interruptingea79a2107e36(Mnemosyne Health Monitor) 2 h 34 m in. Unlike this page's case it closedfailedon a new string (lost its durable fire claim ownership), raised twocron_incidentsrows, and delivered the failure — but the work still did not happen. Any triage keyed on this page'sunknown/ "owner exited before a durable terminal state" wording will miss it. - systems/scheduled-tasks
- incidents/2026-10-18-degenerate-stream-canned-substitute — the other 2026-10-20 non-delivery (a canned substitute served to a user).
- incidents/2026-08-08-gitea-config-backup-push-dead — the other "the row is green, the work did not happen" case found in this audit.