Files
agent-estate-wiki/incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
T

5.8 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status `unknown`), No Alert incident active 2026-10-21 2026-10-27 2026-10-27 2026-11-03 high
gateway
cron
scheduler
lifecycle
restart
silent-failure
live-logs
live-gateway-ledger
live-sqlite-cron-executions
live-sqlite-sessions
live-probe

Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned, Nothing Alerted

Headline

On 2026-10-20 the Hermes gateway process died without running any exit path. The scheduler came back, found a cron execution whose owner had vanished mid-run, and closed it as unknown with the note:

Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.

No incident row was created, no notification was sent, and the affected job's daily digest has no delivery row. This is the first recorded instance of the orphaned-execution failure mode, and the first unclean gateway exit since 2026-09-21.

Timeline (2026-10-20, AWST)

Time Event
08:30 slot for 08eb86f02d37 — Gmail Action Digest, 30 8 * * *, model: smart-route
08:31:03 execution claimed (status: running)
08:50:28 last heartbeat of the previous gateway life (pid 2496376, started 2026-10-17 05:14:08 AWST, rss 394 MB, mem_available 12.2 GB, swap_used 3.5 GB)
09:09:01 ledger records that life as exited UNCLEANLY — “no exit path ran — SIGKILL / OOM / VM death, or a process kill issued by the agent or one of its descendants”; suspected_oom=False
09:09:16 log line Starting Hermes Gateway... (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second)
~09:10 the orphaned execution row closes as unknown

Impact

  • The 2026-10-20 Gmail Action Digest has no delivery row in deliveries.db and its execution carries delivery_outcome: NULL. The ledger therefore offers no evidence the digest was delivered — and no evidence it ran to completion either. Both questions are open by design of the unknown status.
  • The failure is invisible from the job table. 08eb86f02d37 shows last_status: error with a 503 from its 2026-10-19 slot — i.e. the row that the operator sees is the previous day's failure, not this one.
  • The incident ledger only ingests execution errors it can classify, and the orphan note exists only in the execution row: the string exited before a durable terminal state does not appear anywhere in ~/.hermes/logs/*.log*. Log-only forensics would miss this entirely.

How often does this happen?

  • 20 unclean exits are recorded in the retained gateway ledger, spanning 2026-08-01 → 2026-10-20 — but they cluster hard: 5 in 24 h on 2026-08-14/15 (the worst cluster, during the 5 GB era), 3 in the first week of August, 4 across 08-19 → 08-24. Since 2026-09-01 there have been only four (09-09 ×2, 09-21, 10-20).
  • The 2026-10-20 exit ended a ~27-day clean run. The preceding unclean life (pid 249364) started 2026-09-21 22:11 AWST and last heartbeated 2026-09-23 13:07 AWST.
  • suspected_oom=False on every entry, including this one — at 12.2 GB free and 394 MB RSS this was not memory pressure. The ledger's own candidate list (SIGKILL / VM death / a pkill issued by the agent or a descendant) points at an operator- or agent-initiated kill, not a crash.
  • Only one execution has ever been orphaned by it. Across the same ledger window the other 19 exits either happened between slots or were survived by their runs — this one landed 38 minutes into a live run.

What would make this recoverable

  1. Orphan the run loudly. An unknown execution that had a delivery target and produced no delivery row is a probable non-delivery — that should raise an incident row like any other job failure.
  2. Record the orphan marker in agent.log as well as the execution row, so a log-only audit can see it.
  3. Re-run on restart where the job is idempotent (a digest is), instead of leaving the slot empty.
  4. Reconcile last_status with the orphaned execution — the row currently shows the previous slot's error, which is actively misleading.

None of the above is applied; the fix belongs with whoever owns the scheduler's restart path.

Verification commands

# unclean gateway exits, newest first
grep -ah "lifecycle_ledger" ~/.hermes/logs/gateway.log* 
# orphaned executions
sqlite3 ~/.hermes/.hermes/cron/executions.db \
  "SELECT job_id,started_at,status,error FROM executions WHERE status='unknown' ORDER BY started_at DESC LIMIT 5"
# was the slot delivered?
sqlite3 ~/.hermes/.hermes/cron/deliveries.db \
  "SELECT * FROM deliveries WHERE created_at LIKE '2026-10-20%'"