--- title: Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status `unknown`), No Alert type: incident status: active created: 2026-10-21 updated: 2026-11-18 verified_on: 2026-11-18 review_after: 2026-11-03 confidence: high tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure] sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] --- # Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned, Nothing Alerted ## Headline On **2026-10-20** the Hermes gateway process died **without running any exit path**. The scheduler came back, found a cron execution whose owner had vanished mid-run, and closed it as **`unknown`** with the note: > *Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects > ran is unknown.* **No incident row was created, no notification was sent, and the affected job's daily digest has no delivery row.** This is the first *recorded* instance of the orphaned-execution failure mode, and the first unclean gateway exit since 2026-09-21. ## Timeline (2026-10-20, AWST) | Time | Event | |---|---| | 08:30 | slot for `08eb86f02d37` — **Gmail Action Digest**, `30 8 * * *`, `model: smart-route` | | 08:31:03 | execution claimed (`status: running`) | | 08:50:28 | **last heartbeat of the previous gateway life** (`pid 2496376`, started 2026-10-17 05:14:08 AWST, `rss 394 MB`, `mem_available 12.2 GB`, `swap_used 3.5 GB`) | | 09:09:01 | ledger records that life as **exited UNCLEANLY** — *“no exit path ran — SIGKILL / OOM / VM death, or a process kill issued by the agent or one of its descendants”*; **`suspected_oom=False`** | | 09:09:16 | log line `Starting Hermes Gateway...` (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second) | | ~09:10 | the orphaned execution row closes as **`unknown`** | > **Re-verified 2026-11-18 — no recurrence in the retained window.** Zero `status: unknown` executions exist in the retained `executions.db` (retention floor ~18 d, so the 10-20 row itself has aged out — its facts below stand on the 10-27 read). Retained `gateway.log*` files carry no UNCLEANLY line newer than **2026-08-31** (log rotation has since dropped the 10-20 entry the ledger once showed; per that rotation, `0` means *not in the retained window*, not *never happened*). Zero unclosed `cron_incidents` rows. Status unchanged — the silent-orphan failure mode still has no alerting of its own. ## Impact - **The 2026-10-20 Gmail Action Digest has no delivery row in `deliveries.db` and its execution carries `delivery_outcome: NULL`.** The ledger therefore offers **no evidence the digest was delivered** — and no evidence it ran to completion either. Both questions are open by design of the `unknown` status. - **The failure is invisible from the job table.** `08eb86f02d37` shows `last_status: error` with a **503 from its 2026-10-19 slot** — i.e. the row that the operator sees is the *previous day's* failure, not this one. - The incident ledger only ingests *execution errors it can classify*, and the orphan note exists **only in the execution row**: the string `exited before a durable terminal state` does not appear anywhere in `~/.hermes/logs/*.log*`. Log-only forensics would miss this entirely. ## How often does this happen? - **20 unclean exits** are recorded in the retained gateway ledger, spanning **2026-08-01 → 2026-10-20** — but they cluster hard: **5 in 24 h on 2026-08-14/15** (the worst cluster, during the 5 GB era), 3 in the first week of August, 4 across 08-19 → 08-24. Since 2026-09-01 there have been only **four** (09-09 ×2, 09-21, 10-20). - **The 2026-10-20 exit ended a ~27-day clean run.** The preceding unclean life (`pid 249364`) started 2026-09-21 22:11 AWST and last heartbeated 2026-09-23 13:07 AWST. - **`suspected_oom=False` on every entry**, including this one — at 12.2 GB free and 394 MB RSS this was not memory pressure. The ledger's own candidate list (SIGKILL / VM death / a `pkill` issued by the agent or a descendant) points at an operator- or agent-initiated kill, not a crash. - **Only one execution has ever been orphaned by it.** Across the same ledger window the other 19 exits either happened between slots or were survived by their runs — this one landed 38 minutes into a live run. ## What would make this recoverable 1. **Orphan the run loudly.** An `unknown` execution that had a delivery target and produced no delivery row is a *probable non-delivery* — that should raise an incident row like any other job failure. 2. **Record the orphan marker in `agent.log`** as well as the execution row, so a log-only audit can see it. 3. **Re-run on restart** where the job is idempotent (a digest is), instead of leaving the slot empty. 4. **Reconcile `last_status` with the orphaned execution** — the row currently shows the *previous* slot's error, which is actively misleading. None of the above is applied; the fix belongs with whoever owns the scheduler's restart path. ## Verification commands ``` # unclean gateway exits, newest first grep -ah "lifecycle_ledger" ~/.hermes/logs/gateway.log* # orphaned executions sqlite3 ~/.hermes/.hermes/cron/executions.db \ "SELECT job_id,started_at,status,error FROM executions WHERE status='unknown' ORDER BY started_at DESC LIMIT 5" # was the slot delivered? sqlite3 ~/.hermes/.hermes/cron/deliveries.db \ "SELECT * FROM deliveries WHERE created_at LIKE '2026-10-20%'" ``` ## Related - [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **recurrence on 2026-10-26, and the loud-orphan gap here is now partly closed.** This one was a **planned `SIGTERM`** (not an unclean exit) and the drain still **timed out after 1.35 s**, interrupting `ea79a2107e36` (Mnemosyne Health Monitor) 2 h 34 m in. Unlike this page's case it closed **`failed`** on a **new** string (`lost its durable fire claim ownership`), **raised two `cron_incidents` rows**, and **delivered** the failure — but the work still did not happen. Any triage keyed on this page's `unknown` / "owner exited before a durable terminal state" wording will miss it. - [[systems/scheduled-tasks]] - [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — the other 2026-10-20 non-delivery (a canned substitute served to a user). - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the other "the row is green, the work did not happen" case found in this audit.