Files
agent-estate-wiki/incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
T

94 lines
5.8 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status `unknown`), No Alert
type: incident
status: active
created: 2026-10-21
updated: 2026-10-27
verified_on: 2026-10-27
review_after: 2026-11-03
confidence: high
tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure]
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
---
# Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned, Nothing Alerted
## Headline
On **2026-10-20** the Hermes gateway process died **without running any exit path**. The scheduler came back,
found a cron execution whose owner had vanished mid-run, and closed it as **`unknown`** with the note:
> *Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects
> ran is unknown.*
**No incident row was created, no notification was sent, and the affected job's daily digest has no delivery
row.** This is the first *recorded* instance of the orphaned-execution failure mode, and the first unclean
gateway exit since 2026-09-21.
## Timeline (2026-10-20, AWST)
| Time | Event |
|---|---|
| 08:30 | slot for `08eb86f02d37` — **Gmail Action Digest**, `30 8 * * *`, `model: smart-route` |
| 08:31:03 | execution claimed (`status: running`) |
| 08:50:28 | **last heartbeat of the previous gateway life** (`pid 2496376`, started 2026-10-17 05:14:08 AWST, `rss 394 MB`, `mem_available 12.2 GB`, `swap_used 3.5 GB`) |
| 09:09:01 | ledger records that life as **exited UNCLEANLY** — *“no exit path ran — SIGKILL / OOM / VM death, or a process kill issued by the agent or one of its descendants”*; **`suspected_oom=False`** |
| 09:09:16 | log line `Starting Hermes Gateway...` (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second) |
| ~09:10 | the orphaned execution row closes as **`unknown`** |
## Impact
- **The 2026-10-20 Gmail Action Digest has no delivery row in `deliveries.db` and its execution carries
`delivery_outcome: NULL`.** The ledger therefore offers **no evidence the digest was delivered** — and no
evidence it ran to completion either. Both questions are open by design of the `unknown` status.
- **The failure is invisible from the job table.** `08eb86f02d37` shows `last_status: error` with a **503 from
its 2026-10-19 slot** — i.e. the row that the operator sees is the *previous day's* failure, not this one.
- The incident ledger only ingests *execution errors it can classify*, and the orphan note exists **only in the
execution row**: the string `exited before a durable terminal state` does not appear anywhere in
`~/.hermes/logs/*.log*`. Log-only forensics would miss this entirely.
## How often does this happen?
- **20 unclean exits** are recorded in the retained gateway ledger, spanning **2026-08-01 → 2026-10-20** — but
they cluster hard: **5 in 24 h on 2026-08-14/15** (the worst cluster, during the 5 GB era), 3 in the first
week of August, 4 across 08-19 → 08-24. Since 2026-09-01 there have been only **four** (09-09 ×2, 09-21,
10-20).
- **The 2026-10-20 exit ended a ~27-day clean run.** The preceding unclean life (`pid 249364`) started
2026-09-21 22:11 AWST and last heartbeated 2026-09-23 13:07 AWST.
- **`suspected_oom=False` on every entry**, including this one — at 12.2 GB free and 394 MB RSS this was not
memory pressure. The ledger's own candidate list (SIGKILL / VM death / a `pkill` issued by the agent or a
descendant) points at an operator- or agent-initiated kill, not a crash.
- **Only one execution has ever been orphaned by it.** Across the same ledger window the other 19 exits either
happened between slots or were survived by their runs — this one landed 38 minutes into a live run.
## What would make this recoverable
1. **Orphan the run loudly.** An `unknown` execution that had a delivery target and produced no delivery row is a
*probable non-delivery* — that should raise an incident row like any other job failure.
2. **Record the orphan marker in `agent.log`** as well as the execution row, so a log-only audit can see it.
3. **Re-run on restart** where the job is idempotent (a digest is), instead of leaving the slot empty.
4. **Reconcile `last_status` with the orphaned execution** — the row currently shows the *previous* slot's error,
which is actively misleading.
None of the above is applied; the fix belongs with whoever owns the scheduler's restart path.
## Verification commands
```
# unclean gateway exits, newest first
grep -ah "lifecycle_ledger" ~/.hermes/logs/gateway.log*
# orphaned executions
sqlite3 ~/.hermes/.hermes/cron/executions.db \
"SELECT job_id,started_at,status,error FROM executions WHERE status='unknown' ORDER BY started_at DESC LIMIT 5"
# was the slot delivered?
sqlite3 ~/.hermes/.hermes/cron/deliveries.db \
"SELECT * FROM deliveries WHERE created_at LIKE '2026-10-20%'"
```
## Related
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **recurrence on 2026-10-26, and the loud-orphan gap here is now partly closed.** This one was a **planned `SIGTERM`** (not an unclean exit) and the drain still **timed out after 1.35 s**, interrupting `ea79a2107e36` (Mnemosyne Health Monitor) 2 h 34 m in. Unlike this page's case it closed **`failed`** on a **new** string (`lost its durable fire claim ownership`), **raised two `cron_incidents` rows**, and **delivered** the failure — but the work still did not happen. Any triage keyed on this page's `unknown` / "owner exited before a durable terminal state" wording will miss it.
- [[systems/scheduled-tasks]]
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — the other 2026-10-20 non-delivery (a canned substitute served to a user).
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the other "the row is green, the work did not happen" case found in this audit.