🦉 wiki audit 2026-10-21: gateway unclean-exit orphaned run (new), Gitea config-backup push dead since 2026-08-08 (new), model default → nous, 503 back to monitoring, canned-substitute trigger corrected

This commit is contained in:
Tony0410
2026-10-21 00:11:44 +08:00
parent fbd4f297eb
commit b662254569
10 changed files with 386 additions and 25 deletions
@@ -0,0 +1,92 @@
---
title: Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status `unknown`), No Alert
type: incident
status: active
created: 2026-10-21
updated: 2026-10-21
verified_on: 2026-10-21
review_after: 2026-10-28
confidence: high
tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure]
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
---
# Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned, Nothing Alerted
## Headline
On **2026-10-20** the Hermes gateway process died **without running any exit path**. The scheduler came back,
found a cron execution whose owner had vanished mid-run, and closed it as **`unknown`** with the note:
> *Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects
> ran is unknown.*
**No incident row was created, no notification was sent, and the affected job's daily digest has no delivery
row.** This is the first *recorded* instance of the orphaned-execution failure mode, and the first unclean
gateway exit since 2026-09-21.
## Timeline (2026-10-20, AWST)
| Time | Event |
|---|---|
| 08:30 | slot for `08eb86f02d37` — **Gmail Action Digest**, `30 8 * * *`, `model: smart-route` |
| 08:31:03 | execution claimed (`status: running`) |
| 08:50:28 | **last heartbeat of the previous gateway life** (`pid 2496376`, started 2026-10-17 05:14:08 AWST, `rss 394 MB`, `mem_available 12.2 GB`, `swap_used 3.5 GB`) |
| 09:09:01 | ledger records that life as **exited UNCLEANLY** — *“no exit path ran — SIGKILL / OOM / VM death, or a process kill issued by the agent or one of its descendants”*; **`suspected_oom=False`** |
| 09:09:16 | log line `Starting Hermes Gateway...` (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second) |
| ~09:10 | the orphaned execution row closes as **`unknown`** |
## Impact
- **The 2026-10-20 Gmail Action Digest has no delivery row in `deliveries.db` and its execution carries
`delivery_outcome: NULL`.** The ledger therefore offers **no evidence the digest was delivered** — and no
evidence it ran to completion either. Both questions are open by design of the `unknown` status.
- **The failure is invisible from the job table.** `08eb86f02d37` shows `last_status: error` with a **503 from
its 2026-10-19 slot** — i.e. the row that the operator sees is the *previous day's* failure, not this one.
- The incident ledger only ingests *execution errors it can classify*, and the orphan note exists **only in the
execution row**: the string `exited before a durable terminal state` does not appear anywhere in
`~/.hermes/logs/*.log*`. Log-only forensics would miss this entirely.
## How often does this happen?
- **20 unclean exits** are recorded in the retained gateway ledger, spanning **2026-08-01 → 2026-10-20** — but
they cluster hard: **5 in 24 h on 2026-08-14/15** (the worst cluster, during the 5 GB era), 3 in the first
week of August, 4 across 08-19 → 08-24. Since 2026-09-01 there have been only **four** (09-09 ×2, 09-21,
10-20).
- **The 2026-10-20 exit ended a ~27-day clean run.** The preceding unclean life (`pid 249364`) started
2026-09-21 22:11 AWST and last heartbeated 2026-09-23 13:07 AWST.
- **`suspected_oom=False` on every entry**, including this one — at 12.2 GB free and 394 MB RSS this was not
memory pressure. The ledger's own candidate list (SIGKILL / VM death / a `pkill` issued by the agent or a
descendant) points at an operator- or agent-initiated kill, not a crash.
- **Only one execution has ever been orphaned by it.** Across the same ledger window the other 19 exits either
happened between slots or were survived by their runs — this one landed 38 minutes into a live run.
## What would make this recoverable
1. **Orphan the run loudly.** An `unknown` execution that had a delivery target and produced no delivery row is a
*probable non-delivery* — that should raise an incident row like any other job failure.
2. **Record the orphan marker in `agent.log`** as well as the execution row, so a log-only audit can see it.
3. **Re-run on restart** where the job is idempotent (a digest is), instead of leaving the slot empty.
4. **Reconcile `last_status` with the orphaned execution** — the row currently shows the *previous* slot's error,
which is actively misleading.
None of the above is applied; the fix belongs with whoever owns the scheduler's restart path.
## Verification commands
```
# unclean gateway exits, newest first
grep -ah "lifecycle_ledger" ~/.hermes/logs/gateway.log*
# orphaned executions
sqlite3 ~/.hermes/.hermes/cron/executions.db \
"SELECT job_id,started_at,status,error FROM executions WHERE status='unknown' ORDER BY started_at DESC LIMIT 5"
# was the slot delivered?
sqlite3 ~/.hermes/.hermes/cron/deliveries.db \
"SELECT * FROM deliveries WHERE created_at LIKE '2026-10-20%'"
```
## Related
- [[systems/scheduled-tasks]]
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — the other 2026-10-20 non-delivery (a canned substitute served to a user).
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the other "the row is green, the work did not happen" case found in this audit.