Files
agent-estate-wiki/incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
Tony0410 fda56e53fe audit 2026-11-02: Weekly Review lost to gateway handoff failure; three cron workers SIGTERM'd
Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown.

NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active)
NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved)
UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section)
UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h)
UPDATED index.md, log.md
2026-11-02 00:27:09 +08:00

116 lines
7.1 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State
type: incident
status: resolved
created: 2026-11-02
updated: 2026-11-02
verified_on: 2026-11-02
review_after: 2026-11-09
confidence: high
tags: [cron, scheduler, worker, sigterm, orphan, cold-start, silent-failure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-logs, live-jobs-json, live-probe]
---
# Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State
## Headline
On **2026-11-01** three separate `cron.scheduler` **external workers** were terminated with
**`SIGTERM` (`-15`) after they had adopted a durable execution claim but before they wrote a terminal
state**, so each run closed as *"whether side effects ran is unknown"*. The three casualties were
`85edf6772bca` (Tech & AI Digest, 07:30:46), `96eb95ce12f2` (log-error watchdog, 09:02:18) and
`0e753e9720fd` (Facebook scrape, 09:02:18). **This is a fourth variant of the orphan class** — after the
2026-10-20 unclean crash, the 2026-10-26 planned `SIGTERM` whose drain timed out in 1.35 s, and the
**2026-10-31 force-drain at the full 1 800 s cap** ([[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]).
The distinguishing feature here: **no gateway restart, reload or drain occurred anywhere near these
signals.** Gateway PID `175269` had been up since **2026-10-31 19:24:40** and stayed up; the relay
reconnect counter ran continuously through both incidents (attempt 147 at 07:24:58 → 166 at 09:01:34 →
336 at 2026-11-02 00:10), which it could not do across a process restart. The `-15` came from the cron
worker supervision path itself.
## Evidence
Identical `error` text on all three incident rows (`~/.hermes/cron/executions.db` → `cron_incidents`):
```
Restart-safe cron worker exited with status -15 after adopting this execution, without writing a
durable terminal state; whether side effects ran is unknown.;
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in sys.modules after import
of package 'cron', but prior to execution of 'cron.scheduler'; this may result in unpredictable behaviour
```
| Incident id | Job | `first_seen_at` | `closed_at` | Execution outcome |
|---|---|---|---|---|
| `85edf6_94cd563ae0df` | `85edf6772bca` Tech & AI Digest | 2026-11-01 07:30:46 | 2026-11-01 11:01:12 | row **`completed`** / `delivered` (07:01:19 → 11:01:12, **4 h 00 m**) |
| `96eb95_8091fba135e9` | `96eb95ce12f2` log-error watchdog | 2026-11-01 09:02:18 | 2026-11-01 09:16:18 | row **`unknown`** (09:01:03 → 09:02:13) |
| `0e753e_2183c9040706` | `0e753e9720fd` Facebook scrape | 2026-11-01 09:02:18 | 2026-11-01 10:44:12 | row **`completed`** / `delivered` (09:01:03 → 10:44:12) |
All three rows carry `failure_type: unknown`, `acked_at: None`. Only `85edf6` and `0e753e` were
auto-resolved by their own later successful run; `96eb95`'s row stayed `unknown` — **the only
permanently non-clean execution of the whole day**.
## The 09:00 pair — the cold-start window, in full
| Time (AWST) | Event |
|---|---|
| 09:00:49.708 | `cron.scheduler: Cron external worker for job '96eb95ce12f2' did not acknowledge within 30s; leaving the durable execution claim untouched` |
| 09:00:49 → 09:01:02 | The freshly spawned worker boots a full runtime — **`Plugin discovery complete: 83 found, 69 enabled`**, MCP capacity checks |
| 09:01:13.552 | `Running job 'Facebook Birthday Monthly Scrape' (ID: 0e753e9720fd)` — the same worker then picks up a second job |
| 09:02:13 | `96eb95ce12f2` execution row closes **`unknown`** |
| 09:02:18.189 / .211 | Both incident rows written (`alerted` 09:02:29 / 09:02:32) |
| 10:44:12 | `0e753e9720fd` row closes `completed` + `delivered` → incident resolved |
| 09:15:45 → 09:16:19 | Next `96eb95ce12f2` slot runs `completed` → incident resolved |
**Mechanism, as far as the retained logs show it:** the spawn path treats a **30 s acknowledgement
window** as the health signal for an external worker. A cold worker must import the scheduler, discover
**83 plugins / 69 enabled** and register MCP tool schemas before it can acknowledge — a cost that
plainly exceeds 30 s on this host. The supervisor therefore logs the miss and **leaves the durable claim
untouched** rather than reclaiming it; the worker then starts the job anyway and is `SIGTERM`ed ~60–90 s
later, orphan-claiming the run.
**What is NOT established:** *who* sent the `SIGTERM` in these three cases. No `SIGUSR1`, no
`Restart requested`, no `drain`, no `UNCLEANLY`, and no `Previous gateway exited` marker appears in
`gateway.log`, `agent.log` or `agent.log.1` in the 07:2x–09:1x window, and the relay/polling counters
prove no process restart. Do not attribute this to a gateway reload — the signal source is unidentified.
Treat this as an open mechanism question, not a solved one.
## Blast radius and recovery
- **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`** against 161 / 4 on 10-31,
138 / 0 on 10-30 and 177 / 0 on 10-29. The two `unknown` rows are precisely
`96eb95ce12f2` @09:01:03 and `79f9bb41ecf5` @18:00:39 (a handoff failure — see
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]).
- **All three executions still reached the user** where they were meant to: the digest closed
`delivered` at 11:01:12 and the Facebook scrape `delivered` at 10:44:12 (delivery rows
`2026-11-01 11:01:12` and `2026-11-01 10:44:12`).
- **No recurrence after 09:02:18** for the remaining ~15 h of 11-01 — every later `96eb95ce12f2`
and `0c771463c3cd` slot (23 and 21 runs respectively) closed `completed`.
- The `-15` class is **not** `[SILENT]`-shaped: each instance raised an incident row and each alerted
(`85edf6` 07:31:14, `96eb95` 09:02:32, `0e753e` 09:02:29). Only the `unknown` execution status is
invisible to the dashboard's status field.
## Why it matters
A `-15`-after-adoption means the scheduler cannot say whether the job's side effects ran. For a job with
external writes (Gitea pushes, API POSTs, email sends) that is a **possible double-execution** on the
next slot. The current design resolved this properly for two of three, but the class keeps recurring in
a new form roughly every 10 days — 10-20, 10-26, 10-31, 11-01. **The recurring element across all four
variants is the same: a live cron worker is signalled mid-adoption and the durable claim is left open.**
## Open questions
1. Which supervisor sends the `-15` after a 30 s ack miss — and is the kill a timeout guard or an
unrelated cleanup path?
2. Does the ack window scale with plugin count? 83 plugins discovered in a cold worker is the visible
cost driver; if the window is fixed at 30 s, cold-start jobs on this host are structurally exposed.
3. Should a job with `failure_type: unknown` external side effects be blocked from its next slot until
`~/.hermes/cron/admissions/*.json` is reconciled? Two admission files from 11-01 are still
unreconciled (`141448f9…`, `69b5588f…`).
## Related
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]
- [[systems/scheduled-tasks]]
- [[systems/hermes-agent]]