--- title: Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State type: incident status: resolved created: 2026-11-02 updated: 2026-11-02 verified_on: 2026-11-02 review_after: 2026-11-09 confidence: high tags: [cron, scheduler, worker, sigterm, orphan, cold-start, silent-failure] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-logs, live-jobs-json, live-probe] --- # Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State ## Headline On **2026-11-01** three separate `cron.scheduler` **external workers** were terminated with **`SIGTERM` (`-15`) after they had adopted a durable execution claim but before they wrote a terminal state**, so each run closed as *"whether side effects ran is unknown"*. The three casualties were `85edf6772bca` (Tech & AI Digest, 07:30:46), `96eb95ce12f2` (log-error watchdog, 09:02:18) and `0e753e9720fd` (Facebook scrape, 09:02:18). **This is a fourth variant of the orphan class** — after the 2026-10-20 unclean crash, the 2026-10-26 planned `SIGTERM` whose drain timed out in 1.35 s, and the **2026-10-31 force-drain at the full 1 800 s cap** ([[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]). The distinguishing feature here: **no gateway restart, reload or drain occurred anywhere near these signals.** Gateway PID `175269` had been up since **2026-10-31 19:24:40** and stayed up; the relay reconnect counter ran continuously through both incidents (attempt 147 at 07:24:58 → 166 at 09:01:34 → 336 at 2026-11-02 00:10), which it could not do across a process restart. The `-15` came from the cron worker supervision path itself. ## Evidence Identical `error` text on all three incident rows (`~/.hermes/cron/executions.db` → `cron_incidents`): ``` Restart-safe cron worker exited with status -15 after adopting this execution, without writing a durable terminal state; whether side effects ran is unknown.; worker stderr: :130: RuntimeWarning: 'cron.scheduler' found in sys.modules after import of package 'cron', but prior to execution of 'cron.scheduler'; this may result in unpredictable behaviour ``` | Incident id | Job | `first_seen_at` | `closed_at` | Execution outcome | |---|---|---|---|---| | `85edf6_94cd563ae0df` | `85edf6772bca` Tech & AI Digest | 2026-11-01 07:30:46 | 2026-11-01 11:01:12 | row **`completed`** / `delivered` (07:01:19 → 11:01:12, **4 h 00 m**) | | `96eb95_8091fba135e9` | `96eb95ce12f2` log-error watchdog | 2026-11-01 09:02:18 | 2026-11-01 09:16:18 | row **`unknown`** (09:01:03 → 09:02:13) | | `0e753e_2183c9040706` | `0e753e9720fd` Facebook scrape | 2026-11-01 09:02:18 | 2026-11-01 10:44:12 | row **`completed`** / `delivered` (09:01:03 → 10:44:12) | All three rows carry `failure_type: unknown`, `acked_at: None`. Only `85edf6` and `0e753e` were auto-resolved by their own later successful run; `96eb95`'s row stayed `unknown` — **the only permanently non-clean execution of the whole day**. ## The 09:00 pair — the cold-start window, in full | Time (AWST) | Event | |---|---| | 09:00:49.708 | `cron.scheduler: Cron external worker for job '96eb95ce12f2' did not acknowledge within 30s; leaving the durable execution claim untouched` | | 09:00:49 → 09:01:02 | The freshly spawned worker boots a full runtime — **`Plugin discovery complete: 83 found, 69 enabled`**, MCP capacity checks | | 09:01:13.552 | `Running job 'Facebook Birthday Monthly Scrape' (ID: 0e753e9720fd)` — the same worker then picks up a second job | | 09:02:13 | `96eb95ce12f2` execution row closes **`unknown`** | | 09:02:18.189 / .211 | Both incident rows written (`alerted` 09:02:29 / 09:02:32) | | 10:44:12 | `0e753e9720fd` row closes `completed` + `delivered` → incident resolved | | 09:15:45 → 09:16:19 | Next `96eb95ce12f2` slot runs `completed` → incident resolved | **Mechanism, as far as the retained logs show it:** the spawn path treats a **30 s acknowledgement window** as the health signal for an external worker. A cold worker must import the scheduler, discover **83 plugins / 69 enabled** and register MCP tool schemas before it can acknowledge — a cost that plainly exceeds 30 s on this host. The supervisor therefore logs the miss and **leaves the durable claim untouched** rather than reclaiming it; the worker then starts the job anyway and is `SIGTERM`ed ~60–90 s later, orphan-claiming the run. **What is NOT established:** *who* sent the `SIGTERM` in these three cases. No `SIGUSR1`, no `Restart requested`, no `drain`, no `UNCLEANLY`, and no `Previous gateway exited` marker appears in `gateway.log`, `agent.log` or `agent.log.1` in the 07:2x–09:1x window, and the relay/polling counters prove no process restart. Do not attribute this to a gateway reload — the signal source is unidentified. Treat this as an open mechanism question, not a solved one. ## Blast radius and recovery - **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`** against 161 / 4 on 10-31, 138 / 0 on 10-30 and 177 / 0 on 10-29. The two `unknown` rows are precisely `96eb95ce12f2` @09:01:03 and `79f9bb41ecf5` @18:00:39 (a handoff failure — see [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]). - **All three executions still reached the user** where they were meant to: the digest closed `delivered` at 11:01:12 and the Facebook scrape `delivered` at 10:44:12 (delivery rows `2026-11-01 11:01:12` and `2026-11-01 10:44:12`). - **No recurrence after 09:02:18** for the remaining ~15 h of 11-01 — every later `96eb95ce12f2` and `0c771463c3cd` slot (23 and 21 runs respectively) closed `completed`. - The `-15` class is **not** `[SILENT]`-shaped: each instance raised an incident row and each alerted (`85edf6` 07:31:14, `96eb95` 09:02:32, `0e753e` 09:02:29). Only the `unknown` execution status is invisible to the dashboard's status field. ## Why it matters A `-15`-after-adoption means the scheduler cannot say whether the job's side effects ran. For a job with external writes (Gitea pushes, API POSTs, email sends) that is a **possible double-execution** on the next slot. The current design resolved this properly for two of three, but the class keeps recurring in a new form roughly every 10 days — 10-20, 10-26, 10-31, 11-01. **The recurring element across all four variants is the same: a live cron worker is signalled mid-adoption and the durable claim is left open.** ## Open questions 1. Which supervisor sends the `-15` after a 30 s ack miss — and is the kill a timeout guard or an unrelated cleanup path? 2. Does the ack window scale with plugin count? 83 plugins discovered in a cold worker is the visible cost driver; if the window is fixed at 30 s, cold-start jobs on this host are structurally exposed. 3. Should a job with `failure_type: unknown` external side effects be blocked from its next slot until `~/.hermes/cron/admissions/*.json` is reconciled? Two admission files from 11-01 are still unreconciled (`141448f9…`, `69b5588f…`). ## Related - [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]] - [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] - [[systems/scheduled-tasks]] - [[systems/hermes-agent]]