audit 2026-11-02: Weekly Review lost to gateway handoff failure; three cron workers SIGTERM'd
Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown. NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active) NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved) UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section) UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h) UPDATED index.md, log.md
This commit is contained in:
@@ -0,0 +1,115 @@
|
||||
---
|
||||
title: Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-11-02
|
||||
updated: 2026-11-02
|
||||
verified_on: 2026-11-02
|
||||
review_after: 2026-11-09
|
||||
confidence: high
|
||||
tags: [cron, scheduler, worker, sigterm, orphan, cold-start, silent-failure]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-logs, live-jobs-json, live-probe]
|
||||
---
|
||||
|
||||
# Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State
|
||||
|
||||
## Headline
|
||||
|
||||
On **2026-11-01** three separate `cron.scheduler` **external workers** were terminated with
|
||||
**`SIGTERM` (`-15`) after they had adopted a durable execution claim but before they wrote a terminal
|
||||
state**, so each run closed as *"whether side effects ran is unknown"*. The three casualties were
|
||||
`85edf6772bca` (Tech & AI Digest, 07:30:46), `96eb95ce12f2` (log-error watchdog, 09:02:18) and
|
||||
`0e753e9720fd` (Facebook scrape, 09:02:18). **This is a fourth variant of the orphan class** — after the
|
||||
2026-10-20 unclean crash, the 2026-10-26 planned `SIGTERM` whose drain timed out in 1.35 s, and the
|
||||
**2026-10-31 force-drain at the full 1 800 s cap** ([[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]).
|
||||
The distinguishing feature here: **no gateway restart, reload or drain occurred anywhere near these
|
||||
signals.** Gateway PID `175269` had been up since **2026-10-31 19:24:40** and stayed up; the relay
|
||||
reconnect counter ran continuously through both incidents (attempt 147 at 07:24:58 → 166 at 09:01:34 →
|
||||
336 at 2026-11-02 00:10), which it could not do across a process restart. The `-15` came from the cron
|
||||
worker supervision path itself.
|
||||
|
||||
## Evidence
|
||||
|
||||
Identical `error` text on all three incident rows (`~/.hermes/cron/executions.db` → `cron_incidents`):
|
||||
|
||||
```
|
||||
Restart-safe cron worker exited with status -15 after adopting this execution, without writing a
|
||||
durable terminal state; whether side effects ran is unknown.;
|
||||
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in sys.modules after import
|
||||
of package 'cron', but prior to execution of 'cron.scheduler'; this may result in unpredictable behaviour
|
||||
```
|
||||
|
||||
| Incident id | Job | `first_seen_at` | `closed_at` | Execution outcome |
|
||||
|---|---|---|---|---|
|
||||
| `85edf6_94cd563ae0df` | `85edf6772bca` Tech & AI Digest | 2026-11-01 07:30:46 | 2026-11-01 11:01:12 | row **`completed`** / `delivered` (07:01:19 → 11:01:12, **4 h 00 m**) |
|
||||
| `96eb95_8091fba135e9` | `96eb95ce12f2` log-error watchdog | 2026-11-01 09:02:18 | 2026-11-01 09:16:18 | row **`unknown`** (09:01:03 → 09:02:13) |
|
||||
| `0e753e_2183c9040706` | `0e753e9720fd` Facebook scrape | 2026-11-01 09:02:18 | 2026-11-01 10:44:12 | row **`completed`** / `delivered` (09:01:03 → 10:44:12) |
|
||||
|
||||
All three rows carry `failure_type: unknown`, `acked_at: None`. Only `85edf6` and `0e753e` were
|
||||
auto-resolved by their own later successful run; `96eb95`'s row stayed `unknown` — **the only
|
||||
permanently non-clean execution of the whole day**.
|
||||
|
||||
## The 09:00 pair — the cold-start window, in full
|
||||
|
||||
| Time (AWST) | Event |
|
||||
|---|---|
|
||||
| 09:00:49.708 | `cron.scheduler: Cron external worker for job '96eb95ce12f2' did not acknowledge within 30s; leaving the durable execution claim untouched` |
|
||||
| 09:00:49 → 09:01:02 | The freshly spawned worker boots a full runtime — **`Plugin discovery complete: 83 found, 69 enabled`**, MCP capacity checks |
|
||||
| 09:01:13.552 | `Running job 'Facebook Birthday Monthly Scrape' (ID: 0e753e9720fd)` — the same worker then picks up a second job |
|
||||
| 09:02:13 | `96eb95ce12f2` execution row closes **`unknown`** |
|
||||
| 09:02:18.189 / .211 | Both incident rows written (`alerted` 09:02:29 / 09:02:32) |
|
||||
| 10:44:12 | `0e753e9720fd` row closes `completed` + `delivered` → incident resolved |
|
||||
| 09:15:45 → 09:16:19 | Next `96eb95ce12f2` slot runs `completed` → incident resolved |
|
||||
|
||||
**Mechanism, as far as the retained logs show it:** the spawn path treats a **30 s acknowledgement
|
||||
window** as the health signal for an external worker. A cold worker must import the scheduler, discover
|
||||
**83 plugins / 69 enabled** and register MCP tool schemas before it can acknowledge — a cost that
|
||||
plainly exceeds 30 s on this host. The supervisor therefore logs the miss and **leaves the durable claim
|
||||
untouched** rather than reclaiming it; the worker then starts the job anyway and is `SIGTERM`ed ~60–90 s
|
||||
later, orphan-claiming the run.
|
||||
|
||||
**What is NOT established:** *who* sent the `SIGTERM` in these three cases. No `SIGUSR1`, no
|
||||
`Restart requested`, no `drain`, no `UNCLEANLY`, and no `Previous gateway exited` marker appears in
|
||||
`gateway.log`, `agent.log` or `agent.log.1` in the 07:2x–09:1x window, and the relay/polling counters
|
||||
prove no process restart. Do not attribute this to a gateway reload — the signal source is unidentified.
|
||||
Treat this as an open mechanism question, not a solved one.
|
||||
|
||||
## Blast radius and recovery
|
||||
|
||||
- **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`** against 161 / 4 on 10-31,
|
||||
138 / 0 on 10-30 and 177 / 0 on 10-29. The two `unknown` rows are precisely
|
||||
`96eb95ce12f2` @09:01:03 and `79f9bb41ecf5` @18:00:39 (a handoff failure — see
|
||||
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]).
|
||||
- **All three executions still reached the user** where they were meant to: the digest closed
|
||||
`delivered` at 11:01:12 and the Facebook scrape `delivered` at 10:44:12 (delivery rows
|
||||
`2026-11-01 11:01:12` and `2026-11-01 10:44:12`).
|
||||
- **No recurrence after 09:02:18** for the remaining ~15 h of 11-01 — every later `96eb95ce12f2`
|
||||
and `0c771463c3cd` slot (23 and 21 runs respectively) closed `completed`.
|
||||
- The `-15` class is **not** `[SILENT]`-shaped: each instance raised an incident row and each alerted
|
||||
(`85edf6` 07:31:14, `96eb95` 09:02:32, `0e753e` 09:02:29). Only the `unknown` execution status is
|
||||
invisible to the dashboard's status field.
|
||||
|
||||
## Why it matters
|
||||
|
||||
A `-15`-after-adoption means the scheduler cannot say whether the job's side effects ran. For a job with
|
||||
external writes (Gitea pushes, API POSTs, email sends) that is a **possible double-execution** on the
|
||||
next slot. The current design resolved this properly for two of three, but the class keeps recurring in
|
||||
a new form roughly every 10 days — 10-20, 10-26, 10-31, 11-01. **The recurring element across all four
|
||||
variants is the same: a live cron worker is signalled mid-adoption and the durable claim is left open.**
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Which supervisor sends the `-15` after a 30 s ack miss — and is the kill a timeout guard or an
|
||||
unrelated cleanup path?
|
||||
2. Does the ack window scale with plugin count? 83 plugins discovered in a cold worker is the visible
|
||||
cost driver; if the window is fixed at 30 s, cold-start jobs on this host are structurally exposed.
|
||||
3. Should a job with `failure_type: unknown` external side effects be blocked from its next slot until
|
||||
`~/.hermes/cron/admissions/*.json` is reconciled? Two admission files from 11-01 are still
|
||||
unreconciled (`141448f9…`, `69b5588f…`).
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
|
||||
- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/hermes-agent]]
|
||||
Reference in New Issue
Block a user