Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown. NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active) NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved) UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section) UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h) UPDATED index.md, log.md
7.1 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State | incident | resolved | 2026-11-02 | 2026-11-02 | 2026-11-02 | 2026-11-09 | high |
|
|
Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State
Headline
On 2026-11-01 three separate cron.scheduler external workers were terminated with
SIGTERM (-15) after they had adopted a durable execution claim but before they wrote a terminal
state, so each run closed as "whether side effects ran is unknown". The three casualties were
85edf6772bca (Tech & AI Digest, 07:30:46), 96eb95ce12f2 (log-error watchdog, 09:02:18) and
0e753e9720fd (Facebook scrape, 09:02:18). This is a fourth variant of the orphan class — after the
2026-10-20 unclean crash, the 2026-10-26 planned SIGTERM whose drain timed out in 1.35 s, and the
2026-10-31 force-drain at the full 1 800 s cap (incidents/2026-10-31-gateway-reload-force-drain-orphaned-run).
The distinguishing feature here: no gateway restart, reload or drain occurred anywhere near these
signals. Gateway PID 175269 had been up since 2026-10-31 19:24:40 and stayed up; the relay
reconnect counter ran continuously through both incidents (attempt 147 at 07:24:58 → 166 at 09:01:34 →
336 at 2026-11-02 00:10), which it could not do across a process restart. The -15 came from the cron
worker supervision path itself.
Evidence
Identical error text on all three incident rows (~/.hermes/cron/executions.db → cron_incidents):
Restart-safe cron worker exited with status -15 after adopting this execution, without writing a
durable terminal state; whether side effects ran is unknown.;
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in sys.modules after import
of package 'cron', but prior to execution of 'cron.scheduler'; this may result in unpredictable behaviour
| Incident id | Job | first_seen_at |
closed_at |
Execution outcome |
|---|---|---|---|---|
85edf6_94cd563ae0df |
85edf6772bca Tech & AI Digest |
2026-11-01 07:30:46 | 2026-11-01 11:01:12 | row completed / delivered (07:01:19 → 11:01:12, 4 h 00 m) |
96eb95_8091fba135e9 |
96eb95ce12f2 log-error watchdog |
2026-11-01 09:02:18 | 2026-11-01 09:16:18 | row unknown (09:01:03 → 09:02:13) |
0e753e_2183c9040706 |
0e753e9720fd Facebook scrape |
2026-11-01 09:02:18 | 2026-11-01 10:44:12 | row completed / delivered (09:01:03 → 10:44:12) |
All three rows carry failure_type: unknown, acked_at: None. Only 85edf6 and 0e753e were
auto-resolved by their own later successful run; 96eb95's row stayed unknown — the only
permanently non-clean execution of the whole day.
The 09:00 pair — the cold-start window, in full
| Time (AWST) | Event |
|---|---|
| 09:00:49.708 | cron.scheduler: Cron external worker for job '96eb95ce12f2' did not acknowledge within 30s; leaving the durable execution claim untouched |
| 09:00:49 → 09:01:02 | The freshly spawned worker boots a full runtime — Plugin discovery complete: 83 found, 69 enabled, MCP capacity checks |
| 09:01:13.552 | Running job 'Facebook Birthday Monthly Scrape' (ID: 0e753e9720fd) — the same worker then picks up a second job |
| 09:02:13 | 96eb95ce12f2 execution row closes unknown |
| 09:02:18.189 / .211 | Both incident rows written (alerted 09:02:29 / 09:02:32) |
| 10:44:12 | 0e753e9720fd row closes completed + delivered → incident resolved |
| 09:15:45 → 09:16:19 | Next 96eb95ce12f2 slot runs completed → incident resolved |
Mechanism, as far as the retained logs show it: the spawn path treats a 30 s acknowledgement
window as the health signal for an external worker. A cold worker must import the scheduler, discover
83 plugins / 69 enabled and register MCP tool schemas before it can acknowledge — a cost that
plainly exceeds 30 s on this host. The supervisor therefore logs the miss and leaves the durable claim
untouched rather than reclaiming it; the worker then starts the job anyway and is SIGTERMed ~60–90 s
later, orphan-claiming the run.
What is NOT established: who sent the SIGTERM in these three cases. No SIGUSR1, no
Restart requested, no drain, no UNCLEANLY, and no Previous gateway exited marker appears in
gateway.log, agent.log or agent.log.1 in the 07:2x–09:1x window, and the relay/polling counters
prove no process restart. Do not attribute this to a gateway reload — the signal source is unidentified.
Treat this as an open mechanism question, not a solved one.
Blast radius and recovery
- 2026-11-01 settled 167
completed/ 1running/ 2unknownagainst 161 / 4 on 10-31, 138 / 0 on 10-30 and 177 / 0 on 10-29. The twounknownrows are precisely96eb95ce12f2@09:01:03 and79f9bb41ecf5@18:00:39 (a handoff failure — see incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure). - All three executions still reached the user where they were meant to: the digest closed
deliveredat 11:01:12 and the Facebook scrapedeliveredat 10:44:12 (delivery rows2026-11-01 11:01:12and2026-11-01 10:44:12). - No recurrence after 09:02:18 for the remaining ~15 h of 11-01 — every later
96eb95ce12f2and0c771463c3cdslot (23 and 21 runs respectively) closedcompleted. - The
-15class is not[SILENT]-shaped: each instance raised an incident row and each alerted (85edf607:31:14,96eb9509:02:32,0e753e09:02:29). Only theunknownexecution status is invisible to the dashboard's status field.
Why it matters
A -15-after-adoption means the scheduler cannot say whether the job's side effects ran. For a job with
external writes (Gitea pushes, API POSTs, email sends) that is a possible double-execution on the
next slot. The current design resolved this properly for two of three, but the class keeps recurring in
a new form roughly every 10 days — 10-20, 10-26, 10-31, 11-01. The recurring element across all four
variants is the same: a live cron worker is signalled mid-adoption and the durable claim is left open.
Open questions
- Which supervisor sends the
-15after a 30 s ack miss — and is the kill a timeout guard or an unrelated cleanup path? - Does the ack window scale with plugin count? 83 plugins discovered in a cold worker is the visible cost driver; if the window is fixed at 30 s, cold-start jobs on this host are structurally exposed.
- Should a job with
failure_type: unknownexternal side effects be blocked from its next slot until~/.hermes/cron/admissions/*.jsonis reconciled? Two admission files from 11-01 are still unreconciled (141448f9…,69b5588f…).