Files
agent-estate-wiki/incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
T
Tony0410 fda56e53fe audit 2026-11-02: Weekly Review lost to gateway handoff failure; three cron workers SIGTERM'd
Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown.

NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active)
NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved)
UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section)
UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h)
UPDATED index.md, log.md
2026-11-02 00:27:09 +08:00

7.1 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State incident resolved 2026-11-02 2026-11-02 2026-11-02 2026-11-09 high
cron
scheduler
worker
sigterm
orphan
cold-start
silent-failure
live-sqlite-cron-executions
live-sqlite-cron-incidents
live-logs
live-jobs-json
live-probe

Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State

Headline

On 2026-11-01 three separate cron.scheduler external workers were terminated with SIGTERM (-15) after they had adopted a durable execution claim but before they wrote a terminal state, so each run closed as "whether side effects ran is unknown". The three casualties were 85edf6772bca (Tech & AI Digest, 07:30:46), 96eb95ce12f2 (log-error watchdog, 09:02:18) and 0e753e9720fd (Facebook scrape, 09:02:18). This is a fourth variant of the orphan class — after the 2026-10-20 unclean crash, the 2026-10-26 planned SIGTERM whose drain timed out in 1.35 s, and the 2026-10-31 force-drain at the full 1 800 s cap (incidents/2026-10-31-gateway-reload-force-drain-orphaned-run). The distinguishing feature here: no gateway restart, reload or drain occurred anywhere near these signals. Gateway PID 175269 had been up since 2026-10-31 19:24:40 and stayed up; the relay reconnect counter ran continuously through both incidents (attempt 147 at 07:24:58 → 166 at 09:01:34 → 336 at 2026-11-02 00:10), which it could not do across a process restart. The -15 came from the cron worker supervision path itself.

Evidence

Identical error text on all three incident rows (~/.hermes/cron/executions.db → cron_incidents):

Restart-safe cron worker exited with status -15 after adopting this execution, without writing a
durable terminal state; whether side effects ran is unknown.;
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in sys.modules after import
of package 'cron', but prior to execution of 'cron.scheduler'; this may result in unpredictable behaviour
Incident id Job first_seen_at closed_at Execution outcome
85edf6_94cd563ae0df 85edf6772bca Tech & AI Digest 2026-11-01 07:30:46 2026-11-01 11:01:12 row completed / delivered (07:01:19 → 11:01:12, 4 h 00 m)
96eb95_8091fba135e9 96eb95ce12f2 log-error watchdog 2026-11-01 09:02:18 2026-11-01 09:16:18 row unknown (09:01:03 → 09:02:13)
0e753e_2183c9040706 0e753e9720fd Facebook scrape 2026-11-01 09:02:18 2026-11-01 10:44:12 row completed / delivered (09:01:03 → 10:44:12)

All three rows carry failure_type: unknown, acked_at: None. Only 85edf6 and 0e753e were auto-resolved by their own later successful run; 96eb95's row stayed unknown — the only permanently non-clean execution of the whole day.

The 09:00 pair — the cold-start window, in full

Time (AWST) Event
09:00:49.708 cron.scheduler: Cron external worker for job '96eb95ce12f2' did not acknowledge within 30s; leaving the durable execution claim untouched
09:00:49 → 09:01:02 The freshly spawned worker boots a full runtime — Plugin discovery complete: 83 found, 69 enabled, MCP capacity checks
09:01:13.552 Running job 'Facebook Birthday Monthly Scrape' (ID: 0e753e9720fd) — the same worker then picks up a second job
09:02:13 96eb95ce12f2 execution row closes unknown
09:02:18.189 / .211 Both incident rows written (alerted 09:02:29 / 09:02:32)
10:44:12 0e753e9720fd row closes completed + delivered → incident resolved
09:15:45 → 09:16:19 Next 96eb95ce12f2 slot runs completed → incident resolved

Mechanism, as far as the retained logs show it: the spawn path treats a 30 s acknowledgement window as the health signal for an external worker. A cold worker must import the scheduler, discover 83 plugins / 69 enabled and register MCP tool schemas before it can acknowledge — a cost that plainly exceeds 30 s on this host. The supervisor therefore logs the miss and leaves the durable claim untouched rather than reclaiming it; the worker then starts the job anyway and is SIGTERMed ~60–90 s later, orphan-claiming the run.

What is NOT established: who sent the SIGTERM in these three cases. No SIGUSR1, no Restart requested, no drain, no UNCLEANLY, and no Previous gateway exited marker appears in gateway.log, agent.log or agent.log.1 in the 07:2x–09:1x window, and the relay/polling counters prove no process restart. Do not attribute this to a gateway reload — the signal source is unidentified. Treat this as an open mechanism question, not a solved one.

Blast radius and recovery

  • 2026-11-01 settled 167 completed / 1 running / 2 unknown against 161 / 4 on 10-31, 138 / 0 on 10-30 and 177 / 0 on 10-29. The two unknown rows are precisely 96eb95ce12f2 @09:01:03 and 79f9bb41ecf5 @18:00:39 (a handoff failure — see incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure).
  • All three executions still reached the user where they were meant to: the digest closed delivered at 11:01:12 and the Facebook scrape delivered at 10:44:12 (delivery rows 2026-11-01 11:01:12 and 2026-11-01 10:44:12).
  • No recurrence after 09:02:18 for the remaining ~15 h of 11-01 — every later 96eb95ce12f2 and 0c771463c3cd slot (23 and 21 runs respectively) closed completed.
  • The -15 class is not [SILENT]-shaped: each instance raised an incident row and each alerted (85edf6 07:31:14, 96eb95 09:02:32, 0e753e 09:02:29). Only the unknown execution status is invisible to the dashboard's status field.

Why it matters

A -15-after-adoption means the scheduler cannot say whether the job's side effects ran. For a job with external writes (Gitea pushes, API POSTs, email sends) that is a possible double-execution on the next slot. The current design resolved this properly for two of three, but the class keeps recurring in a new form roughly every 10 days — 10-20, 10-26, 10-31, 11-01. The recurring element across all four variants is the same: a live cron worker is signalled mid-adoption and the durable claim is left open.

Open questions

  1. Which supervisor sends the -15 after a 30 s ack miss — and is the kill a timeout guard or an unrelated cleanup path?
  2. Does the ack window scale with plugin count? 83 plugins discovered in a cold worker is the visible cost driver; if the window is fixed at 30 s, cold-start jobs on this host are structurally exposed.
  3. Should a job with failure_type: unknown external side effects be blocked from its next slot until ~/.hermes/cron/admissions/*.json is reconciled? Two admission files from 11-01 are still unreconciled (141448f9…, 69b5588f…).