audit 2026-11-02: Weekly Review lost to gateway handoff failure; three cron workers SIGTERM'd

Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown.

NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active)
NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved)
UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section)
UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h)
UPDATED index.md, log.md
This commit is contained in:
Tony0410
2026-11-02 00:27:09 +08:00
parent 8bb3d196b1
commit fda56e53fe
6 changed files with 383 additions and 9 deletions
@@ -0,0 +1,115 @@
---
title: Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State
type: incident
status: resolved
created: 2026-11-02
updated: 2026-11-02
verified_on: 2026-11-02
review_after: 2026-11-09
confidence: high
tags: [cron, scheduler, worker, sigterm, orphan, cold-start, silent-failure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-logs, live-jobs-json, live-probe]
---
# Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State
## Headline
On **2026-11-01** three separate `cron.scheduler` **external workers** were terminated with
**`SIGTERM` (`-15`) after they had adopted a durable execution claim but before they wrote a terminal
state**, so each run closed as *"whether side effects ran is unknown"*. The three casualties were
`85edf6772bca` (Tech & AI Digest, 07:30:46), `96eb95ce12f2` (log-error watchdog, 09:02:18) and
`0e753e9720fd` (Facebook scrape, 09:02:18). **This is a fourth variant of the orphan class** — after the
2026-10-20 unclean crash, the 2026-10-26 planned `SIGTERM` whose drain timed out in 1.35 s, and the
**2026-10-31 force-drain at the full 1 800 s cap** ([[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]).
The distinguishing feature here: **no gateway restart, reload or drain occurred anywhere near these
signals.** Gateway PID `175269` had been up since **2026-10-31 19:24:40** and stayed up; the relay
reconnect counter ran continuously through both incidents (attempt 147 at 07:24:58 → 166 at 09:01:34 →
336 at 2026-11-02 00:10), which it could not do across a process restart. The `-15` came from the cron
worker supervision path itself.
## Evidence
Identical `error` text on all three incident rows (`~/.hermes/cron/executions.db` → `cron_incidents`):
```
Restart-safe cron worker exited with status -15 after adopting this execution, without writing a
durable terminal state; whether side effects ran is unknown.;
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in sys.modules after import
of package 'cron', but prior to execution of 'cron.scheduler'; this may result in unpredictable behaviour
```
| Incident id | Job | `first_seen_at` | `closed_at` | Execution outcome |
|---|---|---|---|---|
| `85edf6_94cd563ae0df` | `85edf6772bca` Tech & AI Digest | 2026-11-01 07:30:46 | 2026-11-01 11:01:12 | row **`completed`** / `delivered` (07:01:19 → 11:01:12, **4 h 00 m**) |
| `96eb95_8091fba135e9` | `96eb95ce12f2` log-error watchdog | 2026-11-01 09:02:18 | 2026-11-01 09:16:18 | row **`unknown`** (09:01:03 → 09:02:13) |
| `0e753e_2183c9040706` | `0e753e9720fd` Facebook scrape | 2026-11-01 09:02:18 | 2026-11-01 10:44:12 | row **`completed`** / `delivered` (09:01:03 → 10:44:12) |
All three rows carry `failure_type: unknown`, `acked_at: None`. Only `85edf6` and `0e753e` were
auto-resolved by their own later successful run; `96eb95`'s row stayed `unknown` — **the only
permanently non-clean execution of the whole day**.
## The 09:00 pair — the cold-start window, in full
| Time (AWST) | Event |
|---|---|
| 09:00:49.708 | `cron.scheduler: Cron external worker for job '96eb95ce12f2' did not acknowledge within 30s; leaving the durable execution claim untouched` |
| 09:00:49 → 09:01:02 | The freshly spawned worker boots a full runtime — **`Plugin discovery complete: 83 found, 69 enabled`**, MCP capacity checks |
| 09:01:13.552 | `Running job 'Facebook Birthday Monthly Scrape' (ID: 0e753e9720fd)` — the same worker then picks up a second job |
| 09:02:13 | `96eb95ce12f2` execution row closes **`unknown`** |
| 09:02:18.189 / .211 | Both incident rows written (`alerted` 09:02:29 / 09:02:32) |
| 10:44:12 | `0e753e9720fd` row closes `completed` + `delivered` → incident resolved |
| 09:15:45 → 09:16:19 | Next `96eb95ce12f2` slot runs `completed` → incident resolved |
**Mechanism, as far as the retained logs show it:** the spawn path treats a **30 s acknowledgement
window** as the health signal for an external worker. A cold worker must import the scheduler, discover
**83 plugins / 69 enabled** and register MCP tool schemas before it can acknowledge — a cost that
plainly exceeds 30 s on this host. The supervisor therefore logs the miss and **leaves the durable claim
untouched** rather than reclaiming it; the worker then starts the job anyway and is `SIGTERM`ed ~60–90 s
later, orphan-claiming the run.
**What is NOT established:** *who* sent the `SIGTERM` in these three cases. No `SIGUSR1`, no
`Restart requested`, no `drain`, no `UNCLEANLY`, and no `Previous gateway exited` marker appears in
`gateway.log`, `agent.log` or `agent.log.1` in the 07:2x–09:1x window, and the relay/polling counters
prove no process restart. Do not attribute this to a gateway reload — the signal source is unidentified.
Treat this as an open mechanism question, not a solved one.
## Blast radius and recovery
- **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`** against 161 / 4 on 10-31,
138 / 0 on 10-30 and 177 / 0 on 10-29. The two `unknown` rows are precisely
`96eb95ce12f2` @09:01:03 and `79f9bb41ecf5` @18:00:39 (a handoff failure — see
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]).
- **All three executions still reached the user** where they were meant to: the digest closed
`delivered` at 11:01:12 and the Facebook scrape `delivered` at 10:44:12 (delivery rows
`2026-11-01 11:01:12` and `2026-11-01 10:44:12`).
- **No recurrence after 09:02:18** for the remaining ~15 h of 11-01 — every later `96eb95ce12f2`
and `0c771463c3cd` slot (23 and 21 runs respectively) closed `completed`.
- The `-15` class is **not** `[SILENT]`-shaped: each instance raised an incident row and each alerted
(`85edf6` 07:31:14, `96eb95` 09:02:32, `0e753e` 09:02:29). Only the `unknown` execution status is
invisible to the dashboard's status field.
## Why it matters
A `-15`-after-adoption means the scheduler cannot say whether the job's side effects ran. For a job with
external writes (Gitea pushes, API POSTs, email sends) that is a **possible double-execution** on the
next slot. The current design resolved this properly for two of three, but the class keeps recurring in
a new form roughly every 10 days — 10-20, 10-26, 10-31, 11-01. **The recurring element across all four
variants is the same: a live cron worker is signalled mid-adoption and the durable claim is left open.**
## Open questions
1. Which supervisor sends the `-15` after a 30 s ack miss — and is the kill a timeout guard or an
unrelated cleanup path?
2. Does the ack window scale with plugin count? 83 plugins discovered in a cold worker is the visible
cost driver; if the window is fixed at 30 s, cold-start jobs on this host are structurally exposed.
3. Should a job with `failure_type: unknown` external side effects be blocked from its next slot until
`~/.hermes/cron/admissions/*.json` is reconciled? Two admission files from 11-01 are still
unreconciled (`141448f9…`, `69b5588f…`).
## Related
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]
- [[systems/scheduled-tasks]]
- [[systems/hermes-agent]]
@@ -0,0 +1,96 @@
---
title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open
type: incident
status: active
created: 2026-11-02
updated: 2026-11-02
verified_on: 2026-11-02
review_after: 2026-11-09
confidence: high
tags: [cron, gateway, handoff, delivery, weekly-review, silent-failure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-jobs-json, live-logs, live-probe]
---
# Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open
## Headline
The **Sunday 18:00 Weekly Review (`79f9bb41ecf5`, `0 18 * * 0`) did not run and nothing was delivered on
2026-11-01.** At **18:05:13** its execution closed with `status: unknown`, `delivery_outcome: NULL` and
the error *"Cron execution unverified; reconcile `…/cron/admissions/141448f9…json`: **Gateway connection
failed; no local fallback**"*. Its incident row **`79f9bb_9c443b602eec` is still `alerted` at the time of
this audit** (`first_seen` 18:05:13, `alerted_at` 18:05:32, `closed_at: NULL`) and the job reads
`last_status: error`, **`failure_streak: 1`**. There is **no `deliveries.db` row for any slot after
11:01:12 on 11-01** — the last delivery of the day was the Tech & AI Digest at 11:01:12.
## Timeline (AWST, 2026-11-01)
| Time | Event |
|---|---|
| 18:00:39 | **Four jobs handed to external workers in one batch**: `2fd50c87b0d0`, `96eb95ce12f2`, `b63ed5817212`, `79f9bb41ecf5` |
| 18:00:57 / 18:01:04 | `96eb95ce12f2` and `2fd50c87b0d0` run fine (`completed`) |
| **18:05:13.697679** | `b63ed5_6f45aeea97f0` opens — *"Cron execution unverified; reconcile `…/admissions/69b5588f…json`: Gateway connection failed; no local fallback"* (`alerted_at: None` — **never alerted**) |
| **18:05:13.708153** | `79f9bb_9c443b602eec` opens with the **identical** message, pointing at `…/admissions/141448f9…json` |
| 18:05:32 | `79f9bb` incident **alerted** |
| 18:06:21 | `b63ed5817212` execution closes `completed` (18:00:39 → 18:06:21) → its incident auto-resolves **68 s later** |
| — | `79f9bb41ecf5` closes at 18:05:13 with **no terminal state, no delivery, no recovery** |
| 18:04:24 → 18:10 | `agent.log`: `[reaper] Active gateway work detected … deferring restart`, then *"Reaper: Plan A restart completed successfully (5m02s)"* — the **Plan A/B path is RSS compaction, not a process restart** (no `SIGTERM`/`SIGUSR1` and no relay-counter reset) |
## What actually reached the user
**Nothing.** Across the whole of 11-01 `deliveries.db` holds 8 rows:
| Time | Status | Note |
|---|---|---|
| 02:39:33 | delivered | Mnemosyne-era failure notice (closes the 10-26 and 10-31 rows) |
| 03:00:37 | delivered | stocktake |
| **07:18:50** | **failed** | `configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id` — job `066bb5191280` (see [[systems/messaging-integrations]]) |
| 07:34:49 | delivered | — |
| 08:36:51 | delivered | Gmail digest |
| 10:44:12 | delivered | Facebook scrape |
| 11:01:12 | delivered | Tech & AI Digest |
| — | — | **no row after 11:01:12** |
So of the day's eight delivery attempts, **seven reached their target and one (`066bb5191280`) landed in
the DM thread-less while being recorded `failed`**. The Weekly Review's absence is not a suppressed
delivery — it is an absent run.
## Second silent gap: a job that reported `error` with no execution row at all
`ecc1fc13655a` (FreshRSS Midday News Briefing) reads **`last_status: error`, `failure_streak: 1`,
`last_run_at: 2026-11-01T12:01:00.374089+08:00`** — yet **no `executions` row exists for it on 11-01**
(the last is 2026-10-31T12:01:07, `completed`/`delivered`) and **no `cron_incidents` row was raised.**
That 12:00 slot overlapped the same `18:00:39`-style handoff batch pattern (three jobs claimed at
12:00:53). This is the **second job of the day whose failure left no durable record**, and the third
appearance of the "green/red row, no artefact" anti-pattern the wiki already documents
([[systems/scheduled-tasks]], §9 of the maintenance playbook).
## Fleet state at this audit
- **Fleet 29 jobs — 26 enabled / 3 disabled** (`jobs.json`), unchanged from 10-31.
- **Unclosed incident rows: 1 → 1, but a different row.** The 10-31 audit left
`ea79a2_a4661554a3c2` / `ea79a2_9197df32320b` / `0e753e_e7f8194a7d06` open; **all three closed on
11-01** (02:39:33 for the two `ea79a2` rows, **10:44:12** for `0e753e_e7f8194a7d06` — the
Facebook-repin row that had been open since **2026-10-01 09:00:55**, i.e. 31 days). The new sole
open row is `79f9bb_9c443b602eec`.
- **Non-clean `last_status` 4 → 2** (`ea79a2107e36`, `79f9bb41ecf5`); **failing outright 1 → 1**
(`79f9bb41ecf5`).
- **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`.**
## Open questions
1. **What did "Gateway connection failed; no local fallback" mean?** Same message for two different
jobs 11 ms apart, both in the same handoff batch, with no gateway restart anywhere in the window.
2. **Why did `b63ed5` self-heal in 68 s and `79f9bb` never do so?** Both were in the same batch with the
same error string; one closed `completed`, the other never wrote a terminal state.
3. **Should `79f9bb41ecf5` be manually re-run for 2026-11-01?** Anthony did not receive his weekly
review.
4. Are the two unreconciled admission files (`141448f9…`, `69b5588f…`) the durable record needed to
decide whether either run had side effects? Reconcile before the next 18:00 Sunday slot.
## Related
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]]
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
- [[systems/scheduled-tasks]]
- [[systems/messaging-integrations]]