--- title: Job Radar — a concurrent fire discarded the 11:30 slot's result type: incident status: open created: 2026-11-07 updated: 2026-11-07 verified_on: 2026-11-07 review_after: 2026-11-14 confidence: high tags: [cron, job-radar, fire-claim, concurrency, scheduler] sources: [live-executions-db, live-deliveries-db, live-agent-log, live-cron-incidents, cron-output] --- # Job Radar — a concurrent fire discarded the 11:30 slot's result (2026-11-06) **Summary.** On **2026-11-06** the *Daily Marketing Job Radar — Write and Letters* (`5fc292ff0f3b`, schedule `30 11,14 * * 1-5`) had **two live runs at the same time**. A `direct`-source fire started at **08:27:09** and ran **4 h 59 m 13 s**, completing and delivering its report at **13:26:22**. The scheduler's own **11:30:32** slot had already started and was still running; when the direct fire settled the claim, the 11:30 run was judged **stale**, interrupted at **13:26:38**, and closed **`failed`** at **13:26:44** with a **new** error string: ``` Fire claim ownership lost; stale result was discarded. ``` This is the **first occurrence of a claim-loss failure with no gateway restart involved**, and it raised **no `cron_incidents` row** — the fleet's only red row for the day is invisible to the alerting ledger. ## Observed sequence (live `executions.db`, `deliveries.db`, `agent.log.1`) | Time (AWST) | Event | Evidence | |---|---|---| | 03:00:32 -> 04:11:35 | wiki stocktake (`1a8e90ea2271`) completes in 1 h 11 m 03 s — overrun streak over | executions row | | 07:00:44 -> 08:29:21 | Stage 1 (`dca8482e4f76`) completes (1 h 28 m 37 s, `suppressed`), second clean run running | executions row | | **08:27:09 -> 13:26:22** | **`direct` fire of `5fc292ff0f3b`** — 4 h 59 m 13 s, `completed`, **delivered** | row `ad1fe896…`; log 08:27:13 *"Running job 'Daily Marketing Job Radar - Write and Letters'"* | | 11:30:32 -> 13:26:44 | scheduler's **own 11:30 slot** fires and runs **concurrently** for 1 h 56 m 12 s | row `95347fa5…` | | 13:26:22 | direct fire finishes and delivers its report | delivery row 13:26:22 | | **13:26:38** | *"Job '5fc292ff0f3b': fire claim ownership lost; interrupting stale run"* | `agent.log.1` WARNING | | **13:26:43** | *"Job '…' failed: RuntimeError: Cron job '…' lost its durable fire claim"* | `agent.log.1` ERROR | | **13:26:44** | 11:30 row closes **`failed`** — `Fire claim ownership lost; stale result was discarded.` | executions row | | 14:30:32 -> 14:42:59 | 14:30 slot runs a normal cycle (**12 m 27 s**) and delivers a complete report | row `e7462cf1…` | **No `.md` output file** exists for the discarded 11:30 run — on 11-06 that job produced only `2026-11-06_13-26-22.md` (the direct fire) and `2026-11-06_14-42-58.md` (the 14:30 slot). ## What is *not* affected The day still landed in full. The 14:30 report (read from `cron/output/5fc292ff0f3b/2026-11-06_14-42-58.md`) shows `SAME_DAY_STRONG_COUNT 4`, `VERIFIED_COVER_LETTER_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`, cache total **494**, 10 job rows dated 11-06, 20 artifacts present and non-empty, *Run status: COMPLETE*. ## Why it matters 1. **New cause for a known string.** Every earlier `lost its durable fire claim` case came from a **restart or reload orphaning a live run** ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]], [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]), where the text was *"lost its durable fire claim ownership"*. Here **no restart occurred** — the claim was lost to *another fire of the same job*. The string is a **generic claim-loss signal, not a restart indicator**; and `executions.db` currently holds **exactly one** row whose `error` contains `fire claim`: this one. 2. **The failure produced no alert.** No `cron_incidents` row has `last_seen_at` on or after 2026-11-06 — the ledger's newest activity remains the 2026-11-05T12:36:17 closures. A `failed` execution with no ledger row means **no notification path fired**; the run only surfaces as a red row in `executions.db`. 3. **~2 hours of a scheduled slot's work was discarded, and the *later, scheduler-owned* run lost** to an earlier out-of-band run. The stale-discard rule did what it was written to do (protect against stale results) — but the loser was the scheduler's own slot, so the guard silently ate planned work. 4. **The winner was out-of-band.** Its `source` is `direct`, not the scheduler's `builtin`, and it started ~2 minutes before Stage 1's handover run closed (08:29:21). The timing is consistent with [[runbooks/job-radar]]'s Stage-1 handover trigger; the trigger's exact origin was **not directly verified** in this audit. ## Open questions / candidate fixes (none applied) - Should the scheduler **skip or suppress** a slot when a live claim for the same job already exists, instead of launching a second concurrent run? - Should an **out-of-band `direct` fire** (e.g. Stage 1's handover) mark the next scheduled slot consumed, the way a consumed handoff file silences it? - Should a **discarded-stale run raise an incident row** at all? Today the fault is silent by construction — a `failed` execution with no ledger row and no alert. ## Related - [[systems/scheduled-tasks]] - [[runbooks/job-radar]] - [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]] - [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]