5.3 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Job Radar — a concurrent fire discarded the 11:30 slot's result | incident | open | 2026-11-07 | 2026-11-07 | 2026-11-07 | 2026-11-14 | high |
|
|
Job Radar — a concurrent fire discarded the 11:30 slot's result (2026-11-06)
Summary. On 2026-11-06 the Daily Marketing Job Radar — Write and Letters (5fc292ff0f3b,
schedule 30 11,14 * * 1-5) had two live runs at the same time. A direct-source fire started at
08:27:09 and ran 4 h 59 m 13 s, completing and delivering its report at 13:26:22. The
scheduler's own 11:30:32 slot had already started and was still running; when the direct fire settled
the claim, the 11:30 run was judged stale, interrupted at 13:26:38, and closed failed at
13:26:44 with a new error string:
Fire claim ownership lost; stale result was discarded.
This is the first occurrence of a claim-loss failure with no gateway restart involved, and it raised
no cron_incidents row — the fleet's only red row for the day is invisible to the alerting ledger.
Observed sequence (live executions.db, deliveries.db, agent.log.1)
| Time (AWST) | Event | Evidence |
|---|---|---|
| 03:00:32 -> 04:11:35 | wiki stocktake (1a8e90ea2271) completes in 1 h 11 m 03 s — overrun streak over |
executions row |
| 07:00:44 -> 08:29:21 | Stage 1 (dca8482e4f76) completes (1 h 28 m 37 s, suppressed), second clean run running |
executions row |
| 08:27:09 -> 13:26:22 | direct fire of 5fc292ff0f3b — 4 h 59 m 13 s, completed, delivered |
row ad1fe896…; log 08:27:13 "Running job 'Daily Marketing Job Radar - Write and Letters'" |
| 11:30:32 -> 13:26:44 | scheduler's own 11:30 slot fires and runs concurrently for 1 h 56 m 12 s | row 95347fa5… |
| 13:26:22 | direct fire finishes and delivers its report | delivery row 13:26:22 |
| 13:26:38 | "Job '5fc292ff0f3b': fire claim ownership lost; interrupting stale run" | agent.log.1 WARNING |
| 13:26:43 | "Job '…' failed: RuntimeError: Cron job '…' lost its durable fire claim" | agent.log.1 ERROR |
| 13:26:44 | 11:30 row closes failed — Fire claim ownership lost; stale result was discarded. |
executions row |
| 14:30:32 -> 14:42:59 | 14:30 slot runs a normal cycle (12 m 27 s) and delivers a complete report | row e7462cf1… |
No .md output file exists for the discarded 11:30 run — on 11-06 that job produced only
2026-11-06_13-26-22.md (the direct fire) and 2026-11-06_14-42-58.md (the 14:30 slot).
What is not affected
The day still landed in full. The 14:30 report (read from cron/output/5fc292ff0f3b/2026-11-06_14-42-58.md)
shows SAME_DAY_STRONG_COUNT 4, VERIFIED_COVER_LETTER_COUNT 4, MISSING_COVER_LETTER_COUNT 0, cache
total 494, 10 job rows dated 11-06, 20 artifacts present and non-empty, Run status: COMPLETE.
Why it matters
- New cause for a known string. Every earlier
lost its durable fire claimcase came from a restart or reload orphaning a live run (incidents/2026-10-26-planned-restart-orphaned-cron-run, incidents/2026-10-31-gateway-reload-force-drain-orphaned-run), where the text was "lost its durable fire claim ownership". Here no restart occurred — the claim was lost to another fire of the same job. The string is a generic claim-loss signal, not a restart indicator; andexecutions.dbcurrently holds exactly one row whoseerrorcontainsfire claim: this one. - The failure produced no alert. No
cron_incidentsrow haslast_seen_aton or after 2026-11-06 — the ledger's newest activity remains the 2026-11-05T12:36:17 closures. Afailedexecution with no ledger row means no notification path fired; the run only surfaces as a red row inexecutions.db. - ~2 hours of a scheduled slot's work was discarded, and the later, scheduler-owned run lost to an earlier out-of-band run. The stale-discard rule did what it was written to do (protect against stale results) — but the loser was the scheduler's own slot, so the guard silently ate planned work.
- The winner was out-of-band. Its
sourceisdirect, not the scheduler'sbuiltin, and it started ~2 minutes before Stage 1's handover run closed (08:29:21). The timing is consistent with runbooks/job-radar's Stage-1 handover trigger; the trigger's exact origin was not directly verified in this audit.
Open questions / candidate fixes (none applied)
- Should the scheduler skip or suppress a slot when a live claim for the same job already exists, instead of launching a second concurrent run?
- Should an out-of-band
directfire (e.g. Stage 1's handover) mark the next scheduled slot consumed, the way a consumed handoff file silences it? - Should a discarded-stale run raise an incident row at all? Today the fault is silent by construction —
a
failedexecution with no ledger row and no alert.