Files
agent-estate-wiki/incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
T

5.3 KiB

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Job Radar — a concurrent fire discarded the 11:30 slot's result incident open 2026-11-07 2026-11-07 2026-11-07 2026-11-14 high
cron
job-radar
fire-claim
concurrency
scheduler
live-executions-db
live-deliveries-db
live-agent-log
live-cron-incidents
cron-output

Job Radar — a concurrent fire discarded the 11:30 slot's result (2026-11-06)

Summary. On 2026-11-06 the Daily Marketing Job Radar — Write and Letters (5fc292ff0f3b, schedule 30 11,14 * * 1-5) had two live runs at the same time. A direct-source fire started at 08:27:09 and ran 4 h 59 m 13 s, completing and delivering its report at 13:26:22. The scheduler's own 11:30:32 slot had already started and was still running; when the direct fire settled the claim, the 11:30 run was judged stale, interrupted at 13:26:38, and closed failed at 13:26:44 with a new error string:

Fire claim ownership lost; stale result was discarded.

This is the first occurrence of a claim-loss failure with no gateway restart involved, and it raised no cron_incidents row — the fleet's only red row for the day is invisible to the alerting ledger.

Observed sequence (live executions.db, deliveries.db, agent.log.1)

Time (AWST) Event Evidence
03:00:32 -> 04:11:35 wiki stocktake (1a8e90ea2271) completes in 1 h 11 m 03 s — overrun streak over executions row
07:00:44 -> 08:29:21 Stage 1 (dca8482e4f76) completes (1 h 28 m 37 s, suppressed), second clean run running executions row
08:27:09 -> 13:26:22 direct fire of 5fc292ff0f3b — 4 h 59 m 13 s, completed, delivered row ad1fe896…; log 08:27:13 "Running job 'Daily Marketing Job Radar - Write and Letters'"
11:30:32 -> 13:26:44 scheduler's own 11:30 slot fires and runs concurrently for 1 h 56 m 12 s row 95347fa5…
13:26:22 direct fire finishes and delivers its report delivery row 13:26:22
13:26:38 "Job '5fc292ff0f3b': fire claim ownership lost; interrupting stale run" agent.log.1 WARNING
13:26:43 "Job '…' failed: RuntimeError: Cron job '…' lost its durable fire claim" agent.log.1 ERROR
13:26:44 11:30 row closes failed — Fire claim ownership lost; stale result was discarded. executions row
14:30:32 -> 14:42:59 14:30 slot runs a normal cycle (12 m 27 s) and delivers a complete report row e7462cf1…

No .md output file exists for the discarded 11:30 run — on 11-06 that job produced only 2026-11-06_13-26-22.md (the direct fire) and 2026-11-06_14-42-58.md (the 14:30 slot).

What is not affected

The day still landed in full. The 14:30 report (read from cron/output/5fc292ff0f3b/2026-11-06_14-42-58.md) shows SAME_DAY_STRONG_COUNT 4, VERIFIED_COVER_LETTER_COUNT 4, MISSING_COVER_LETTER_COUNT 0, cache total 494, 10 job rows dated 11-06, 20 artifacts present and non-empty, Run status: COMPLETE.

Why it matters

  1. New cause for a known string. Every earlier lost its durable fire claim case came from a restart or reload orphaning a live run (incidents/2026-10-26-planned-restart-orphaned-cron-run, incidents/2026-10-31-gateway-reload-force-drain-orphaned-run), where the text was "lost its durable fire claim ownership". Here no restart occurred — the claim was lost to another fire of the same job. The string is a generic claim-loss signal, not a restart indicator; and executions.db currently holds exactly one row whose error contains fire claim: this one.
  2. The failure produced no alert. No cron_incidents row has last_seen_at on or after 2026-11-06 — the ledger's newest activity remains the 2026-11-05T12:36:17 closures. A failed execution with no ledger row means no notification path fired; the run only surfaces as a red row in executions.db.
  3. ~2 hours of a scheduled slot's work was discarded, and the later, scheduler-owned run lost to an earlier out-of-band run. The stale-discard rule did what it was written to do (protect against stale results) — but the loser was the scheduler's own slot, so the guard silently ate planned work.
  4. The winner was out-of-band. Its source is direct, not the scheduler's builtin, and it started ~2 minutes before Stage 1's handover run closed (08:29:21). The timing is consistent with runbooks/job-radar's Stage-1 handover trigger; the trigger's exact origin was not directly verified in this audit.

Open questions / candidate fixes (none applied)

  • Should the scheduler skip or suppress a slot when a live claim for the same job already exists, instead of launching a second concurrent run?
  • Should an out-of-band direct fire (e.g. Stage 1's handover) mark the next scheduled slot consumed, the way a consumed handoff file silences it?
  • Should a discarded-stale run raise an incident row at all? Today the fault is silent by construction — a failed execution with no ledger row and no alert.