wiki audit 2026-11-07: Job Radar concurrent-fire claim discard (new incident page), scheduled-tasks re-verified section, index+log+runbook updates, freshness 2026-11-07
This commit is contained in:
@@ -0,0 +1,86 @@
|
||||
---
|
||||
title: Job Radar — a concurrent fire discarded the 11:30 slot's result
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-11-07
|
||||
updated: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
review_after: 2026-11-14
|
||||
confidence: high
|
||||
tags: [cron, job-radar, fire-claim, concurrency, scheduler]
|
||||
sources: [live-executions-db, live-deliveries-db, live-agent-log, live-cron-incidents, cron-output]
|
||||
---
|
||||
|
||||
# Job Radar — a concurrent fire discarded the 11:30 slot's result (2026-11-06)
|
||||
|
||||
**Summary.** On **2026-11-06** the *Daily Marketing Job Radar — Write and Letters* (`5fc292ff0f3b`,
|
||||
schedule `30 11,14 * * 1-5`) had **two live runs at the same time**. A `direct`-source fire started at
|
||||
**08:27:09** and ran **4 h 59 m 13 s**, completing and delivering its report at **13:26:22**. The
|
||||
scheduler's own **11:30:32** slot had already started and was still running; when the direct fire settled
|
||||
the claim, the 11:30 run was judged **stale**, interrupted at **13:26:38**, and closed **`failed`** at
|
||||
**13:26:44** with a **new** error string:
|
||||
|
||||
```
|
||||
Fire claim ownership lost; stale result was discarded.
|
||||
```
|
||||
|
||||
This is the **first occurrence of a claim-loss failure with no gateway restart involved**, and it raised
|
||||
**no `cron_incidents` row** — the fleet's only red row for the day is invisible to the alerting ledger.
|
||||
|
||||
## Observed sequence (live `executions.db`, `deliveries.db`, `agent.log.1`)
|
||||
|
||||
| Time (AWST) | Event | Evidence |
|
||||
|---|---|---|
|
||||
| 03:00:32 -> 04:11:35 | wiki stocktake (`1a8e90ea2271`) completes in 1 h 11 m 03 s — overrun streak over | executions row |
|
||||
| 07:00:44 -> 08:29:21 | Stage 1 (`dca8482e4f76`) completes (1 h 28 m 37 s, `suppressed`), second clean run running | executions row |
|
||||
| **08:27:09 -> 13:26:22** | **`direct` fire of `5fc292ff0f3b`** — 4 h 59 m 13 s, `completed`, **delivered** | row `ad1fe896…`; log 08:27:13 *"Running job 'Daily Marketing Job Radar - Write and Letters'"* |
|
||||
| 11:30:32 -> 13:26:44 | scheduler's **own 11:30 slot** fires and runs **concurrently** for 1 h 56 m 12 s | row `95347fa5…` |
|
||||
| 13:26:22 | direct fire finishes and delivers its report | delivery row 13:26:22 |
|
||||
| **13:26:38** | *"Job '5fc292ff0f3b': fire claim ownership lost; interrupting stale run"* | `agent.log.1` WARNING |
|
||||
| **13:26:43** | *"Job '…' failed: RuntimeError: Cron job '…' lost its durable fire claim"* | `agent.log.1` ERROR |
|
||||
| **13:26:44** | 11:30 row closes **`failed`** — `Fire claim ownership lost; stale result was discarded.` | executions row |
|
||||
| 14:30:32 -> 14:42:59 | 14:30 slot runs a normal cycle (**12 m 27 s**) and delivers a complete report | row `e7462cf1…` |
|
||||
|
||||
**No `.md` output file** exists for the discarded 11:30 run — on 11-06 that job produced only
|
||||
`2026-11-06_13-26-22.md` (the direct fire) and `2026-11-06_14-42-58.md` (the 14:30 slot).
|
||||
|
||||
## What is *not* affected
|
||||
|
||||
The day still landed in full. The 14:30 report (read from `cron/output/5fc292ff0f3b/2026-11-06_14-42-58.md`)
|
||||
shows `SAME_DAY_STRONG_COUNT 4`, `VERIFIED_COVER_LETTER_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`, cache
|
||||
total **494**, 10 job rows dated 11-06, 20 artifacts present and non-empty, *Run status: COMPLETE*.
|
||||
|
||||
## Why it matters
|
||||
|
||||
1. **New cause for a known string.** Every earlier `lost its durable fire claim` case came from a
|
||||
**restart or reload orphaning a live run** ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]],
|
||||
[[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]), where the text was *"lost its durable
|
||||
fire claim ownership"*. Here **no restart occurred** — the claim was lost to *another fire of the same
|
||||
job*. The string is a **generic claim-loss signal, not a restart indicator**; and `executions.db`
|
||||
currently holds **exactly one** row whose `error` contains `fire claim`: this one.
|
||||
2. **The failure produced no alert.** No `cron_incidents` row has `last_seen_at` on or after 2026-11-06 —
|
||||
the ledger's newest activity remains the 2026-11-05T12:36:17 closures. A `failed` execution with no
|
||||
ledger row means **no notification path fired**; the run only surfaces as a red row in `executions.db`.
|
||||
3. **~2 hours of a scheduled slot's work was discarded, and the *later, scheduler-owned* run lost** to an
|
||||
earlier out-of-band run. The stale-discard rule did what it was written to do (protect against stale
|
||||
results) — but the loser was the scheduler's own slot, so the guard silently ate planned work.
|
||||
4. **The winner was out-of-band.** Its `source` is `direct`, not the scheduler's `builtin`, and it started
|
||||
~2 minutes before Stage 1's handover run closed (08:29:21). The timing is consistent with
|
||||
[[runbooks/job-radar]]'s Stage-1 handover trigger; the trigger's exact origin was **not directly verified**
|
||||
in this audit.
|
||||
|
||||
## Open questions / candidate fixes (none applied)
|
||||
|
||||
- Should the scheduler **skip or suppress** a slot when a live claim for the same job already exists,
|
||||
instead of launching a second concurrent run?
|
||||
- Should an **out-of-band `direct` fire** (e.g. Stage 1's handover) mark the next scheduled slot consumed,
|
||||
the way a consumed handoff file silences it?
|
||||
- Should a **discarded-stale run raise an incident row** at all? Today the fault is silent by construction —
|
||||
a `failed` execution with no ledger row and no alert.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[runbooks/job-radar]]
|
||||
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
|
||||
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]
|
||||
Reference in New Issue
Block a user