--- title: "2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op" type: incident status: active created: 2026-11-10 updated: 2026-11-10 verified_on: 2026-11-10 confidence: high tags: [cron, job-radar, stall, silent-failure, handoff, read-loop] sources: ["executions.db (dca8482e4f76: 2026-11-09T07:00:48+08:00 → 2026-11-09T16:10:12+08:00, status completed, delivery_outcome suppressed; 5fc292ff0f3b: 11:30:38 → 12:10:07 and 14:30:34 → 14:37:27, both completed/suppressed)", "~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md (ends INCOMPLETE: tool budget exhausted; last words \\u201cI stopped retrying because I kept running read_file 5 times without making progress.\\u201d)", "~/.hermes/radar-ref/handoff/ (newest handoff on disk: 2026-11-06; no 2026-11-07/08/09 files)", "cron_incidents (no row raised for this run)"] --- # 2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op ## Summary On 2026-11-09 the Job Radar's Stage 1 (`dca8482e4f76`) ran for **9 h 09 m 24 s** (07:00:48 → 16:10:12), ended `INCOMPLETE: tool budget exhausted`, and **never wrote its handoff file**. Because both Stage-2 slots run on their own schedule, **both fired before Stage 1 finished** and each declined silently rather than erroring. Net effect: **the radar landed nothing on 11-09 and raised no alert.** ## Timeline (AWST, 2026-11-09) | Time | Event | |---|---| | 07:00:48 | Stage 1 starts | | 11:30:38 → 12:10:07 | Stage-2 slot 1 fires — Stage 1 still running → silent no-op (`suppressed`) | | 14:30:34 → 14:37:27 | Stage-2 slot 2 fires — Stage 1 still running → silent no-op (`suppressed`) | | ~16:10:1x | Stage 1 writes `INCOMPLETE: tool budget exhausted` | | 16:10:12 | Stage 1 closes — `completed`, delivery `suppressed` | | all day | `radar-ref/handoff/2026-11-09.json` is never created | ## Root Cause Two compounding faults, both previously documented for this job: 1. **Stage 2 never checks that a fresh handoff exists**, so when Stage 1 fails it proceeds and self-suppresses instead of failing loudly. Two chances to notice, zero alerting. 2. **Stage 1 overran without producing output.** Its own final message: *"I stopped retrying because I kept running read_file 5 times without making progress."* This is a **new trigger** for the same silent family seen on 11-02/11-03 (there: iteration cap) — here: an unproductive read loop that consumed the run's budget. ## Evidence - `executions.db` `dca8482e4f76`: `2026-11-09T07:00:48+08:00` → `2026-11-09T16:10:12+08:00`, `status completed`, `delivery_outcome suppressed`. - `5fc292ff0f3b` (Stage 2): two rows on 11-09, both **before** Stage 1's finish stamp, both `completed` / `suppressed`. - `~/.hermes/radar-ref/handoff/` — newest file is **2026-11-06**; nothing for 11-07, 11-08 or 11-09. - `~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md` ends `INCOMPLETE: tool budget exhausted`. - **No row in `cron_incidents`** for `dca8482e4f76` on 11-09 — the failure was never raised. ## Impact - The Job Radar — the estate's job-discovery feed — produced **no output for a third day** (no handoff since 11-06). - The failure is invisible: the run's ledger entry says `completed`, delivery is `suppressed`, and no incident row exists. ## Fix Applied **None — documentation only.** No configuration change was made this window. ## Prevention - **Alert on the artefact, not the exit code.** Stage 2 should hard-fail when the expected `radar-ref/handoff/.json` is missing, rather than proceeding to a suppression. That single check converts both 11-09 slots into loud failures. - Add a **read-progress guard** to Stage 1: when the same file has been read N times without a state change, stop and emit a partial handoff instead of burning the budget to `INCOMPLETE`. - Note the pattern across incidents: **completion-time-only alerting hides every failure in this family** (stocktake 11-09, radar 11-09, radar 11-02/03). ## Related - [[systems/scheduled-tasks]] - [[incidents/2026-11-09-stocktake-completes-with-zero-edits]] - [[runbooks/provider-health]]