--- title: "2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op" type: incident status: resolved created: 2026-11-10 updated: 2026-11-11 verified_on: 2026-11-11 confidence: high tags: [cron, job-radar, stall, silent-failure, handoff, read-loop] sources: ["executions.db (dca8482e4f76: 2026-11-09T07:00:48+08:00 → 2026-11-09T16:10:12+08:00, status completed, delivery_outcome suppressed; 5fc292ff0f3b: 11:30:38 → 12:10:07 and 14:30:34 → 14:37:27, both completed/suppressed)", "~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md (ends INCOMPLETE: tool budget exhausted; last words \\u201cI stopped retrying because I kept running read_file 5 times without making progress.\\u201d)", "~/.hermes/radar-ref/handoff/ (newest handoff on disk: 2026-11-06; no 2026-11-07/08/09 files)", "cron_incidents (no row raised for this run)"] --- # 2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op ## Summary On 2026-11-09 the Job Radar's Stage 1 (`dca8482e4f76`) ran for **9 h 09 m 24 s** (07:00:48 → 16:10:12), ended `INCOMPLETE: tool budget exhausted`, and **never wrote its handoff file**. Because both Stage-2 slots run on their own schedule, **both fired before Stage 1 finished** and each declined silently rather than erroring. Net effect: **the radar landed nothing on 11-09 and raised no alert.** ## Timeline (AWST, 2026-11-09) | Time | Event | |---|---| | 07:00:48 | Stage 1 starts | | 11:30:38 → 12:10:07 | Stage-2 slot 1 fires — Stage 1 still running → silent no-op (`suppressed`) | | 14:30:34 → 14:37:27 | Stage-2 slot 2 fires — Stage 1 still running → silent no-op (`suppressed`) | | ~16:10:1x | Stage 1 writes `INCOMPLETE: tool budget exhausted` | | 16:10:12 | Stage 1 closes — `completed`, delivery `suppressed` | | all day | `radar-ref/handoff/2026-11-09.json` is never created | ## Root Cause Two compounding faults, both previously documented for this job: 1. **Stage 2 never checks that a fresh handoff exists**, so when Stage 1 fails it proceeds and self-suppresses instead of failing loudly. Two chances to notice, zero alerting. 2. **Stage 1 overran without producing output.** Its own final message: *"I stopped retrying because I kept running read_file 5 times without making progress."* This is a **new trigger** for the same silent family seen on 11-02/11-03 (there: iteration cap) — here: an unproductive read loop that consumed the run's budget. ## Evidence - `executions.db` `dca8482e4f76`: `2026-11-09T07:00:48+08:00` → `2026-11-09T16:10:12+08:00`, `status completed`, `delivery_outcome suppressed`. - `5fc292ff0f3b` (Stage 2): two rows on 11-09, both **before** Stage 1's finish stamp, both `completed` / `suppressed`. - `~/.hermes/radar-ref/handoff/` — newest file is **2026-11-06**; nothing for 11-07, 11-08 or 11-09. - `~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md` ends `INCOMPLETE: tool budget exhausted`. - **No row in `cron_incidents`** for `dca8482e4f76` on 11-09 — the failure was never raised. ## Impact - The Job Radar — the estate's job-discovery feed — produced **no output for a third day** (no handoff since 11-06). - The failure is invisible: the run's ledger entry says `completed`, delivery is `suppressed`, and no incident row exists. ## Fix Applied **None — documentation only.** No configuration change was made this window. ## Prevention - **Alert on the artefact, not the exit code.** Stage 2 should hard-fail when the expected `radar-ref/handoff/.json` is missing, rather than proceeding to a suppression. That single check converts both 11-09 slots into loud failures. - Add a **read-progress guard** to Stage 1: when the same file has been read N times without a state change, stop and emit a partial handoff instead of burning the budget to `INCOMPLETE`. - Note the pattern across incidents: **completion-time-only alerting hides every failure in this family** (stocktake 11-09, radar 11-09, radar 11-02/03). ## Recovery — 2026-11-11: the handoff landed, and Stage 2 delivered | Field | 11-09 (this incident) | **11-11** | |---|---|---| | Stage 1 | 07:00:48 → 16:10:12 (**9 h 09 m 24 s**), no handoff | **07:01:08 → 07:37:34 = 36 m 26 s**, `completed` | | Handoff on disk | newest was **2026-11-06** | **`radar-ref/handoff/2026-11-11.json` — 22,682 B, mtime 07:59** | | Stage 2 (11:30 slot) | fired early, declined silently | **11:30:56 → 11:35:18**, `completed` | | Stage 2 (14:30 slot) | declined silently | **14:30:55 → 14:34:56**, `completed` / **`delivered`** | | Incident row | **none raised** | `5fc292_773822455856` closed **`resolved` 2026-11-11T08:03:47.644226** | - **The first handoff in five days** (`2026-11-06` → `2026-11-11`) and the fastest Stage-1 cycle since the two-stage split: **36 m 26 s** against 3 h 15 m (10-29, the last full success), 9 h 01 m (11-02), 6 h 28 m (11-03) and 9 h 09 m (11-09). - **Stage 2's recovery fire** (`source: direct`, 07:35:56 → 08:03:47) consumed the handoff and delivered at **08:03:47**; its completion is what auto-closed the row opened by 11-10's `interrupted` slot. - ⚠ **Carried forward from 11-10:** the *scheduled* Stage-2 slot that day ran **11:30:35 → 22:18:49 = 10 h 48 m 14 s** and closed **`failed` / `interrupted`** (delivery `delivered`, `for_failure: 1`) — the longest run of either window, and itself a new data point for the "completion-time-only alerting" family on this page. - **Job state:** both radar jobs read `last_status: ok`, `failure_streak` 0. **Status: `resolved`.** Both prevention rules on this page still stand and remain unimplemented: Stage 2 should hard-fail on a missing `handoff/.json` rather than suppressing, and Stage 1 needs a read-progress guard. Neither was needed on 11-11 because the run finished inside budget. ## Related - [[systems/scheduled-tasks]] - [[incidents/2026-11-09-stocktake-completes-with-zero-edits]] - [[runbooks/provider-health]]