--- title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response type: incident status: resolved created: 2026-10-28 updated: 2026-10-30 verified_on: 2026-10-30 review_after: 2026-11-13 confidence: high tags: [automation, cron, jobs, notion, silent-failure, scheduling] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api] --- # Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response ## Headline On **2026-10-27** the Daily Marketing Job Radar (`dca8482e4f76`) ran **07:01:08 → 11:03:54 AWST** (**4 h 2 m 46 s** wall clock), closed **`failed`**, and delivered a report that opens: > **Daily Marketing Job Radar — run 2026-10-27 — INCOMPLETE (tool-call budget exhausted before Notion write stage)** Discovery, verification, dedup and classification completed. **Notion ingestion, cover-letter generation and the final cover-letter audit were not executed** — so the run is recorded as failed rather than successful. Self-reported run time **10:47:57**; freshness cutoff **2026-10-20**. This is the second consecutive budget overrun: the **2026-10-26** 07:00 slot ran 07:01:08 → 09:10:27 (**2 h 9 m 19 s**), closed **`completed`**, and *still* delivered the same `RUN INCOMPLETE` marker — the day's backlog had to be finished by hand. 2026-10-27 is the first overrun recorded as **`failed`**, which is the improvement: the failure reached the incident ledger instead of hiding inside a green row. ## Ledger state | Field | Value | |---|---| | Execution row | `dca8482e4f76`, started `2026-10-27T07:01:08.265913+08:00`, finished `2026-10-27T11:03:54.143535+08:00`, `failed` | | Delivery | `delivered` (the incomplete report reached Telegram topic 1030) | | Incident row | `dca848_e93a538ade6b` — `state: alerted`, `failure_type: auth`, first/last seen `2026-10-27T11:03:53.715494+08:00`, alerted `11:03:59.013115` | | Incident status | **open at the 2026-10-28 00:2x audit** — no ack, no close | | `jobs.json` | `last_status: error`, `failure_streak: 1`, pin unchanged at `custom:omni-101` / `smart-route` | ⚠ `failure_type: auth` does **not** describe this fault. A tool-call budget exhaustion is being classified as an auth failure; the field is not trustworthy as a diagnosis for this job. ## What the run actually did (from its own report) - Live Notion DB `0f90ba2b-8b10-4d02-a62c-6f692d5b1168`: **451 entries**, `Date Added = 2026-10-27` → **0**. - Board composition at run time: **Agent — Rhino 402, Chaos 12, Manual 2, blank 35**; **Source — SEEK 295, LinkedIn 88** (row text truncated in the ledger). ## Response, same day: the radar is now two jobs At **2026-10-27 17:34:15** the single radar job was split into a two-stage pipeline (see [[runbooks/job-radar]] for the full contract): | Stage | Job | Schedule | Delivery | Job id | |---|---|---|---|---| | 1 — Discovery | Daily Marketing Job Radar - Discovery — `DISCOVERY → VERIFY → DEDUP → SCORE`, no Notion write, no cover letters; writes a handoff file | `0 7 * * 1-5` | `local` (failures → Telegram topic 1030) | `dca8482e4f76` (unchanged) | | 2 — Write and Letters | Daily Marketing Job Radar - Write and Letters — lands the handoff: Notion entries, cover letters, audit, report | `30 11,14 * * 1-5` | Telegram group `-1004321904721`, topic `1030` | `5fc292ff0f3b` (created 17:34:15) | Stage 1 keeps its id, pin (`custom:omni-101` / `smart-route`) and its own tool budget; stage 2 is a separate job whose prompt lands the handoff rather than repeating the discovery narrative. Two test fires of stage 2 ran the same evening — **17:35:00 → 17:36:52** and **17:38:38 → 17:43:42** — both `completed`, both with delivery `suppressed`, `last_status: ok`, streak 0. Fleet size went **28 → 29 jobs** (26 enabled / 3 disabled). ## Why it matters 1. **The failure mode is partial work, not a lost run.** Four hours of browser-heavy discovery produced a correct report and zero writes. Nothing in `jobs.json` distinguished it from a good day except `last_status: error` on 10-27 (`completed` on 10-26, with the same incompleteness buried in the text). 2. **Splitting is the mitigation, not a fix.** Stage 2 inherits its own tool budget; if stage 1 still overruns and never writes the handoff file, stage 2 has nothing to land — watch the 11:30 slot for that. 3. **The budget is consumed by discovery**, which is exactly the stage that now owns it alone. ## Open questions 1. **Verify one full cycle end-to-end** (stage 1 07:00 → stage 2 11:30): does stage 2 find stage 1's handoff file, do the Notion writes and cover letters land, and is `MISSING_COVER_LETTER_COUNT=0` reported? 2. **Make budget exhaustion self-describing.** The classifier tagged it `auth`; a distinct budget/partial-run type would put this on the same footing as any other failure. 3. **Do the two stages fit the same wall-clock envelope?** Stage 1 must finish inside its own budget *and* leave the handoff before 11:30. 4. The incident row stays open until a clean stage-1 slot auto-closes it — confirm the 2026-10-28 07:00 slot closes `dca848_e93a538ade6b`. ## Follow-up — the 2026-10-28 slot (recorded 2026-10-29) The **first real two-stage cycle ran on 2026-10-28** and answered the open questions above — not favourably: - **Stage 1 overran again, on a new signature.** `dca8482e4f76` ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, `failure_streak` **2**, on *"hit the agent iteration cap before the handoff could be written"* → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. - **No handoff file was written** — `~/.hermes/radar-ref/handoff/` held only `2026-10-27.json` at the 2026-10-29 audit, so **Notion writes and cover letters did not run again**. - **Both stage-2 slots no-oped correctly** — `5fc292ff0f3b` fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` with `delivery_outcome: suppressed`. Open question 1 is therefore answered only in the negative: the recovery rule ("a stage-2 run with no handoff is a no-op") holds in production, but nothing has yet proved the positive path end-to-end. - **Open question 4 is answered: the row did not close.** `dca848_e93a538ade6b` stayed open because the 10-28 slot was not clean, and a second row (`dca848_0a2f03b5952b`) opened — open incident rows **2 → 4**. ## Follow-up — the 2026-10-29 slot: the split held (recorded 2026-10-30) **Resolved.** Stage 1 (`dca8482e4f76`) ran **2026-10-29 07:00:59 → 10:16:12** — 3 h 15 m 13 s — and closed **`completed`**. It wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` (24.8 KB): `run_date 2026-10-29T08:23:51+08:00`, **170 discovered / 33 duplicates avoided / 37 verified**, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Both ledger rows this incident opened — `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` — closed **`resolved` at 10:16:12**, and the run opened no new row. Open cron incident rows went **4 → 1**; the survivor is the Facebook monthly's `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Stage 2 (`5fc292ff0f3b`) fired at **10:13:46**, consumed the handoff at **10:40:59**, finished **10:44:48 `completed`** and **delivered** to topic 1030 in the same second (delivery `delivered`). Its `stage2_result`: **20 pages written / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went **455 → 475**. The later slots no-oped as designed (11:30 → 10 m 46 s, 14:30 → 1 m 51 s, already-consumed handoff). Stage 1 itself no longer delivers — correct, because stage 2 owns the report. The questions this page left open are answered for this run: the handoff **was** written inside budget and consumed, and **no cover letter went missing on the first real run of the split**. ## Related - [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun - [[systems/scheduled-tasks]] — fleet ledger, job table and the 2026-10-27 window - [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — the radar's previous failure mode