Files
agent-estate-wiki/incidents/2026-10-27-job-radar-budget-exhausted.md
T

8.2 KiB

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response incident resolved 2026-10-28 2026-10-30 2026-10-30 2026-11-13 high
automation
cron
jobs
notion
silent-failure
scheduling
live-sqlite-cron-executions
live-sqlite-cron-incidents
live-jobs-json
live-notion-api

Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response

Headline

On 2026-10-27 the Daily Marketing Job Radar (dca8482e4f76) ran 07:01:08 → 11:03:54 AWST (4 h 2 m 46 s wall clock), closed failed, and delivered a report that opens:

Daily Marketing Job Radar — run 2026-10-27 — INCOMPLETE (tool-call budget exhausted before Notion write stage)

Discovery, verification, dedup and classification completed. Notion ingestion, cover-letter generation and the final cover-letter audit were not executed — so the run is recorded as failed rather than successful. Self-reported run time 10:47:57; freshness cutoff 2026-10-20.

This is the second consecutive budget overrun: the 2026-10-26 07:00 slot ran 07:01:08 → 09:10:27 (2 h 9 m 19 s), closed completed, and still delivered the same RUN INCOMPLETE marker — the day's backlog had to be finished by hand. 2026-10-27 is the first overrun recorded as failed, which is the improvement: the failure reached the incident ledger instead of hiding inside a green row.

Ledger state

Field Value
Execution row dca8482e4f76, started 2026-10-27T07:01:08.265913+08:00, finished 2026-10-27T11:03:54.143535+08:00, failed
Delivery delivered (the incomplete report reached Telegram topic 1030)
Incident row dca848_e93a538ade6b — state: alerted, failure_type: auth, first/last seen 2026-10-27T11:03:53.715494+08:00, alerted 11:03:59.013115
Incident status open at the 2026-10-28 00:2x audit — no ack, no close
jobs.json last_status: error, failure_streak: 1, pin unchanged at custom:omni-101 / smart-route

⚠ failure_type: auth does not describe this fault. A tool-call budget exhaustion is being classified as an auth failure; the field is not trustworthy as a diagnosis for this job.

What the run actually did (from its own report)

  • Live Notion DB 0f90ba2b-8b10-4d02-a62c-6f692d5b1168: 451 entries, Date Added = 2026-10-27 → 0.
  • Board composition at run time: Agent — Rhino 402, Chaos 12, Manual 2, blank 35; Source — SEEK 295, LinkedIn 88 (row text truncated in the ledger).

Response, same day: the radar is now two jobs

At 2026-10-27 17:34:15 the single radar job was split into a two-stage pipeline (see runbooks/job-radar for the full contract):

Stage Job Schedule Delivery Job id
1 — Discovery Daily Marketing Job Radar - Discovery — DISCOVERY → VERIFY → DEDUP → SCORE, no Notion write, no cover letters; writes a handoff file 0 7 * * 1-5 local (failures → Telegram topic 1030) dca8482e4f76 (unchanged)
2 — Write and Letters Daily Marketing Job Radar - Write and Letters — lands the handoff: Notion entries, cover letters, audit, report 30 11,14 * * 1-5 Telegram group -1004321904721, topic 1030 5fc292ff0f3b (created 17:34:15)

Stage 1 keeps its id, pin (custom:omni-101 / smart-route) and its own tool budget; stage 2 is a separate job whose prompt lands the handoff rather than repeating the discovery narrative. Two test fires of stage 2 ran the same evening — 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42 — both completed, both with delivery suppressed, last_status: ok, streak 0.

Fleet size went 28 → 29 jobs (26 enabled / 3 disabled).

Why it matters

  1. The failure mode is partial work, not a lost run. Four hours of browser-heavy discovery produced a correct report and zero writes. Nothing in jobs.json distinguished it from a good day except last_status: error on 10-27 (completed on 10-26, with the same incompleteness buried in the text).
  2. Splitting is the mitigation, not a fix. Stage 2 inherits its own tool budget; if stage 1 still overruns and never writes the handoff file, stage 2 has nothing to land — watch the 11:30 slot for that.
  3. The budget is consumed by discovery, which is exactly the stage that now owns it alone.

Open questions

  1. Verify one full cycle end-to-end (stage 1 07:00 → stage 2 11:30): does stage 2 find stage 1's handoff file, do the Notion writes and cover letters land, and is MISSING_COVER_LETTER_COUNT=0 reported?
  2. Make budget exhaustion self-describing. The classifier tagged it auth; a distinct budget/partial-run type would put this on the same footing as any other failure.
  3. Do the two stages fit the same wall-clock envelope? Stage 1 must finish inside its own budget and leave the handoff before 11:30.
  4. The incident row stays open until a clean stage-1 slot auto-closes it — confirm the 2026-10-28 07:00 slot closes dca848_e93a538ade6b.

Follow-up — the 2026-10-28 slot (recorded 2026-10-29)

The first real two-stage cycle ran on 2026-10-28 and answered the open questions above — not favourably:

  • Stage 1 overran again, on a new signature. dca8482e4f76 ran 07:00:32 → 10:33:18 (3 h 32 m 46 s), closed failed, failure_streak 2, on "hit the agent iteration cap before the handoff could be written" → incidents/2026-10-28-job-radar-stage1-iteration-cap.
  • No handoff file was written — ~/.hermes/radar-ref/handoff/ held only 2026-10-27.json at the 2026-10-29 audit, so Notion writes and cover letters did not run again.
  • Both stage-2 slots no-oped correctly — 5fc292ff0f3b fired 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43, both completed with delivery_outcome: suppressed. Open question 1 is therefore answered only in the negative: the recovery rule ("a stage-2 run with no handoff is a no-op") holds in production, but nothing has yet proved the positive path end-to-end.
  • Open question 4 is answered: the row did not close. dca848_e93a538ade6b stayed open because the 10-28 slot was not clean, and a second row (dca848_0a2f03b5952b) opened — open incident rows 2 → 4.

Follow-up — the 2026-10-29 slot: the split held (recorded 2026-10-30)

Resolved. Stage 1 (dca8482e4f76) ran 2026-10-29 07:00:59 → 10:16:12 — 3 h 15 m 13 s — and closed completed. It wrote ~/.hermes/radar-ref/handoff/2026-10-29.json (24.8 KB): run_date 2026-10-29T08:23:51+08:00, 170 discovered / 33 duplicates avoided / 37 verified, sources seek ok, linkedin ok, indeed blocked, wa_gov ok. Both ledger rows this incident opened — dca848_e93a538ade6b and dca848_0a2f03b5952b — closed resolved at 10:16:12, and the run opened no new row. Open cron incident rows went 4 → 1; the survivor is the Facebook monthly's 0e753e_e7f8194a7d06, which cannot clear before its 2026-11-01 slot.

Stage 2 (5fc292ff0f3b) fired at 10:13:46, consumed the handoff at 10:40:59, finished 10:44:48 completed and delivered to topic 1030 in the same second (delivery delivered). Its stage2_result: 20 pages written / 20 page_ids / 2 cover letters / 2 same-day strong matches / verified_cover_letter_count 2 / missing_cover_letter_count 0; the Notion board went 455 → 475. The later slots no-oped as designed (11:30 → 10 m 46 s, 14:30 → 1 m 51 s, already-consumed handoff). Stage 1 itself no longer delivers — correct, because stage 2 owns the report.

The questions this page left open are answered for this run: the handoff was written inside budget and consumed, and no cover letter went missing on the first real run of the split.