Files
agent-estate-wiki/incidents/2026-10-28-job-radar-stage1-iteration-cap.md

6.5 KiB

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped incident resolved 2026-10-29 2026-10-30 2026-10-30 2026-11-13 high
automation
cron
jobs
notion
silent-failure
scheduling
live-sqlite-cron-executions
live-sqlite-cron-incidents
live-jobs-json
live-radar-handoff-dir

Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped

Headline

On 2026-10-28 the Discovery stage of the Daily Marketing Job Radar (dca8482e4f76) ran 07:00:32 → 10:33:18 AWST (3 h 32 m 46 s) and closed failed on "hit the agent iteration cap before the handoff could be written" — its second consecutive failure (failure_streak: 2).

This is the first real cycle of the two-stage split introduced on 2026-10-27 (see incidents/2026-10-27-job-radar-budget-exhausted and runbooks/job-radar), and it produced a clean negative result: stage 1 never wrote its handoff file, so there was nothing for stage 2 to land. Both stage-2 slots ran anyway — 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43, both completed, both delivery_outcome: suppressed — which is the designed no-op path working correctly in production for the first time. Notion ingestion and cover letters did not execute for the day.

Ledger state

Field Value
Execution row dca8482e4f76, started 2026-10-28T07:00:32.463309+08:00, finished 2026-10-28T10:33:18.833788+08:00, failed
Duration 3 h 32 m 46 s (10-27: 4 h 2 m 46 s; 10-26: 2 h 9 m 19 s)
jobs.json last_status: error, failure_streak: 2; last_error opens "Daily Marketing Job Radar — Stage 1 (Discovery) — 2026-10-28 — hit the agent iteration cap before the handoff could be…"; pin unchanged (custom:omni-101 / smart-route)
Incident row dca848_0a2f03b5952b — state: alerted, failure_type: timeout, first_seen 2026-10-28T10:33:18, open
Handoff not written — ~/.hermes/radar-ref/handoff/ holds only 2026-10-27.json (verified 2026-10-29 00:3x)
Stage 2 5fc292ff0f3b fired 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43, both completed / delivery_outcome: suppressed

The 10-27 row (dca848_e93a538ade6b, classified failure_type: auth) did not close — the 2026-10-28 07:00 slot was not clean. Open cron_incidents rows went 2 → 4.

What the split proved, and what it did not

  1. It proved the recovery rule. "A stage-2 run with no handoff is a no-op, not a failed search" — exactly as written in runbooks/job-radar — held in production: stage 2 invented no work, raised no error, and sent nothing to topic 1030.
  2. It did not make discovery finish. The two post-split runs died at different ceilings:
    • 2026-10-27 — stopped at the tool-call budget after 4 h 2 m: discovery/verify/dedup/classify complete, writes never started.
    • 2026-10-28 — stopped at the agent iteration cap after 3 h 32 m, earlier in the pipeline: no handoff, so nothing downstream could run at all. The wall-clock number improving (4 h 2 m → 3 h 32 m) is not an improvement — less of the pipeline completed.
  3. The write stage is no longer the risk; the discovery stage still is. Splitting removed the "writes lost inside a green row" failure mode. What remains is that stage 1 cannot both finish discovery and reach its handoff write inside its own limits.

Why it matters

  • A lost day is now cheap to detect but easy to miss. Stage 1 raises an incident row and flips last_status to error; stage 2's correct silence (suppressed) means the only outward sign of a workless day is stage 1's own failure report — not a "0 jobs added" report.
  • Two consecutive failures, two different classifiers. 10-27 was tagged failure_type: auth (wrong for a budget fault); 10-28 was tagged timeout (also only approximate — an agent-side iteration cap, not a network timeout). Neither string names the actual cause.
  • Budget arithmetic is not the fix. If the cap fires after ~3.5 h of discovery, the pipeline needs a smaller discovery surface (fewer sources per slot), staged/checkpointed handoff writes (so a capped run still leaves something landable), or an explicit per-phase iteration allowance.

Open questions

  1. What is the iteration cap for this job, and can stage 1 reach its handoff write before it? The 10-28 run died before the write; the 10-27 run died after discovery but before writes. Neither produced a handoff.
  2. Should stage 1 checkpoint a partial handoff (after DEDUP/SCORE) so a capped run gives stage 2 something to land instead of a silent no-op day?
  3. Make the abort self-describing — a distinct iteration-cap / partial-run failure type rather than auth / timeout.
  4. Both radar incident rows (dca848_e93a538ade6b, dca848_0a2f03b5952b) close only when a clean stage-1 slot runs — watch the next 07:00 weekday slot.

Follow-up — the 2026-10-29 slot: the cap was not hit (recorded 2026-10-30)

Resolved for this run. The next stage-1 run (dca8482e4f76, 2026-10-29 07:00:59 → 10:16:12, 3 h 15 m 13 s) closed completed and wrote ~/.hermes/radar-ref/handoff/2026-10-29.json — the first run of the split to do both. The handoff records 170 discovered / 33 duplicates avoided / 37 verified, run_date 2026-10-29T08:23:51+08:00, sources seek ok, linkedin ok, indeed blocked, wa_gov ok. Stage 2 consumed it at 10:40:59, delivered at 10:44:48 with 20 Notion pages and 2 cover letters (missing_cover_letter_count 0; board 455 → 475). Its two ledger rows closed resolved at 10:16:12.

⚠ Flaky, not solved. A 3 h 15 m stage-1 run is still long enough for the cap to be reachable, and the previous two slots in the split both died on it (2026-10-27 on the tool-call budget, 2026-10-28 on the iteration cap). Treat a completed run with no handoff file as the same fault, whatever the execution status says → incidents/2026-10-27-job-radar-budget-exhausted.