6.5 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped | incident | resolved | 2026-10-29 | 2026-10-30 | 2026-10-30 | 2026-11-13 | high |
|
|
Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped
Headline
On 2026-10-28 the Discovery stage of the Daily Marketing Job Radar (dca8482e4f76) ran
07:00:32 → 10:33:18 AWST (3 h 32 m 46 s) and closed failed on "hit the agent iteration cap before
the handoff could be written" — its second consecutive failure (failure_streak: 2).
This is the first real cycle of the two-stage split introduced on 2026-10-27 (see
incidents/2026-10-27-job-radar-budget-exhausted and runbooks/job-radar), and it produced a clean negative
result: stage 1 never wrote its handoff file, so there was nothing for stage 2 to land. Both stage-2 slots
ran anyway — 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43, both completed, both
delivery_outcome: suppressed — which is the designed no-op path working correctly in production for the first
time. Notion ingestion and cover letters did not execute for the day.
Ledger state
| Field | Value |
|---|---|
| Execution row | dca8482e4f76, started 2026-10-28T07:00:32.463309+08:00, finished 2026-10-28T10:33:18.833788+08:00, failed |
| Duration | 3 h 32 m 46 s (10-27: 4 h 2 m 46 s; 10-26: 2 h 9 m 19 s) |
jobs.json |
last_status: error, failure_streak: 2; last_error opens "Daily Marketing Job Radar — Stage 1 (Discovery) — 2026-10-28 — hit the agent iteration cap before the handoff could be…"; pin unchanged (custom:omni-101 / smart-route) |
| Incident row | dca848_0a2f03b5952b — state: alerted, failure_type: timeout, first_seen 2026-10-28T10:33:18, open |
| Handoff | not written — ~/.hermes/radar-ref/handoff/ holds only 2026-10-27.json (verified 2026-10-29 00:3x) |
| Stage 2 | 5fc292ff0f3b fired 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43, both completed / delivery_outcome: suppressed |
The 10-27 row (dca848_e93a538ade6b, classified failure_type: auth) did not close — the 2026-10-28 07:00
slot was not clean. Open cron_incidents rows went 2 → 4.
What the split proved, and what it did not
- It proved the recovery rule. "A stage-2 run with no handoff is a no-op, not a failed search" — exactly as written in runbooks/job-radar — held in production: stage 2 invented no work, raised no error, and sent nothing to topic 1030.
- It did not make discovery finish. The two post-split runs died at different ceilings:
- 2026-10-27 — stopped at the tool-call budget after 4 h 2 m: discovery/verify/dedup/classify complete, writes never started.
- 2026-10-28 — stopped at the agent iteration cap after 3 h 32 m, earlier in the pipeline: no handoff, so nothing downstream could run at all. The wall-clock number improving (4 h 2 m → 3 h 32 m) is not an improvement — less of the pipeline completed.
- The write stage is no longer the risk; the discovery stage still is. Splitting removed the "writes lost inside a green row" failure mode. What remains is that stage 1 cannot both finish discovery and reach its handoff write inside its own limits.
Why it matters
- A lost day is now cheap to detect but easy to miss. Stage 1 raises an incident row and flips
last_statustoerror; stage 2's correct silence (suppressed) means the only outward sign of a workless day is stage 1's own failure report — not a "0 jobs added" report. - Two consecutive failures, two different classifiers. 10-27 was tagged
failure_type: auth(wrong for a budget fault); 10-28 was taggedtimeout(also only approximate — an agent-side iteration cap, not a network timeout). Neither string names the actual cause. - Budget arithmetic is not the fix. If the cap fires after ~3.5 h of discovery, the pipeline needs a smaller discovery surface (fewer sources per slot), staged/checkpointed handoff writes (so a capped run still leaves something landable), or an explicit per-phase iteration allowance.
Open questions
- What is the iteration cap for this job, and can stage 1 reach its handoff write before it? The 10-28 run died before the write; the 10-27 run died after discovery but before writes. Neither produced a handoff.
- Should stage 1 checkpoint a partial handoff (after DEDUP/SCORE) so a capped run gives stage 2 something to land instead of a silent no-op day?
- Make the abort self-describing — a distinct
iteration-cap/partial-runfailure type rather thanauth/timeout. - Both radar incident rows (
dca848_e93a538ade6b,dca848_0a2f03b5952b) close only when a clean stage-1 slot runs — watch the next 07:00 weekday slot.
Follow-up — the 2026-10-29 slot: the cap was not hit (recorded 2026-10-30)
Resolved for this run. The next stage-1 run (dca8482e4f76, 2026-10-29 07:00:59 → 10:16:12,
3 h 15 m 13 s) closed completed and wrote ~/.hermes/radar-ref/handoff/2026-10-29.json — the
first run of the split to do both. The handoff records 170 discovered / 33 duplicates avoided /
37 verified, run_date 2026-10-29T08:23:51+08:00, sources seek ok, linkedin ok,
indeed blocked, wa_gov ok. Stage 2 consumed it at 10:40:59, delivered at 10:44:48 with
20 Notion pages and 2 cover letters (missing_cover_letter_count 0; board 455 → 475). Its two
ledger rows closed resolved at 10:16:12.
⚠ Flaky, not solved. A 3 h 15 m stage-1 run is still long enough for the cap to be reachable, and
the previous two slots in the split both died on it (2026-10-27 on the tool-call budget, 2026-10-28 on
the iteration cap). Treat a completed run with no handoff file as the same fault, whatever the
execution status says → incidents/2026-10-27-job-radar-budget-exhausted.
Related
- runbooks/job-radar — the two-stage contract, recovery rule and the 2026-10-28 slot write-up
- incidents/2026-10-27-job-radar-budget-exhausted — the prior day's budget exhaustion and the split
- systems/scheduled-tasks — fleet ledger and the 2026-10-28 window
- systems/browser-backend — the browser layer this run depends on