8.6 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded `completed`, Output Suppressed | incident | active | 2026-11-03 | 2026-11-04 | 2026-11-04 | 2026-11-17 | high |
|
|
Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded completed, Output Suppressed
Headline
Third occurrence of the Stage-1 budget class, and the first that the cron ledger reports as healthy.
On 2026-11-02 the 07:00 dca8482e4f76 (Daily Marketing Job Radar — Stage 1 (Discovery)) ran for
9 h 01 m 19 s (started 07:00:33.378239 → finished 16:01:52.496335 +08:00), consumed the harness's
entire tool-calling iteration budget, and closed with:
| Field | Value |
|---|---|
status |
completed |
error |
NULL |
delivery_outcome |
suppressed |
No handoff artefact was written. Nothing reached Notion, no cover letters were generated, and Stage 2 was
never triggered. Because the run's own status is completed with no error, a health check that inspects
executions.status alone reports this day as green — the failure is invisible in the ledger.
What the run did and did not do
The session (Daily Marketing Job Radar - Discovery · Nov 02 16:01, local-368c8fe9…) shows the discovery
half genuinely worked:
- Camofox was restarted at run start after its HTTP server wedged (see systems/browser-backend).
- SEEK 51 unique roles, LinkedIn 60 unique job IDs, Indeed live, WA Government 35 results — 197 raw candidates, 164 new after three-level dedup against the 479-row Notion board.
- A URL-canonicalisation bug was found and fixed mid-run:
au.seek.comvsau.seek.com.auhad been dropping SEEK and Indeed entries from the consolidated list; the fix was a job-ID-based canonicalisation. - Adverts were individually verified fresh and active; salaries, closing dates and JD bodies were pulled.
Then the ceiling hit. The runtime injected "You've reached the maximum number of tool-calling iterations allowed. Please provide a final response…" while the agent was still reading JD text for the borderline finalists, and the run terminated in the scoring phase.
The agent's own closing report was explicit — and it was not delivered:
"Status: INCOMPLETE — handoff file NOT written, Stage 2 NOT triggered. … Because the mandatory deliverable (the handoff JSON) was not produced, today's roles have not been handed to Stage 2."
That text is delivery_outcome: suppressed, so Anthony saw nothing on 11-02 — not the roles and not the
warning that the roles were missing.
Downstream: both Stage 2 slots behaved correctly
| Slot | Session | Ran | Result |
|---|---|---|---|
| 11:30 | local-92665337… (Write and Letters) |
11:30 → 12:25 | Exited [SILENT] after verifying Stage 1 was alive but had no handoff |
| 14:30 (safety net) | local-05a3f00b… (Write and Letters) |
14:30 → 14:30 | Exited [SILENT] — 6 messages, 2 tool calls |
Both declines were correct: no handoff existed to consume. The safety-net design worked as intended — the gap is that Stage 1 cannot finish inside its budget.
Recurrence pattern
| Date | Incident page | Failure shape |
|---|---|---|
| 2026-10-27 | incidents/2026-10-27-job-radar-budget-exhausted | Tool-call budget exhausted before the Notion write stage |
| 2026-10-28 | incidents/2026-10-28-job-radar-stage1-iteration-cap | Stage 1 hit the agent iteration cap before the handoff |
| 2026-11-02 | this page | Same cap; run recorded completed and its report suppressed |
| 2026-11-03 | incidents/2026-11-03-job-radar-stream-active-timeout | A different ceiling — RuntimeError: stream active timeout after 6 h 28 m; failed, notice delivered |
Three hits in seven days, each ending in the same place: discovery complete, handoff absent, Notion write stage never reached. A fourth run — 2026-11-03 — ended the same way (no handoff, both Stage 2 slots declined) but on a different fault, and this time the ledger did go red; see the 2026-11-04 update below.
The "green row, no artefact" anti-pattern
This is the fourth appearance of the class systems/scheduled-tasks §9 of the maintenance playbook already documents (a job whose durable ledger entry says success while the thing it was supposed to produce does not exist). The new variant here is that the ledger flipped from red to green:
| Date | cron_incidents row for dca8482e4f76 |
executions.status |
|---|---|---|
| 2026-10-27 | dca848_e93a538ade6b (failure_type: auth) |
non-clean |
| 2026-10-28 | dca848_0a2f03b5952b (failure_type: timeout) |
non-clean |
| 2026-11-02 | none raised | completed, error: NULL |
The two earlier caps each raised an incident whose error text is literally the run's INCOMPLETE report.
The 2026-11-02 cap raised nothing at all (cron_incidents holds no row for dca8482e4f76 with a
first_seen_at on 2026-11-02; fleet-wide unclosed rows were 0 at the 2026-11-03 audit). The run's
delivery_outcome is suppressed, so no message reached Anthony either.
Detection rule: for any job whose deliverable is a file (a handoff JSON, a report, a generated page),
executions.status is not sufficient evidence. Verify the artefact exists at its expected path, or that
cron_incidents stayed quiet for a reason, before treating the slot as served. Silence from
cron_incidents is not the same as success — here it is the symptom.
2026-11-04 update — the missing incident row was a property of the cap, not of long runs
Job Radar Stage 1 lost a second consecutive day on 2026-11-03 — and this time the harness killed the
turn, so the ledger went red. The 07:00 dca8482e4f76 ran 07:00:33.819190 → 13:28:52.305995
(6 h 28 m 18 s) and closed status: failed / error: RuntimeError: stream active timeout /
delivery_outcome: delivered, raising incident dca848_dfd4bea84f2a (failure_type: timeout,
alerted 13:28:55.141618, still open) — fleet unclosed rows 0 → 1.
→ incidents/2026-11-03-job-radar-stream-active-timeout
This answers open question 3 below (partly): suppression tracked the terminal state, not the run length.
When the run ends completed — as the cap produces — its report is suppressed; when the harness itself
kills the turn, the terminal state is failed and both the incident row and the notice appear. 11-02:
nothing delivered. 11-03: the failure notice went out at 13:28:51.741724 and Pulse surfaced it to
Anthony in the desktop session 20261031_190813_95fbde (report p34c8e899-1, covering Tue 03 Nov 13:01
onward): "The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a
stream active timeout."
It also narrows open question 2 below: the ledger can record a non-completed state when a long run
dies — the iteration cap is the specific case that still escapes as completed.
And it partially answers open question 4: the candidate set was lost again. Both Stage 2 slots declined
silently (5fc292ff0f3b 11:30:36 → 11:32:16 and 14:30:32 → 14:33:11) and
~/.hermes/radar-ref/handoff/ still holds only 2026-10-27.json, 2026-10-29.json and
2026-10-30.json — so 2026-11-02's 164 deduped candidates and 2026-11-03's discovery are both
unrecovered, two days running.
Open questions
- Is the iteration budget the right control, or is the task too large for one agent turn? Three caps in a row suggest Stage 1's work (four job boards → dedup → individual advert verification → JD reading → scoring) does not fit a single turn at any budget.
- Should the harness record a cap-hit as something other than
completed? A distinct terminal status (or a non-nullerror) would make this visible to every existing health check. - Should the run's final response be delivered even when the job "succeeded"? Suppressing an INCOMPLETE report converts a loud failure into silence.
- Did the 11-02 roles get picked up on the next day's run, or is 2026-11-02's candidate set lost? The 164 deduped candidates were never persisted.