wiki(audit): 2026-10-28 window — radar stage-1 iteration cap (no handoff, stage-2 no-oped), stocktake 400 never alerted, Steel removed from the stack
This commit is contained in:
@@ -3,8 +3,8 @@ title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-10-28
|
||||
updated: 2026-10-28
|
||||
verified_on: 2026-10-28
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
review_after: 2026-11-04
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
||||
@@ -85,6 +85,22 @@ Fleet size went **28 → 29 jobs** (26 enabled / 3 disabled).
|
||||
4. The incident row stays open until a clean stage-1 slot auto-closes it — confirm the 2026-10-28 07:00 slot
|
||||
closes `dca848_e93a538ade6b`.
|
||||
|
||||
## Follow-up — the 2026-10-28 slot (recorded 2026-10-29)
|
||||
|
||||
The **first real two-stage cycle ran on 2026-10-28** and answered the open questions above — not favourably:
|
||||
|
||||
- **Stage 1 overran again, on a new signature.** `dca8482e4f76` ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**),
|
||||
closed **`failed`**, `failure_streak` **2**, on *"hit the agent iteration cap before the handoff could be
|
||||
written"* → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
|
||||
- **No handoff file was written** — `~/.hermes/radar-ref/handoff/` held only `2026-10-27.json` at the 2026-10-29
|
||||
audit, so **Notion writes and cover letters did not run again**.
|
||||
- **Both stage-2 slots no-oped correctly** — `5fc292ff0f3b` fired **11:30:27 → 11:31:31** and
|
||||
**14:30:27 → 14:31:43**, both `completed` with `delivery_outcome: suppressed`. Open question 1 is therefore
|
||||
answered only in the negative: the recovery rule ("a stage-2 run with no handoff is a no-op") holds in
|
||||
production, but nothing has yet proved the positive path end-to-end.
|
||||
- **Open question 4 is answered: the row did not close.** `dca848_e93a538ade6b` stayed open because the 10-28
|
||||
slot was not clean, and a second row (`dca848_0a2f03b5952b`) opened — open incident rows **2 → 4**.
|
||||
|
||||
## Related
|
||||
|
||||
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-10-29
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
review_after: 2026-11-05
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir]
|
||||
---
|
||||
|
||||
# Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped
|
||||
|
||||
## Headline
|
||||
|
||||
On **2026-10-28** the Discovery stage of the Daily Marketing Job Radar (`dca8482e4f76`) ran
|
||||
**07:00:32 → 10:33:18 AWST** (**3 h 32 m 46 s**) and closed **`failed`** on *"hit the agent iteration cap before
|
||||
the handoff could be written"* — its **second consecutive failure** (`failure_streak: 2`).
|
||||
|
||||
This is the first real cycle of the **two-stage split** introduced on 2026-10-27 (see
|
||||
[[incidents/2026-10-27-job-radar-budget-exhausted]] and [[runbooks/job-radar]]), and it produced a clean negative
|
||||
result: **stage 1 never wrote its handoff file, so there was nothing for stage 2 to land.** Both stage-2 slots
|
||||
ran anyway — **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed`, both
|
||||
`delivery_outcome: suppressed` — which is the designed no-op path working correctly in production for the first
|
||||
time. Notion ingestion and cover letters did not execute for the day.
|
||||
|
||||
## Ledger state
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Execution row | `dca8482e4f76`, started `2026-10-28T07:00:32.463309+08:00`, finished `2026-10-28T10:33:18.833788+08:00`, **`failed`** |
|
||||
| Duration | **3 h 32 m 46 s** (10-27: 4 h 2 m 46 s; 10-26: 2 h 9 m 19 s) |
|
||||
| `jobs.json` | `last_status: error`, **`failure_streak: 2`**; `last_error` opens *"Daily Marketing Job Radar — Stage 1 (Discovery) — 2026-10-28 — hit the agent iteration cap before the handoff could be…"*; pin unchanged (`custom:omni-101` / `smart-route`) |
|
||||
| Incident row | `dca848_0a2f03b5952b` — `state: alerted`, `failure_type: timeout`, first_seen `2026-10-28T10:33:18`, **open** |
|
||||
| Handoff | **not written** — `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json` (verified 2026-10-29 00:3x) |
|
||||
| Stage 2 | `5fc292ff0f3b` fired `11:30:27 → 11:31:31` and `14:30:27 → 14:31:43`, both `completed` / `delivery_outcome: suppressed` |
|
||||
|
||||
The 10-27 row (`dca848_e93a538ade6b`, classified `failure_type: auth`) did **not** close — the 2026-10-28 07:00
|
||||
slot was not clean. Open `cron_incidents` rows went **2 → 4**.
|
||||
|
||||
## What the split proved, and what it did not
|
||||
|
||||
1. **It proved the recovery rule.** *"A stage-2 run with no handoff is a no-op, not a failed search"* — exactly as
|
||||
written in [[runbooks/job-radar]] — held in production: stage 2 invented no work, raised no error, and sent
|
||||
nothing to topic 1030.
|
||||
2. **It did not make discovery finish.** The two post-split runs died at *different* ceilings:
|
||||
- **2026-10-27** — stopped at the **tool-call budget** after 4 h 2 m: discovery/verify/dedup/classify complete, writes never started.
|
||||
- **2026-10-28** — stopped at the **agent iteration cap** after 3 h 32 m, *earlier in the pipeline*: no handoff, so nothing downstream could run at all.
|
||||
The wall-clock number improving (4 h 2 m → 3 h 32 m) is **not** an improvement — less of the pipeline completed.
|
||||
3. **The write stage is no longer the risk; the discovery stage still is.** Splitting removed the
|
||||
"writes lost inside a green row" failure mode. What remains is that stage 1 cannot both finish discovery and
|
||||
reach its handoff write inside its own limits.
|
||||
|
||||
## Why it matters
|
||||
|
||||
- **A lost day is now cheap to detect but easy to miss.** Stage 1 raises an incident row and flips `last_status`
|
||||
to `error`; stage 2's *correct* silence (`suppressed`) means the only outward sign of a workless day is stage
|
||||
1's own failure report — not a "0 jobs added" report.
|
||||
- **Two consecutive failures, two different classifiers.** 10-27 was tagged `failure_type: auth` (wrong for a
|
||||
budget fault); 10-28 was tagged `timeout` (also only approximate — an agent-side iteration cap, not a network
|
||||
timeout). Neither string names the actual cause.
|
||||
- **Budget arithmetic is not the fix.** If the cap fires after ~3.5 h of discovery, the pipeline needs a smaller
|
||||
discovery surface (fewer sources per slot), **staged/checkpointed handoff writes** (so a capped run still
|
||||
leaves something landable), or an explicit per-phase iteration allowance.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **What is the iteration cap for this job, and can stage 1 reach its handoff write before it?** The 10-28 run
|
||||
died *before* the write; the 10-27 run died *after* discovery but *before* writes. Neither produced a handoff.
|
||||
2. **Should stage 1 checkpoint a partial handoff** (after DEDUP/SCORE) so a capped run gives stage 2 something to
|
||||
land instead of a silent no-op day?
|
||||
3. **Make the abort self-describing** — a distinct `iteration-cap` / `partial-run` failure type rather than
|
||||
`auth` / `timeout`.
|
||||
4. Both radar incident rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) close only when a **clean** stage-1
|
||||
slot runs — watch the next 07:00 weekday slot.
|
||||
|
||||
## Related
|
||||
|
||||
- [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up
|
||||
- [[incidents/2026-10-27-job-radar-budget-exhausted]] — the prior day's budget exhaustion and the split
|
||||
- [[systems/scheduled-tasks]] — fleet ledger and the 2026-10-28 window
|
||||
- [[systems/browser-backend]] — the browser layer this run depends on
|
||||
Reference in New Issue
Block a user