wiki(audit): 2026-10-28 window — radar stage-1 iteration cap (no handoff, stage-2 no-oped), stocktake 400 never alerted, Steel removed from the stack
This commit is contained in:
+22
-4
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-27
|
||||
updated: 2026-10-28
|
||||
updated: 2026-10-29
|
||||
review_after: 2026-11-10
|
||||
verified_on: 2026-10-28
|
||||
verified_on: 2026-10-29
|
||||
version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2)
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, dashboard, runbook]
|
||||
@@ -326,13 +326,31 @@ now owns a job of its own and the writes own a second one.
|
||||
- Stage 1 keeps the `custom:omni-101` / `smart-route` pin. Both jobs still inherit the profile-level
|
||||
`fallback_providers` chain; Hermes stores no per-job fallback list.
|
||||
- First fires: two test runs of stage 2 on 2026-10-27 (17:35:00 → 17:36:52 and 17:38:38 → 17:43:42), both
|
||||
`completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle had not yet run at the
|
||||
2026-10-28 audit.**
|
||||
`completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle
|
||||
ran on 2026-10-28 and landed nothing** — stage 1 never wrote its handoff (see the 2026-10-28 slot below).
|
||||
|
||||
**Recovery if stage 2 lands nothing:** check that stage 1 actually wrote its handoff file before running stage 2
|
||||
by hand (`hermes cron run 5fc292ff0f3b`). A stage-2 run with no handoff is a no-op, not a failed search — do not
|
||||
re-run stage 1's discovery just to satisfy it.
|
||||
|
||||
### 2026-10-28 07:00 slot — stage 1 hit the iteration cap, so stage 2 had nothing to land
|
||||
|
||||
Stage 1 (`dca8482e4f76`) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**) and closed **`failed`** on
|
||||
*"hit the agent iteration cap before the handoff could be written"* — its **second** consecutive failure
|
||||
(`failure_streak: 2`). No handoff file was written: `~/.hermes/radar-ref/handoff/` held only `2026-10-27.json`
|
||||
at the 2026-10-29 audit.
|
||||
|
||||
Both stage-2 slots then behaved exactly as designed — **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both
|
||||
`completed`, both `delivery_outcome: suppressed`. **A stage-2 run with no handoff is a silent no-op, not a
|
||||
failure**, which is the recovery rule below holding in production for the first time.
|
||||
|
||||
Read with 10-27, the picture is: *the split stops the writes from being lost, but it does not make discovery
|
||||
finish.* Two runs, two different ceilings — 10-27 stopped at the **tool-call budget** after 4 h 2 m (writes not
|
||||
started), 10-28 stopped at the **agent iteration cap** after 3 h 32 m, earlier still (no handoff at all). The
|
||||
discovery stage is the thing to bound, not the wall clock.
|
||||
|
||||
Full ledger: [[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
|
||||
|
||||
## Browser-layer hazard inside the run (2026-10-12)
|
||||
|
||||
This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove
|
||||
|
||||
Reference in New Issue
Block a user