wiki(audit): 2026-10-28 window — radar stage-1 iteration cap (no handoff, stage-2 no-oped), stocktake 400 never alerted, Steel removed from the stack

This commit is contained in:
hermes
2026-10-29 00:41:30 +08:00
parent 2cbf0b65c1
commit ec0c30fb30
12 changed files with 374 additions and 27 deletions
+22 -4
View File
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery
type: runbook
status: active
created: 2026-07-27
updated: 2026-10-28
updated: 2026-10-29
review_after: 2026-11-10
verified_on: 2026-10-28
verified_on: 2026-10-29
version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2)
confidence: high
tags: [automation, cron, jobs, notion, dashboard, runbook]
@@ -326,13 +326,31 @@ now owns a job of its own and the writes own a second one.
- Stage 1 keeps the `custom:omni-101` / `smart-route` pin. Both jobs still inherit the profile-level
`fallback_providers` chain; Hermes stores no per-job fallback list.
- First fires: two test runs of stage 2 on 2026-10-27 (17:35:00 → 17:36:52 and 17:38:38 → 17:43:42), both
`completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle had not yet run at the
2026-10-28 audit.**
`completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle
ran on 2026-10-28 and landed nothing** — stage 1 never wrote its handoff (see the 2026-10-28 slot below).
**Recovery if stage 2 lands nothing:** check that stage 1 actually wrote its handoff file before running stage 2
by hand (`hermes cron run 5fc292ff0f3b`). A stage-2 run with no handoff is a no-op, not a failed search — do not
re-run stage 1's discovery just to satisfy it.
### 2026-10-28 07:00 slot — stage 1 hit the iteration cap, so stage 2 had nothing to land
Stage 1 (`dca8482e4f76`) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**) and closed **`failed`** on
*"hit the agent iteration cap before the handoff could be written"* — its **second** consecutive failure
(`failure_streak: 2`). No handoff file was written: `~/.hermes/radar-ref/handoff/` held only `2026-10-27.json`
at the 2026-10-29 audit.
Both stage-2 slots then behaved exactly as designed — **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both
`completed`, both `delivery_outcome: suppressed`. **A stage-2 run with no handoff is a silent no-op, not a
failure**, which is the recovery rule below holding in production for the first time.
Read with 10-27, the picture is: *the split stops the writes from being lost, but it does not make discovery
finish.* Two runs, two different ceilings — 10-27 stopped at the **tool-call budget** after 4 h 2 m (writes not
started), 10-28 stopped at the **agent iteration cap** after 3 h 32 m, earlier still (no handoff at all). The
discovery stage is the thing to bound, not the wall clock.
Full ledger: [[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
## Browser-layer hazard inside the run (2026-10-12)
This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove