4.0 KiB
4.0 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op | incident | active | 2026-11-10 | 2026-11-10 | 2026-11-10 | high |
|
|
2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op
Summary
On 2026-11-09 the Job Radar's Stage 1 (dca8482e4f76) ran for 9 h 09 m 24 s (07:00:48 → 16:10:12), ended INCOMPLETE: tool budget exhausted, and never wrote its handoff file. Because both Stage-2 slots run on their own schedule, both fired before Stage 1 finished and each declined silently rather than erroring. Net effect: the radar landed nothing on 11-09 and raised no alert.
Timeline (AWST, 2026-11-09)
| Time | Event |
|---|---|
| 07:00:48 | Stage 1 starts |
| 11:30:38 → 12:10:07 | Stage-2 slot 1 fires — Stage 1 still running → silent no-op (suppressed) |
| 14:30:34 → 14:37:27 | Stage-2 slot 2 fires — Stage 1 still running → silent no-op (suppressed) |
| ~16:10:1x | Stage 1 writes INCOMPLETE: tool budget exhausted |
| 16:10:12 | Stage 1 closes — completed, delivery suppressed |
| all day | radar-ref/handoff/2026-11-09.json is never created |
Root Cause
Two compounding faults, both previously documented for this job:
- Stage 2 never checks that a fresh handoff exists, so when Stage 1 fails it proceeds and self-suppresses instead of failing loudly. Two chances to notice, zero alerting.
- Stage 1 overran without producing output. Its own final message: "I stopped retrying because I kept running read_file 5 times without making progress." This is a new trigger for the same silent family seen on 11-02/11-03 (there: iteration cap) — here: an unproductive read loop that consumed the run's budget.
Evidence
executions.dbdca8482e4f76:2026-11-09T07:00:48+08:00→2026-11-09T16:10:12+08:00,status completed,delivery_outcome suppressed.5fc292ff0f3b(Stage 2): two rows on 11-09, both before Stage 1's finish stamp, bothcompleted/suppressed.~/.hermes/radar-ref/handoff/— newest file is 2026-11-06; nothing for 11-07, 11-08 or 11-09.~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.mdendsINCOMPLETE: tool budget exhausted.- No row in
cron_incidentsfordca8482e4f76on 11-09 — the failure was never raised.
Impact
- The Job Radar — the estate's job-discovery feed — produced no output for a third day (no handoff since 11-06).
- The failure is invisible: the run's ledger entry says
completed, delivery issuppressed, and no incident row exists.
Fix Applied
None — documentation only. No configuration change was made this window.
Prevention
- Alert on the artefact, not the exit code. Stage 2 should hard-fail when the expected
radar-ref/handoff/<date>.jsonis missing, rather than proceeding to a suppression. That single check converts both 11-09 slots into loud failures. - Add a read-progress guard to Stage 1: when the same file has been read N times without a state change, stop and emit a partial handoff instead of burning the budget to
INCOMPLETE. - Note the pattern across incidents: completion-time-only alerting hides every failure in this family (stocktake 11-09, radar 11-09, radar 11-02/03).