Files
agent-estate-wiki/incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
T

4.0 KiB

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op incident active 2026-11-10 2026-11-10 2026-11-10 high
cron
job-radar
stall
silent-failure
handoff
read-loop
executions.db (dca8482e4f76: 2026-11-09T07:00:48+08:00 → 2026-11-09T16:10:12+08:00, status completed, delivery_outcome suppressed; 5fc292ff0f3b: 11:30:38 → 12:10:07 and 14:30:34 → 14:37:27, both completed/suppressed)
~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md (ends INCOMPLETE: tool budget exhausted; last words \u201cI stopped retrying because I kept running read_file 5 times without making progress.\u201d)
~/.hermes/radar-ref/handoff/ (newest handoff on disk: 2026-11-06; no 2026-11-07/08/09 files)
cron_incidents (no row raised for this run)

2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op

Summary

On 2026-11-09 the Job Radar's Stage 1 (dca8482e4f76) ran for 9 h 09 m 24 s (07:00:48 → 16:10:12), ended INCOMPLETE: tool budget exhausted, and never wrote its handoff file. Because both Stage-2 slots run on their own schedule, both fired before Stage 1 finished and each declined silently rather than erroring. Net effect: the radar landed nothing on 11-09 and raised no alert.

Timeline (AWST, 2026-11-09)

Time Event
07:00:48 Stage 1 starts
11:30:38 → 12:10:07 Stage-2 slot 1 fires — Stage 1 still running → silent no-op (suppressed)
14:30:34 → 14:37:27 Stage-2 slot 2 fires — Stage 1 still running → silent no-op (suppressed)
~16:10:1x Stage 1 writes INCOMPLETE: tool budget exhausted
16:10:12 Stage 1 closes — completed, delivery suppressed
all day radar-ref/handoff/2026-11-09.json is never created

Root Cause

Two compounding faults, both previously documented for this job:

  1. Stage 2 never checks that a fresh handoff exists, so when Stage 1 fails it proceeds and self-suppresses instead of failing loudly. Two chances to notice, zero alerting.
  2. Stage 1 overran without producing output. Its own final message: "I stopped retrying because I kept running read_file 5 times without making progress." This is a new trigger for the same silent family seen on 11-02/11-03 (there: iteration cap) — here: an unproductive read loop that consumed the run's budget.

Evidence

  • executions.db dca8482e4f76: 2026-11-09T07:00:48+08:00 → 2026-11-09T16:10:12+08:00, status completed, delivery_outcome suppressed.
  • 5fc292ff0f3b (Stage 2): two rows on 11-09, both before Stage 1's finish stamp, both completed / suppressed.
  • ~/.hermes/radar-ref/handoff/ — newest file is 2026-11-06; nothing for 11-07, 11-08 or 11-09.
  • ~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md ends INCOMPLETE: tool budget exhausted.
  • No row in cron_incidents for dca8482e4f76 on 11-09 — the failure was never raised.

Impact

  • The Job Radar — the estate's job-discovery feed — produced no output for a third day (no handoff since 11-06).
  • The failure is invisible: the run's ledger entry says completed, delivery is suppressed, and no incident row exists.

Fix Applied

None — documentation only. No configuration change was made this window.

Prevention

  • Alert on the artefact, not the exit code. Stage 2 should hard-fail when the expected radar-ref/handoff/<date>.json is missing, rather than proceeding to a suppression. That single check converts both 11-09 slots into loud failures.
  • Add a read-progress guard to Stage 1: when the same file has been read N times without a state change, stop and emit a partial handoff instead of burning the budget to INCOMPLETE.
  • Note the pattern across incidents: completion-time-only alerting hides every failure in this family (stocktake 11-09, radar 11-09, radar 11-02/03).