Files
agent-estate-wiki/incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
Hermes Wiki Audit 729bb597cf audit(2026-11-11): the audit's own 429 killed the 11-10 window; stocktake + radar recover with output
Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed).

NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
 - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed
 - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6)
   reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded
 - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row
 - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2

RESOLVED with output (dated Recovery sections added):
 - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0
 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B)
 - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved

UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md
(opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x),
incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
2026-11-12 00:16:39 +08:00

5.8 KiB

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op incident resolved 2026-11-10 2026-11-11 2026-11-11 high
cron
job-radar
stall
silent-failure
handoff
read-loop
executions.db (dca8482e4f76: 2026-11-09T07:00:48+08:00 → 2026-11-09T16:10:12+08:00, status completed, delivery_outcome suppressed; 5fc292ff0f3b: 11:30:38 → 12:10:07 and 14:30:34 → 14:37:27, both completed/suppressed)
~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md (ends INCOMPLETE: tool budget exhausted; last words \u201cI stopped retrying because I kept running read_file 5 times without making progress.\u201d)
~/.hermes/radar-ref/handoff/ (newest handoff on disk: 2026-11-06; no 2026-11-07/08/09 files)
cron_incidents (no row raised for this run)

2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op

Summary

On 2026-11-09 the Job Radar's Stage 1 (dca8482e4f76) ran for 9 h 09 m 24 s (07:00:48 → 16:10:12), ended INCOMPLETE: tool budget exhausted, and never wrote its handoff file. Because both Stage-2 slots run on their own schedule, both fired before Stage 1 finished and each declined silently rather than erroring. Net effect: the radar landed nothing on 11-09 and raised no alert.

Timeline (AWST, 2026-11-09)

Time Event
07:00:48 Stage 1 starts
11:30:38 → 12:10:07 Stage-2 slot 1 fires — Stage 1 still running → silent no-op (suppressed)
14:30:34 → 14:37:27 Stage-2 slot 2 fires — Stage 1 still running → silent no-op (suppressed)
~16:10:1x Stage 1 writes INCOMPLETE: tool budget exhausted
16:10:12 Stage 1 closes — completed, delivery suppressed
all day radar-ref/handoff/2026-11-09.json is never created

Root Cause

Two compounding faults, both previously documented for this job:

  1. Stage 2 never checks that a fresh handoff exists, so when Stage 1 fails it proceeds and self-suppresses instead of failing loudly. Two chances to notice, zero alerting.
  2. Stage 1 overran without producing output. Its own final message: "I stopped retrying because I kept running read_file 5 times without making progress." This is a new trigger for the same silent family seen on 11-02/11-03 (there: iteration cap) — here: an unproductive read loop that consumed the run's budget.

Evidence

  • executions.db dca8482e4f76: 2026-11-09T07:00:48+08:00 → 2026-11-09T16:10:12+08:00, status completed, delivery_outcome suppressed.
  • 5fc292ff0f3b (Stage 2): two rows on 11-09, both before Stage 1's finish stamp, both completed / suppressed.
  • ~/.hermes/radar-ref/handoff/ — newest file is 2026-11-06; nothing for 11-07, 11-08 or 11-09.
  • ~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md ends INCOMPLETE: tool budget exhausted.
  • No row in cron_incidents for dca8482e4f76 on 11-09 — the failure was never raised.

Impact

  • The Job Radar — the estate's job-discovery feed — produced no output for a third day (no handoff since 11-06).
  • The failure is invisible: the run's ledger entry says completed, delivery is suppressed, and no incident row exists.

Fix Applied

None — documentation only. No configuration change was made this window.

Prevention

  • Alert on the artefact, not the exit code. Stage 2 should hard-fail when the expected radar-ref/handoff/<date>.json is missing, rather than proceeding to a suppression. That single check converts both 11-09 slots into loud failures.
  • Add a read-progress guard to Stage 1: when the same file has been read N times without a state change, stop and emit a partial handoff instead of burning the budget to INCOMPLETE.
  • Note the pattern across incidents: completion-time-only alerting hides every failure in this family (stocktake 11-09, radar 11-09, radar 11-02/03).

Recovery — 2026-11-11: the handoff landed, and Stage 2 delivered

Field 11-09 (this incident) 11-11
Stage 1 07:00:48 → 16:10:12 (9 h 09 m 24 s), no handoff 07:01:08 → 07:37:34 = 36 m 26 s, completed
Handoff on disk newest was 2026-11-06 radar-ref/handoff/2026-11-11.json — 22,682 B, mtime 07:59
Stage 2 (11:30 slot) fired early, declined silently 11:30:56 → 11:35:18, completed
Stage 2 (14:30 slot) declined silently 14:30:55 → 14:34:56, completed / delivered
Incident row none raised 5fc292_773822455856 closed resolved 2026-11-11T08:03:47.644226
  • The first handoff in five days (2026-11-06 → 2026-11-11) and the fastest Stage-1 cycle since the two-stage split: 36 m 26 s against 3 h 15 m (10-29, the last full success), 9 h 01 m (11-02), 6 h 28 m (11-03) and 9 h 09 m (11-09).
  • Stage 2's recovery fire (source: direct, 07:35:56 → 08:03:47) consumed the handoff and delivered at 08:03:47; its completion is what auto-closed the row opened by 11-10's interrupted slot.
  • ⚠ Carried forward from 11-10: the scheduled Stage-2 slot that day ran 11:30:35 → 22:18:49 = 10 h 48 m 14 s and closed failed / interrupted (delivery delivered, for_failure: 1) — the longest run of either window, and itself a new data point for the "completion-time-only alerting" family on this page.
  • Job state: both radar jobs read last_status: ok, failure_streak 0.

Status: resolved. Both prevention rules on this page still stand and remain unimplemented: Stage 2 should hard-fail on a missing handoff/<date>.json rather than suppressing, and Stage 1 needs a read-progress guard. Neither was needed on 11-11 because the run finished inside budget.