audit(2026-11-10): 11-09 first failed-free cron day since 11-05; stocktake 'completed' with zero edits; Job Radar Stage 1 stall (9h09m, no handoff) documented; tailscale 58 nodes; omniroute 55 combos
This commit is contained in:
@@ -91,6 +91,13 @@ the error string does not name the hook.
|
||||
fault; if it recurs while the hook budget is 120 s, this hypothesis is dead.
|
||||
- Decide whether a skipped `pre_tool_call` should **fail loudly** rather than silently degrade tool dispatch.
|
||||
|
||||
## Follow-up (2026-11-10 nightly audit)
|
||||
|
||||
- **The fix held; the cascade did not recur.** `errors.log` (1,035 retained lines) contains **no `pre_tool_call` timeout entry at all**. Its only two hook lines are benign skips — `on_session_end` 2026-11-09 22:58:35 and `pre_llm_call` 2026-11-10 00:02:54 (*"skipped after previous timeout or while still running"*).
|
||||
- **No 11-09 run showed the cascade signature.** Outside the two long-runners documented separately (stocktake **9 h 10 m**, Job Radar Stage 1 **9 h 09 m** — both stalled in *tool iteration*, not in a hook), the longest 11-09 execution was **74 min**, then 40 min and 11 min.
|
||||
- **2026-11-09 was the first `failed`-free cron day since 2026-11-05** (11-06: 1 failed, 11-07: 3, 11-08: 1, 11-09: **0 of 174**), and unclosed incident rows went 3 → 0.
|
||||
- **Residual:** the two long-runners above are the same 9-hour-turn environment this bug leaves behind, and one (`wiki-stocktake-maintenance`) is pinned to `free-stack` — a combo recorded broken on 2026-08-14. The job-iteration budget stays in force.
|
||||
|
||||
## See also
|
||||
|
||||
- [[systems/conduit]]
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
title: "2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op"
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-10
|
||||
updated: 2026-11-10
|
||||
verified_on: 2026-11-10
|
||||
confidence: high
|
||||
tags: [cron, job-radar, stall, silent-failure, handoff, read-loop]
|
||||
sources: ["executions.db (dca8482e4f76: 2026-11-09T07:00:48+08:00 → 2026-11-09T16:10:12+08:00, status completed, delivery_outcome suppressed; 5fc292ff0f3b: 11:30:38 → 12:10:07 and 14:30:34 → 14:37:27, both completed/suppressed)", "~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md (ends INCOMPLETE: tool budget exhausted; last words \\u201cI stopped retrying because I kept running read_file 5 times without making progress.\\u201d)", "~/.hermes/radar-ref/handoff/ (newest handoff on disk: 2026-11-06; no 2026-11-07/08/09 files)", "cron_incidents (no row raised for this run)"]
|
||||
---
|
||||
|
||||
# 2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op
|
||||
|
||||
## Summary
|
||||
|
||||
On 2026-11-09 the Job Radar's Stage 1 (`dca8482e4f76`) ran for **9 h 09 m 24 s** (07:00:48 → 16:10:12), ended `INCOMPLETE: tool budget exhausted`, and **never wrote its handoff file**. Because both Stage-2 slots run on their own schedule, **both fired before Stage 1 finished** and each declined silently rather than erroring. Net effect: **the radar landed nothing on 11-09 and raised no alert.**
|
||||
|
||||
## Timeline (AWST, 2026-11-09)
|
||||
|
||||
| Time | Event |
|
||||
|---|---|
|
||||
| 07:00:48 | Stage 1 starts |
|
||||
| 11:30:38 → 12:10:07 | Stage-2 slot 1 fires — Stage 1 still running → silent no-op (`suppressed`) |
|
||||
| 14:30:34 → 14:37:27 | Stage-2 slot 2 fires — Stage 1 still running → silent no-op (`suppressed`) |
|
||||
| ~16:10:1x | Stage 1 writes `INCOMPLETE: tool budget exhausted` |
|
||||
| 16:10:12 | Stage 1 closes — `completed`, delivery `suppressed` |
|
||||
| all day | `radar-ref/handoff/2026-11-09.json` is never created |
|
||||
|
||||
## Root Cause
|
||||
|
||||
Two compounding faults, both previously documented for this job:
|
||||
|
||||
1. **Stage 2 never checks that a fresh handoff exists**, so when Stage 1 fails it proceeds and self-suppresses instead of failing loudly. Two chances to notice, zero alerting.
|
||||
2. **Stage 1 overran without producing output.** Its own final message: *"I stopped retrying because I kept running read_file 5 times without making progress."* This is a **new trigger** for the same silent family seen on 11-02/11-03 (there: iteration cap) — here: an unproductive read loop that consumed the run's budget.
|
||||
|
||||
## Evidence
|
||||
|
||||
- `executions.db` `dca8482e4f76`: `2026-11-09T07:00:48+08:00` → `2026-11-09T16:10:12+08:00`, `status completed`, `delivery_outcome suppressed`.
|
||||
- `5fc292ff0f3b` (Stage 2): two rows on 11-09, both **before** Stage 1's finish stamp, both `completed` / `suppressed`.
|
||||
- `~/.hermes/radar-ref/handoff/` — newest file is **2026-11-06**; nothing for 11-07, 11-08 or 11-09.
|
||||
- `~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md` ends `INCOMPLETE: tool budget exhausted`.
|
||||
- **No row in `cron_incidents`** for `dca8482e4f76` on 11-09 — the failure was never raised.
|
||||
|
||||
## Impact
|
||||
|
||||
- The Job Radar — the estate's job-discovery feed — produced **no output for a third day** (no handoff since 11-06).
|
||||
- The failure is invisible: the run's ledger entry says `completed`, delivery is `suppressed`, and no incident row exists.
|
||||
|
||||
## Fix Applied
|
||||
|
||||
**None — documentation only.** No configuration change was made this window.
|
||||
|
||||
## Prevention
|
||||
|
||||
- **Alert on the artefact, not the exit code.** Stage 2 should hard-fail when the expected `radar-ref/handoff/<date>.json` is missing, rather than proceeding to a suppression. That single check converts both 11-09 slots into loud failures.
|
||||
- Add a **read-progress guard** to Stage 1: when the same file has been read N times without a state change, stop and emit a partial handoff instead of burning the budget to `INCOMPLETE`.
|
||||
- Note the pattern across incidents: **completion-time-only alerting hides every failure in this family** (stocktake 11-09, radar 11-09, radar 11-02/03).
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]
|
||||
- [[runbooks/provider-health]]
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
title: "2026-11-09 — Stocktake Completes With Zero Edits"
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-10
|
||||
updated: 2026-11-10
|
||||
verified_on: 2026-11-10
|
||||
confidence: high
|
||||
tags: [cron, wiki, stocktake, tool-budget, silent-failure, omni-101, free-stack]
|
||||
sources: ["executions.db (1a8e90ea2271: 2026-11-09T03:00:43+08:00 → 2026-11-09T12:10:47+08:00, status completed, delivery_outcome suppressed)", "~/.hermes/cron/output/1a8e90ea2271/2026-11-09_12-10-46.md (8,305 bytes; RUN INCOMPLETE, \\u201ctool-iteration budget exhausted during verification\\u201d)", "cron_incidents (1a8e90_628d914f1f97, 1a8e90_2f370ef764f5, 1a8e90_a9cad99cb150 — all closed 2026-11-09T12:10:47.125205+08:00)", "wiki-stocktake-maintenance job 1a8e90ea2271 (provider custom:omni-101, model free-stack)"]
|
||||
---
|
||||
|
||||
# 2026-11-09 — Stocktake Completes With Zero Edits
|
||||
|
||||
## Summary
|
||||
|
||||
The nightly wiki stocktake (`wiki-stocktake-maintenance`, `1a8e90ea2271`) ran on 2026-11-09 from **03:00:43 to 12:10:47** — **9 h 10 m** — and the harness recorded it **`completed`** with delivery `suppressed`. Its own output file records the opposite: the run ran out of tool-iteration budget inside its verification phase, applied **zero edits**, made **no commit**, and its findings never landed.
|
||||
|
||||
**This is the third consecutive day the stocktake's work did not land** — 11-07 died at 2 h 02 m, 11-08 at 1 h 35 m (both `failed`), 11-09 at 9 h 10 m (`completed`). The visible status this time is *worse*, not better: `failure_streak` fell **3 → 0** and `last_status` shows **ok**, while the work stayed undone.
|
||||
|
||||
## Timeline (AWST, 2026-11-09)
|
||||
|
||||
| Time | Event |
|
||||
|---|---|
|
||||
| 03:00:43 | Run starts (`1a8e90ea2271`) |
|
||||
| ~04:1x | Tool-iteration budget exhausted during verification; output file written with `RUN INCOMPLETE` |
|
||||
| 12:10:47.125205 | Harness closes the run — `completed`, delivery `suppressed` |
|
||||
| 12:10:47.125205 | The **three** long-open stocktake incident rows close at the same instant: `1a8e90_628d914f1f97` (11-07T04:29:54), `1a8e90_2f370ef764f5` (11-07T05:02:21), `1a8e90_a9cad99cb150` (11-08T04:35:38) |
|
||||
|
||||
## Root Cause
|
||||
|
||||
Same family as 11-07/11-08: **the run spends its entire tool budget before reaching the write phase.** The run's own output file records the stop as the tool-iteration budget being exhausted **during verification**, with zero edits applied — the edits it had identified were never written.
|
||||
|
||||
Secondary cause (confirmed again this window): the job is pinned to `custom:omni-101` / **`free-stack`** — a combo recorded broken on **2026-08-14** (see [[systems/scheduled-tasks]]).
|
||||
|
||||
## Evidence
|
||||
|
||||
- `executions.db`: `started_at 2026-11-09T03:00:43+08:00` → `finished_at 2026-11-09T12:10:47+08:00`, `status completed`, `delivery_outcome suppressed`. Previous two runs: 11-08 `failed` (03:01:11 → 04:36:08, 1 h 35 m), 11-07 `failed` (03:00:31 → 05:02:22, 2 h 02 m).
|
||||
- Output file `/home/hermes/.hermes/cron/output/1a8e90ea2271/2026-11-09_12-10-46.md` (8,305 bytes) ends `RUN INCOMPLETE`, with the tool-iteration budget exhausted inside verification.
|
||||
- The three stocktake incident rows all carry `closed_at = 2026-11-09T12:10:47.125205+08:00` — i.e. they closed on **completion**, not on a fix. All three have `alerted_at = null`: they never alerted.
|
||||
- Git: **no stocktake commit on 11-09** — zero edits were applied, so there was nothing to commit and nothing to revert. (On 11-07/11-08 the page was left dirty and then reverted; on 11-09 no edit was ever made.)
|
||||
|
||||
## Impact
|
||||
|
||||
- Nine consecutive stalled stocktake runs (11-01 … 11-09) — the nightly wiki freshness pass is producing no durable output.
|
||||
- The stocktake's *top-priority* brief item on 11-09 was the **monotonically worsening failure streak** itself. The job closed its own incident while failing that item, which is the failure mode to watch: **`completed` and `ok` now overstate the stocktake's health.**
|
||||
- Three incident rows were silently cleared, so the fleet's incident board now reads **0 open** while the underlying problem is unfixed.
|
||||
|
||||
## Fix Applied
|
||||
|
||||
**None — this is a detection-and-verification record.** The page documents the failure and the misleading status so the next audit starts from truth.
|
||||
|
||||
Also recorded this window: the `free-stack` pin on `1a8e90ea2271` — the only deterministic configuration fault in the chain — is **still in place**, unchanged from the 11-08 record.
|
||||
|
||||
## Prevention
|
||||
|
||||
- **Alerting must fire on write progress, not completion.** This run produced 2 of 3 failure days silently and the third day produced a run that reported success while editing nothing. A check on `commit_count` / `edits_applied` for this job would have caught all three.
|
||||
- Don't read `status`/`failure_streak` as the stocktake's health. Read whether a commit landed.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[infrastructure/tailscale]]
|
||||
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
- [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]
|
||||
Reference in New Issue
Block a user