Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76 (Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and cover letters never ran (Notion DB held 451 entries, 0 added). Incident row dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified). Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5, deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5, topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3). Also: the two incident rows the 2026-10-26 planned restart raised auto-closed 00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of 108) — committed here and index.md corrected; tailscale.md peer table rebuilt from tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained the voice pipeline section. New: incidents/2026-10-27-job-radar-budget-exhausted.md Nothing removed or archived. No secrets.
5.5 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response | incident | active | 2026-10-28 | 2026-10-28 | 2026-10-28 | 2026-11-04 | high |
|
|
Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response
Headline
On 2026-10-27 the Daily Marketing Job Radar (dca8482e4f76) ran 07:01:08 → 11:03:54 AWST
(4 h 2 m 46 s wall clock), closed failed, and delivered a report that opens:
Daily Marketing Job Radar — run 2026-10-27 — INCOMPLETE (tool-call budget exhausted before Notion write stage)
Discovery, verification, dedup and classification completed. Notion ingestion, cover-letter generation and the final cover-letter audit were not executed — so the run is recorded as failed rather than successful. Self-reported run time 10:47:57; freshness cutoff 2026-10-20.
This is the second consecutive budget overrun: the 2026-10-26 07:00 slot ran 07:01:08 → 09:10:27
(2 h 9 m 19 s), closed completed, and still delivered the same RUN INCOMPLETE marker — the day's
backlog had to be finished by hand. 2026-10-27 is the first overrun recorded as failed, which is the
improvement: the failure reached the incident ledger instead of hiding inside a green row.
Ledger state
| Field | Value |
|---|---|
| Execution row | dca8482e4f76, started 2026-10-27T07:01:08.265913+08:00, finished 2026-10-27T11:03:54.143535+08:00, failed |
| Delivery | delivered (the incomplete report reached Telegram topic 1030) |
| Incident row | dca848_e93a538ade6b — state: alerted, failure_type: auth, first/last seen 2026-10-27T11:03:53.715494+08:00, alerted 11:03:59.013115 |
| Incident status | open at the 2026-10-28 00:2x audit — no ack, no close |
jobs.json |
last_status: error, failure_streak: 1, pin unchanged at custom:omni-101 / smart-route |
⚠ failure_type: auth does not describe this fault. A tool-call budget exhaustion is being classified as an
auth failure; the field is not trustworthy as a diagnosis for this job.
What the run actually did (from its own report)
- Live Notion DB
0f90ba2b-8b10-4d02-a62c-6f692d5b1168: 451 entries,Date Added = 2026-10-27→ 0. - Board composition at run time: Agent — Rhino 402, Chaos 12, Manual 2, blank 35; Source — SEEK 295, LinkedIn 88 (row text truncated in the ledger).
Response, same day: the radar is now two jobs
At 2026-10-27 17:34:15 the single radar job was split into a two-stage pipeline (see runbooks/job-radar for the full contract):
| Stage | Job | Schedule | Delivery | Job id |
|---|---|---|---|---|
| 1 — Discovery | Daily Marketing Job Radar - Discovery — DISCOVERY → VERIFY → DEDUP → SCORE, no Notion write, no cover letters; writes a handoff file |
0 7 * * 1-5 |
local (failures → Telegram topic 1030) |
dca8482e4f76 (unchanged) |
| 2 — Write and Letters | Daily Marketing Job Radar - Write and Letters — lands the handoff: Notion entries, cover letters, audit, report | 30 11,14 * * 1-5 |
Telegram group -1004321904721, topic 1030 |
5fc292ff0f3b (created 17:34:15) |
Stage 1 keeps its id, pin (custom:omni-101 / smart-route) and its own tool budget; stage 2 is a separate job
whose prompt lands the handoff rather than repeating the discovery narrative. Two test fires of stage 2 ran the same
evening — 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42 — both completed, both with delivery
suppressed, last_status: ok, streak 0.
Fleet size went 28 → 29 jobs (26 enabled / 3 disabled).
Why it matters
- The failure mode is partial work, not a lost run. Four hours of browser-heavy discovery produced a
correct report and zero writes. Nothing in
jobs.jsondistinguished it from a good day exceptlast_status: erroron 10-27 (completedon 10-26, with the same incompleteness buried in the text). - Splitting is the mitigation, not a fix. Stage 2 inherits its own tool budget; if stage 1 still overruns and never writes the handoff file, stage 2 has nothing to land — watch the 11:30 slot for that.
- The budget is consumed by discovery, which is exactly the stage that now owns it alone.
Open questions
- Verify one full cycle end-to-end (stage 1 07:00 → stage 2 11:30): does stage 2 find stage 1's handoff
file, do the Notion writes and cover letters land, and is
MISSING_COVER_LETTER_COUNT=0reported? - Make budget exhaustion self-describing. The classifier tagged it
auth; a distinct budget/partial-run type would put this on the same footing as any other failure. - Do the two stages fit the same wall-clock envelope? Stage 1 must finish inside its own budget and leave the handoff before 11:30.
- The incident row stays open until a clean stage-1 slot auto-closes it — confirm the 2026-10-28 07:00 slot
closes
dca848_e93a538ade6b.
Related
- runbooks/job-radar — the two-stage contract, recovery procedure and the 2026-10-26 overrun
- systems/scheduled-tasks — fleet ledger, job table and the 2026-10-27 window
- incidents/2026-10-21-omniroute-pre-dispatch-filter-skip — the radar's previous failure mode