audit: 2026-10-28 nightly — radar budget overrun now a recorded failure (new incident page); radar split into two cron jobs; tailscale peer table re-verified; TTS provider corrected to the Mac mini speech server
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76 (Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and cover letters never ran (Notion DB held 451 entries, 0 added). Incident row dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified). Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5, deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5, topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3). Also: the two incident rows the 2026-10-26 planned restart raised auto-closed 00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of 108) — committed here and index.md corrected; tailscale.md peer table rebuilt from tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained the voice pipeline section. New: incidents/2026-10-27-job-radar-budget-exhausted.md Nothing removed or archived. No secrets.
This commit is contained in:
@@ -3,7 +3,7 @@ title: Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemos
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-10-27
|
||||
updated: 2026-10-27
|
||||
updated: 2026-10-28
|
||||
verified_on: 2026-10-27
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
@@ -101,3 +101,13 @@ disappears silently. What it did not do is *finish the work* or *avoid the inter
|
||||
- [[systems/scheduled-tasks]] — fleet ledger and per-window tallies
|
||||
- [[systems/mnemosyne]] — the interrupted job's purpose and settings
|
||||
- [[systems/hermes-agent]] — gateway lifecycle and restart history
|
||||
|
||||
## Update — 2026-10-28 (nightly audit)
|
||||
|
||||
- **No recurrence.** Every run in the 2026-10-27 window except the radar's budget overrun completed, and the
|
||||
watchdog logged no alert for the day.
|
||||
- **Both incident rows this fault raised closed automatically** — `ea79a2_a4661554a3c2` and
|
||||
`ea79a2_c3ab2d784ba3` **auto-closed 2026-10-27T00:38:32+08:00** (unclosed rows fleet-wide fell 3 → 2).
|
||||
- The `lost its durable fire claim ownership` string did not appear in any 10-27 execution row.
|
||||
- Open question from the original write-up still stands: why a *planned* stop bypasses the drain deadline
|
||||
(`timed_out=True` at 1.35 s of a 30 s budget).
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-10-28
|
||||
updated: 2026-10-28
|
||||
verified_on: 2026-10-28
|
||||
review_after: 2026-11-04
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api]
|
||||
---
|
||||
|
||||
# Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response
|
||||
|
||||
## Headline
|
||||
|
||||
On **2026-10-27** the Daily Marketing Job Radar (`dca8482e4f76`) ran **07:01:08 → 11:03:54 AWST**
|
||||
(**4 h 2 m 46 s** wall clock), closed **`failed`**, and delivered a report that opens:
|
||||
|
||||
> **Daily Marketing Job Radar — run 2026-10-27 — INCOMPLETE (tool-call budget exhausted before Notion write stage)**
|
||||
|
||||
Discovery, verification, dedup and classification completed. **Notion ingestion, cover-letter generation and the
|
||||
final cover-letter audit were not executed** — so the run is recorded as failed rather than successful.
|
||||
Self-reported run time **10:47:57**; freshness cutoff **2026-10-20**.
|
||||
|
||||
This is the second consecutive budget overrun: the **2026-10-26** 07:00 slot ran 07:01:08 → 09:10:27
|
||||
(**2 h 9 m 19 s**), closed **`completed`**, and *still* delivered the same `RUN INCOMPLETE` marker — the day's
|
||||
backlog had to be finished by hand. 2026-10-27 is the first overrun recorded as **`failed`**, which is the
|
||||
improvement: the failure reached the incident ledger instead of hiding inside a green row.
|
||||
|
||||
## Ledger state
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Execution row | `dca8482e4f76`, started `2026-10-27T07:01:08.265913+08:00`, finished `2026-10-27T11:03:54.143535+08:00`, `failed` |
|
||||
| Delivery | `delivered` (the incomplete report reached Telegram topic 1030) |
|
||||
| Incident row | `dca848_e93a538ade6b` — `state: alerted`, `failure_type: auth`, first/last seen `2026-10-27T11:03:53.715494+08:00`, alerted `11:03:59.013115` |
|
||||
| Incident status | **open at the 2026-10-28 00:2x audit** — no ack, no close |
|
||||
| `jobs.json` | `last_status: error`, `failure_streak: 1`, pin unchanged at `custom:omni-101` / `smart-route` |
|
||||
|
||||
⚠ `failure_type: auth` does **not** describe this fault. A tool-call budget exhaustion is being classified as an
|
||||
auth failure; the field is not trustworthy as a diagnosis for this job.
|
||||
|
||||
## What the run actually did (from its own report)
|
||||
|
||||
- Live Notion DB `0f90ba2b-8b10-4d02-a62c-6f692d5b1168`: **451 entries**, `Date Added = 2026-10-27` → **0**.
|
||||
- Board composition at run time: **Agent — Rhino 402, Chaos 12, Manual 2, blank 35**; **Source — SEEK 295,
|
||||
LinkedIn 88** (row text truncated in the ledger).
|
||||
|
||||
## Response, same day: the radar is now two jobs
|
||||
|
||||
At **2026-10-27 17:34:15** the single radar job was split into a two-stage pipeline (see
|
||||
[[runbooks/job-radar]] for the full contract):
|
||||
|
||||
| Stage | Job | Schedule | Delivery | Job id |
|
||||
|---|---|---|---|---|
|
||||
| 1 — Discovery | Daily Marketing Job Radar - Discovery — `DISCOVERY → VERIFY → DEDUP → SCORE`, no Notion write, no cover letters; writes a handoff file | `0 7 * * 1-5` | `local` (failures → Telegram topic 1030) | `dca8482e4f76` (unchanged) |
|
||||
| 2 — Write and Letters | Daily Marketing Job Radar - Write and Letters — lands the handoff: Notion entries, cover letters, audit, report | `30 11,14 * * 1-5` | Telegram group `-1004321904721`, topic `1030` | `5fc292ff0f3b` (created 17:34:15) |
|
||||
|
||||
Stage 1 keeps its id, pin (`custom:omni-101` / `smart-route`) and its own tool budget; stage 2 is a separate job
|
||||
whose prompt lands the handoff rather than repeating the discovery narrative. Two test fires of stage 2 ran the same
|
||||
evening — **17:35:00 → 17:36:52** and **17:38:38 → 17:43:42** — both `completed`, both with delivery
|
||||
`suppressed`, `last_status: ok`, streak 0.
|
||||
|
||||
Fleet size went **28 → 29 jobs** (26 enabled / 3 disabled).
|
||||
|
||||
## Why it matters
|
||||
|
||||
1. **The failure mode is partial work, not a lost run.** Four hours of browser-heavy discovery produced a
|
||||
correct report and zero writes. Nothing in `jobs.json` distinguished it from a good day except
|
||||
`last_status: error` on 10-27 (`completed` on 10-26, with the same incompleteness buried in the text).
|
||||
2. **Splitting is the mitigation, not a fix.** Stage 2 inherits its own tool budget; if stage 1 still overruns
|
||||
and never writes the handoff file, stage 2 has nothing to land — watch the 11:30 slot for that.
|
||||
3. **The budget is consumed by discovery**, which is exactly the stage that now owns it alone.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Verify one full cycle end-to-end** (stage 1 07:00 → stage 2 11:30): does stage 2 find stage 1's handoff
|
||||
file, do the Notion writes and cover letters land, and is `MISSING_COVER_LETTER_COUNT=0` reported?
|
||||
2. **Make budget exhaustion self-describing.** The classifier tagged it `auth`; a distinct budget/partial-run
|
||||
type would put this on the same footing as any other failure.
|
||||
3. **Do the two stages fit the same wall-clock envelope?** Stage 1 must finish inside its own budget *and* leave
|
||||
the handoff before 11:30.
|
||||
4. The incident row stays open until a clean stage-1 slot auto-closes it — confirm the 2026-10-28 07:00 slot
|
||||
closes `dca848_e93a538ade6b`.
|
||||
|
||||
## Related
|
||||
|
||||
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
|
||||
- [[systems/scheduled-tasks]] — fleet ledger, job table and the 2026-10-27 window
|
||||
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — the radar's previous failure mode
|
||||
Reference in New Issue
Block a user