audit: 2026-10-28 nightly — radar budget overrun now a recorded failure (new incident page); radar split into two cron jobs; tailscale peer table re-verified; TTS provider corrected to the Mac mini speech server
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76 (Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and cover letters never ran (Notion DB held 451 entries, 0 added). Incident row dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified). Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5, deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5, topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3). Also: the two incident rows the 2026-10-26 planned restart raised auto-closed 00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of 108) — committed here and index.md corrected; tailscale.md peer table rebuilt from tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained the voice pipeline section. New: incidents/2026-10-27-job-radar-budget-exhausted.md Nothing removed or archived. No secrets.
This commit is contained in:
+34
-3
@@ -3,8 +3,8 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-27
|
||||
verified_on: 2026-10-27
|
||||
updated: 2026-10-28
|
||||
verified_on: 2026-10-28
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
tags: [system]
|
||||
@@ -12,7 +12,38 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-27
|
||||
Updated: 2026-10-28
|
||||
|
||||
## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST)
|
||||
|
||||
- **The radar's silent budget overrun became a recorded failure — and the radar is now two jobs.** Window
|
||||
2026-10-27 settled **172 `completed` / 1 `failed`**. The failure is `dca8482e4f76` (Daily Marketing Job
|
||||
Radar - Discovery): **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed **`failed`**, delivery `delivered`,
|
||||
and the delivered report opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* —
|
||||
discovery, verification, dedup and classification completed; **Notion ingestion, cover-letter generation and
|
||||
the final cover-letter audit did not execute**. The prior day (10-26) had the same overrun but closed
|
||||
`completed`, hiding it in a green row. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted`
|
||||
11:03:59, **still open** at this audit (`failure_type: auth` — misclassified). → new page
|
||||
[[incidents/2026-10-27-job-radar-budget-exhausted]]
|
||||
- **Response: the radar was split into a two-stage pipeline the same day.** Stage 1 `dca8482e4f76` — Discovery
|
||||
(`0 7 * * 1-5`, `deliver: local`, failures → topic 1030) writes a handoff and triggers stage 2; new stage 2
|
||||
`5fc292ff0f3b` — Daily Marketing Job Radar - Write and Letters (`30 11,14 * * 1-5`, Telegram topic 1030,
|
||||
created **2026-10-27T17:34:15**) lands the Notion entries, cover letters and audit. Two test fires of stage 2
|
||||
on 10-27 (17:35:00 → 17:36:52, 17:38:38 → 17:43:42) both `completed` with delivery `suppressed`; the first
|
||||
real 07:00 → 11:30 cycle had not yet run. **Fleet 28 → 29 jobs (26 / 3).** → [[runbooks/job-radar]]
|
||||
- **The 2026-10-26 orphan class did not recur, and its two incident rows closed.** `ea79a2_a4661554a3c2` and
|
||||
`ea79a2_c3ab2d784ba3` **auto-closed 00:38:32** (unclosed rows **3 → 2**, the 2 being the Facebook birthday
|
||||
monthly scrape from 10-01 and the new radar row). No 10-27 run besides the radar's failed, so the watchdog
|
||||
and every other job were clean all day.
|
||||
- **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error`
|
||||
(streak 1) and `08eb86f02d37` re-joined as `delivery_queued`. Still non-clean: `0e753e9720fd` (error),
|
||||
`fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.**
|
||||
- **Also updated:** `incidents/2026-10-27-job-radar-budget-exhausted.md` (new),
|
||||
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md` (recurrence check), `runbooks/job-radar.md`,
|
||||
`systems/scheduled-tasks.md`, `systems/hermes-agent.md` (TTS provider correction),
|
||||
`systems/conduit.md` (voice pipeline), `infrastructure/tailscale.md` (peer table re-verified),
|
||||
`systems/skills-index.md` (regenerated 10-27, committed), `index.md`, `log.md`.
|
||||
**Nothing removed or archived.** No secrets written.
|
||||
|
||||
## Recent Changes (2026-10-27, nightly audit; live probes 2026-10-27 00:1x–00:5x AWST)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user