audit: 2026-10-28 nightly — radar budget overrun now a recorded failure (new incident page); radar split into two cron jobs; tailscale peer table re-verified; TTS provider corrected to the Mac mini speech server
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76 (Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and cover letters never ran (Notion DB held 451 entries, 0 added). Incident row dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified). Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5, deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5, topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3). Also: the two incident rows the 2026-10-26 planned restart raised auto-closed 00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of 108) — committed here and index.md corrected; tailscale.md peer table rebuilt from tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained the voice pipeline section. New: incidents/2026-10-27-job-radar-budget-exhausted.md Nothing removed or archived. No secrets.
This commit is contained in:
@@ -1454,3 +1454,55 @@ not pushed.
|
||||
whether `failure_streak` should count the *unverified* row (one interruption currently reads streak 2); and the
|
||||
**`Thread 34795`** delivery target needs re-pointing, or the Monday 03:08 digest will keep falling back to the
|
||||
unthreaded chat.
|
||||
|
||||
## [2026-10-28] audit | Nightly wiki audit — window 2026-10-26 00:00 → 2026-10-27 00:5x AWST: a planned restart orphaned a live cron run
|
||||
|
||||
- **Window 2026-10-27: 172 `completed` / 1 `failed` (+ this audit `running`).** The failure is the radar's
|
||||
budget overrun, and it is the story of the night → new page
|
||||
[[incidents/2026-10-27-job-radar-budget-exhausted]].
|
||||
- **The radar's tool-call budget was exhausted for the second consecutive day — and this time the run is
|
||||
recorded as `failed`.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed` and
|
||||
delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery, verification,
|
||||
dedup and classification completed; **Notion ingestion, cover-letter generation and the final cover-letter
|
||||
audit never ran**. Notion DB `0f90ba2b-8b10-4d02-a62c-6f692d5b1168` held **451 entries** with **0** added
|
||||
for 2026-10-27 (Agent — Rhino 402 / Chaos 12 / Manual 2 / blank 35; Source — SEEK 295 / LinkedIn 88).
|
||||
Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** at this audit
|
||||
(`failure_type: auth` — which does not describe a budget fault). The previous day's identical overrun closed
|
||||
**`completed`** and was found only by reading the delivered text — that is why the 10-27 row matters.
|
||||
- **Response, same day: the radar is now two jobs.** Stage 1 `dca8482e4f76` **Daily Marketing Job Radar -
|
||||
Discovery** (`0 7 * * 1-5`, `deliver: local`, failures → Telegram topic 1030) does
|
||||
`DISCOVERY → VERIFY → DEDUP → SCORE`, writes a handoff file and triggers stage 2. New stage 2
|
||||
**`5fc292ff0f3b`** **Daily Marketing Job Radar - Write and Letters** (`30 11,14 * * 1-5`, Telegram topic
|
||||
1030, created **2026-10-27T17:34:15**) lands the Notion entries, cover letters, missing-letter audit and
|
||||
report. Two stage-2 test fires on 10-27 (**17:35:00 → 17:36:52**, **17:38:38 → 17:43:42**) both
|
||||
`completed`, delivery `suppressed`, `last_status: ok`, streak 0. Pins unchanged (`custom:omni-101` /
|
||||
`smart-route`). **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** → `runbooks/job-radar.md`
|
||||
- **The 2026-10-26 orphan class did not recur; its two incident rows closed.** `ea79a2_a4661554a3c2` and
|
||||
`ea79a2_c3ab2d784ba3` **auto-closed 2026-10-27T00:38:32+08:00** — unclosed rows fleet-wide **3 → 2** (the
|
||||
2: `0e753e_e7f8194a7d06` Facebook birthday monthly scrape, open since 2026-10-01 09:00:55, and the new
|
||||
`dca848_e93a538ade6b`). No 10-27 execution row carries the orphan strings and the watchdog alert log has no
|
||||
10-27 entry. → `incidents/2026-10-26-planned-restart-orphaned-cron-run.md`
|
||||
- **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error`
|
||||
(streak 1) and `08eb86f02d37` re-joined as `delivery_queued`. Also non-clean: `0e753e9720fd` (error),
|
||||
`fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed).
|
||||
- **Two config corrections from live probes.** `tts.provider` is **`openai`** pointed at the **Mac mini speech
|
||||
server** (`https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, model `phonon`, voice `alloy`) — the
|
||||
page still claimed ElevenLabs from 2026-10-10 (change date not established, so the old bullet is marked
|
||||
superseded rather than deleted). Live voice uses `voice.speech_server.url` =
|
||||
`wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/`; `GET /v1/pool` → `{"size":2,"in_use":0}`, both
|
||||
units `idle`, while `/v1/models` returns `{"detail":"Not Found"}`. `voice.auto_tts` is still `false`, and
|
||||
`wake_word.enabled` is `true`. → `systems/hermes-agent.md`, `systems/conduit.md`
|
||||
- **`systems/skills-index.md` was regenerated by the 2026-10-27 03:00 slot but never committed** — the
|
||||
authoring tree was dirty with `updated: 2026-10-27` and a **302 skills / 69 categories** total (against
|
||||
index.md's stale **306 / 70 of 2026-10-23**). Freshness at that run: **0 stale / 104 fresh / 4 no-date of
|
||||
108 pages** (`freshness-20261027.txt`). Committed, and `index.md` corrected to 302.
|
||||
- **`infrastructure/tailscale.md` peer table re-verified** against `tailscale status --json`; the node-count
|
||||
header and any state changes are recorded in that page.
|
||||
- **Files:** new `incidents/2026-10-27-job-radar-budget-exhausted.md`; updated
|
||||
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md`, `runbooks/job-radar.md`,
|
||||
`systems/scheduled-tasks.md`, `systems/hermes-agent.md`, `systems/conduit.md`,
|
||||
`infrastructure/tailscale.md`, `systems/skills-index.md`, `current-state.md`, `index.md`, `log.md`.
|
||||
**Nothing removed or archived.** No secrets written.
|
||||
- **Unresolved:** whether the two-stage split actually fixes the budget problem — the first real 07:00 → 11:30
|
||||
cycle (stage 1 handoff → stage 2 writes) had not run at audit time; why a budget exhaustion is classified
|
||||
`failure_type: auth`; and the 10-27 radar incident row will only close when a clean stage-1 slot runs.
|
||||
|
||||
Reference in New Issue
Block a user