audit: 2026-10-28 nightly — radar budget overrun now a recorded failure (new incident page); radar split into two cron jobs; tailscale peer table re-verified; TTS provider corrected to the Mac mini speech server

Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76
(Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed
failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before
Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and
cover letters never ran (Notion DB held 451 entries, 0 added). Incident row
dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified).

Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5,
deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5,
topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3).

Also: the two incident rows the 2026-10-26 planned restart raised auto-closed
00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean
last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never
committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of
108) — committed here and index.md corrected; tailscale.md peer table rebuilt from
tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo
CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to
provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained
the voice pipeline section.

New: incidents/2026-10-27-job-radar-budget-exhausted.md
Nothing removed or archived. No secrets.
This commit is contained in:
Tony0410
2026-10-28 00:20:59 +08:00
parent a2dd0bd650
commit f880ba981d
13 changed files with 417 additions and 43 deletions
+52
View File
@@ -1454,3 +1454,55 @@ not pushed.
whether `failure_streak` should count the *unverified* row (one interruption currently reads streak 2); and the
**`Thread 34795`** delivery target needs re-pointing, or the Monday 03:08 digest will keep falling back to the
unthreaded chat.
## [2026-10-28] audit | Nightly wiki audit — window 2026-10-26 00:00 → 2026-10-27 00:5x AWST: a planned restart orphaned a live cron run
- **Window 2026-10-27: 172 `completed` / 1 `failed` (+ this audit `running`).** The failure is the radar's
budget overrun, and it is the story of the night → new page
[[incidents/2026-10-27-job-radar-budget-exhausted]].
- **The radar's tool-call budget was exhausted for the second consecutive day — and this time the run is
recorded as `failed`.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed` and
delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery, verification,
dedup and classification completed; **Notion ingestion, cover-letter generation and the final cover-letter
audit never ran**. Notion DB `0f90ba2b-8b10-4d02-a62c-6f692d5b1168` held **451 entries** with **0** added
for 2026-10-27 (Agent — Rhino 402 / Chaos 12 / Manual 2 / blank 35; Source — SEEK 295 / LinkedIn 88).
Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** at this audit
(`failure_type: auth` — which does not describe a budget fault). The previous day's identical overrun closed
**`completed`** and was found only by reading the delivered text — that is why the 10-27 row matters.
- **Response, same day: the radar is now two jobs.** Stage 1 `dca8482e4f76` **Daily Marketing Job Radar -
Discovery** (`0 7 * * 1-5`, `deliver: local`, failures → Telegram topic 1030) does
`DISCOVERY → VERIFY → DEDUP → SCORE`, writes a handoff file and triggers stage 2. New stage 2
**`5fc292ff0f3b`** **Daily Marketing Job Radar - Write and Letters** (`30 11,14 * * 1-5`, Telegram topic
1030, created **2026-10-27T17:34:15**) lands the Notion entries, cover letters, missing-letter audit and
report. Two stage-2 test fires on 10-27 (**17:35:00 → 17:36:52**, **17:38:38 → 17:43:42**) both
`completed`, delivery `suppressed`, `last_status: ok`, streak 0. Pins unchanged (`custom:omni-101` /
`smart-route`). **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** → `runbooks/job-radar.md`
- **The 2026-10-26 orphan class did not recur; its two incident rows closed.** `ea79a2_a4661554a3c2` and
`ea79a2_c3ab2d784ba3` **auto-closed 2026-10-27T00:38:32+08:00** — unclosed rows fleet-wide **3 → 2** (the
2: `0e753e_e7f8194a7d06` Facebook birthday monthly scrape, open since 2026-10-01 09:00:55, and the new
`dca848_e93a538ade6b`). No 10-27 execution row carries the orphan strings and the watchdog alert log has no
10-27 entry. → `incidents/2026-10-26-planned-restart-orphaned-cron-run.md`
- **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error`
(streak 1) and `08eb86f02d37` re-joined as `delivery_queued`. Also non-clean: `0e753e9720fd` (error),
`fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed).
- **Two config corrections from live probes.** `tts.provider` is **`openai`** pointed at the **Mac mini speech
server** (`https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, model `phonon`, voice `alloy`) — the
page still claimed ElevenLabs from 2026-10-10 (change date not established, so the old bullet is marked
superseded rather than deleted). Live voice uses `voice.speech_server.url` =
`wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/`; `GET /v1/pool` → `{"size":2,"in_use":0}`, both
units `idle`, while `/v1/models` returns `{"detail":"Not Found"}`. `voice.auto_tts` is still `false`, and
`wake_word.enabled` is `true`. → `systems/hermes-agent.md`, `systems/conduit.md`
- **`systems/skills-index.md` was regenerated by the 2026-10-27 03:00 slot but never committed** — the
authoring tree was dirty with `updated: 2026-10-27` and a **302 skills / 69 categories** total (against
index.md's stale **306 / 70 of 2026-10-23**). Freshness at that run: **0 stale / 104 fresh / 4 no-date of
108 pages** (`freshness-20261027.txt`). Committed, and `index.md` corrected to 302.
- **`infrastructure/tailscale.md` peer table re-verified** against `tailscale status --json`; the node-count
header and any state changes are recorded in that page.
- **Files:** new `incidents/2026-10-27-job-radar-budget-exhausted.md`; updated
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md`, `runbooks/job-radar.md`,
`systems/scheduled-tasks.md`, `systems/hermes-agent.md`, `systems/conduit.md`,
`infrastructure/tailscale.md`, `systems/skills-index.md`, `current-state.md`, `index.md`, `log.md`.
**Nothing removed or archived.** No secrets written.
- **Unresolved:** whether the two-stage split actually fixes the budget problem — the first real 07:00 → 11:30
cycle (stage 1 handoff → stage 2 writes) had not run at audit time; why a budget exhaustion is classified
`failure_type: auth`; and the 10-27 radar incident row will only close when a clean stage-1 slot runs.