audit: 2026-10-28 nightly — radar budget overrun now a recorded failure (new incident page); radar split into two cron jobs; tailscale peer table re-verified; TTS provider corrected to the Mac mini speech server
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76 (Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and cover letters never ran (Notion DB held 451 entries, 0 added). Incident row dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified). Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5, deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5, topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3). Also: the two incident rows the 2026-10-26 planned restart raised auto-closed 00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of 108) — committed here and index.md corrected; tailscale.md peer table rebuilt from tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained the voice pipeline section. New: incidents/2026-10-27-job-radar-budget-exhausted.md Nothing removed or archived. No secrets.
This commit is contained in:
@@ -3,7 +3,7 @@ title: Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemos
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-10-27
|
||||
updated: 2026-10-27
|
||||
updated: 2026-10-28
|
||||
verified_on: 2026-10-27
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
@@ -101,3 +101,13 @@ disappears silently. What it did not do is *finish the work* or *avoid the inter
|
||||
- [[systems/scheduled-tasks]] — fleet ledger and per-window tallies
|
||||
- [[systems/mnemosyne]] — the interrupted job's purpose and settings
|
||||
- [[systems/hermes-agent]] — gateway lifecycle and restart history
|
||||
|
||||
## Update — 2026-10-28 (nightly audit)
|
||||
|
||||
- **No recurrence.** Every run in the 2026-10-27 window except the radar's budget overrun completed, and the
|
||||
watchdog logged no alert for the day.
|
||||
- **Both incident rows this fault raised closed automatically** — `ea79a2_a4661554a3c2` and
|
||||
`ea79a2_c3ab2d784ba3` **auto-closed 2026-10-27T00:38:32+08:00** (unclosed rows fleet-wide fell 3 → 2).
|
||||
- The `lost its durable fire claim ownership` string did not appear in any 10-27 execution row.
|
||||
- Open question from the original write-up still stands: why a *planned* stop bypasses the drain deadline
|
||||
(`timed_out=True` at 1.35 s of a 30 s budget).
|
||||
|
||||
Reference in New Issue
Block a user