audit 2026-11-06 (window 2026-11-05): incident ledger emptied 3 -> 0, first zero-failed day since 11-02, both 11-04 casualties recovered; NEW OPEN incident - live .env values verbatim in 17 mnemosyne rows, the 1.7 GB session store and two enabled Facebook Birthday prompts; freshness 0 stale / 124 fresh / 3 no-date of 127

This commit is contained in:
Tony0410
2026-11-06 00:29:52 +08:00
parent 2f3fc9952b
commit eba8062735
9 changed files with 460 additions and 22 deletions
@@ -1,10 +1,10 @@
---
title: "Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)"
type: incident
status: open
status: resolved
created: 2026-11-05
updated: 2026-11-05
verified_on: 2026-11-05
updated: 2026-11-06
verified_on: 2026-11-06
confidence: high
tags: [cron, job-radar, newsletter-digest, omni-101, smart-route, stream-failure, incident]
sources: [live-executions-db, live-cron-incidents, live-deliveries-db, live-cron-output, live-jobs-json]
@@ -113,6 +113,18 @@ until re-tested** rather than as proof the route is dead. See [[runbooks/job-rad
route incident - and whether the 11-03 row should be closed.
4. The stocktake's third overrun and its `free-stack` pin need a decision - see [[systems/scheduled-tasks]].
## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)
- **Both affected jobs ran clean on 2026-11-05.** `85edf6772bca` (*Tech & AI Newsletter Digest (Analytical)*)
completed its 07:00 run in **7 m 01 s** (07:00:42 -> 07:07:43), its incident row closing at **07:07:42**;
`dca8482e4f76` (*Daily Marketing Job Radar - Discovery*) completed in **5 h 35 m 35 s** and its two rows
closed at **12:36:17**. **Fleet-wide unclosed incident rows: 3 -> 0** - the ledger was empty for the first
time since 2026-11-02, and 2026-11-05 was the first zero-`failed` ledger day since then too.
- The Stage 2 salvage documented above therefore stands as the whole recovery - no follow-up failure arrived.
- The `CL_MODEL` / HTTP 402 cover-letter caveat is **unchanged and still needs a decision**: the
`gemini/gemini-3.5-flash` pin was never written to `.env`, so it did not persist.
- **Status -> `resolved`.** A recurrence should be opened as a new incident with its own window.
## Related
- [[incidents/2026-11-03-job-radar-stream-active-timeout]]