audit 2026-11-06 (window 2026-11-05): incident ledger emptied 3 -> 0, first zero-failed day since 11-02, both 11-04 casualties recovered; NEW OPEN incident - live .env values verbatim in 17 mnemosyne rows, the 1.7 GB session store and two enabled Facebook Birthday prompts; freshness 0 stale / 124 fresh / 3 no-date of 127
This commit is contained in:
@@ -1,10 +1,10 @@
|
||||
---
|
||||
title: "Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)"
|
||||
type: incident
|
||||
status: open
|
||||
status: resolved
|
||||
created: 2026-11-05
|
||||
updated: 2026-11-05
|
||||
verified_on: 2026-11-05
|
||||
updated: 2026-11-06
|
||||
verified_on: 2026-11-06
|
||||
confidence: high
|
||||
tags: [cron, job-radar, newsletter-digest, omni-101, smart-route, stream-failure, incident]
|
||||
sources: [live-executions-db, live-cron-incidents, live-deliveries-db, live-cron-output, live-jobs-json]
|
||||
@@ -113,6 +113,18 @@ until re-tested** rather than as proof the route is dead. See [[runbooks/job-rad
|
||||
route incident - and whether the 11-03 row should be closed.
|
||||
4. The stocktake's third overrun and its `free-stack` pin need a decision - see [[systems/scheduled-tasks]].
|
||||
|
||||
## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)
|
||||
|
||||
- **Both affected jobs ran clean on 2026-11-05.** `85edf6772bca` (*Tech & AI Newsletter Digest (Analytical)*)
|
||||
completed its 07:00 run in **7 m 01 s** (07:00:42 -> 07:07:43), its incident row closing at **07:07:42**;
|
||||
`dca8482e4f76` (*Daily Marketing Job Radar - Discovery*) completed in **5 h 35 m 35 s** and its two rows
|
||||
closed at **12:36:17**. **Fleet-wide unclosed incident rows: 3 -> 0** - the ledger was empty for the first
|
||||
time since 2026-11-02, and 2026-11-05 was the first zero-`failed` ledger day since then too.
|
||||
- The Stage 2 salvage documented above therefore stands as the whole recovery - no follow-up failure arrived.
|
||||
- The `CL_MODEL` / HTTP 402 cover-letter caveat is **unchanged and still needs a decision**: the
|
||||
`gemini/gemini-3.5-flash` pin was never written to `.env`, so it did not persist.
|
||||
- **Status -> `resolved`.** A recurrence should be opened as a new incident with its own window.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-11-03-job-radar-stream-active-timeout]]
|
||||
|
||||
Reference in New Issue
Block a user