--- title: "Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)" type: incident status: open created: 2026-11-05 updated: 2026-11-05 verified_on: 2026-11-05 confidence: high tags: [cron, job-radar, newsletter-digest, omni-101, smart-route, stream-failure, incident] sources: [live-executions-db, live-cron-incidents, live-deliveries-db, live-cron-output, live-jobs-json] --- # Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04) **Window:** 2026-11-04 07:00:45 to 09:42:20 AWST (2 h 41 m 35 s) - **Status:** both incidents still open (`state = alerted`); no data lost - the Job Radar's day was recovered by its Stage 2 safety-net slot. ## Summary Two independent 07:00 jobs died on the **same upstream fault, 36 seconds apart** on 2026-11-04: - `dca8482e4f76` **Daily Marketing Job Radar - Discovery** (`0 7 * * 1-5`) - `85edf6772bca` **Tech & AI Newsletter Digest (Analytical)** (`0 7 * * *`) Both ran **2 h 41 m** and ended with the identical error, `RuntimeError: Upstream stream failed before completion.` Both are pinned to `custom:omni-101` / `smart-route`. Neither produced output. Two `cron_incidents` rows were raised and alerted, and both notices were delivered. Unlike the 2026-11-02 and 2026-11-03 incidents - where an aborted Stage 1 cost the day's candidate set - **the Job Radar's day survived**: the 11:30 Stage 2 slot reconstructed the missing handoff, re-verified every candidate live, and still wrote 5 roles and 1 cover letter. See *The recovery* below. ## Evidence Live `executions.db`, `cron_incidents`, `deliveries.db` and `jobs.json`, read 2026-11-05 00:5x AWST: | Job | Start | End | Duration | Status | Incident row | |---|---|---|---|---|---| | `dca8482e4f76` Daily Marketing Job Radar - Discovery | 07:00:45 | 09:42:20 | 2 h 41 m 35 s | `error` | `dca848_3b1bb1e9b003` (`unknown`, open) | | `85edf6772bca` Tech & AI Newsletter Digest (Analytical) | 07:00:45 | 09:41:44 | 2 h 40 m 59 s | `error` | `85edf6_c52a11a62044` (`unknown`, open) | Both error strings: `RuntimeError: Upstream stream failed before completion.` All three open incident rows carry `state = alerted` with `closed_at` NULL - the 2026-11-03 `timeout` row `dca848_dfd4bea84f2a` (first seen `2026-11-03T13:28:51`) plus these two (`09:41:43` and `09:42:19`). **Unclosed incident rows 1 -> 3.** Deliveries on 11-04: **8 rows, 8 `delivered`** - including both failure notices (09:41:43, 09:42:20). The alerts were not silent. ## Why this is not simply the 2026-11-03 incident again | | 2026-11-03 | 2026-11-04 | |---|---|---| | Error | `RuntimeError: stream active timeout` | `RuntimeError: Upstream stream failed before completion.` | | Duration | 6 h 28 m 18 s | 2 h 41 m (both jobs) | | Jobs hit | 1 (`dca8482e4f76`) | 2, in the same minute | | Row | `dca848_dfd4bea84f2a` | `85edf6_c52a11a62044`, `dca848_3b1bb1e9b003` | A timeout is a stall; "stream failed before completion" is an upstream drop. Both land in the stream layer of the `smart-route` combo, on `custom:omni-101`. Two unrelated prompts - job discovery and a newsletter digest - failing within 36 seconds of each other points at the route/provider, not at either job. ## The recovery Stage 2 (`5fc292ff0f3b`) ran in its 11:30 safety-net slot and finished **15:58-16:09 AWST: Result PASS, 0 missing cover letters**: - Stage 1 wrote **no handoff**. Stage 2 **reconstructed it from Stage 1's own candidate files** and re-verified every surviving candidate **live in Camofox** before writing anything. - **5 roles written to Notion** (board **479 -> 484**): Colab Resources (Group Marketing Manager, strong match), Foundation for the WA Museum, Mindful Meditation Australia, HPH Solutions, RecruitShop. - Discovery totals carried forward: **34 discovered / 10 duplicates avoided / 20 rejected**. - **1 strong match -> 1 canonical cover letter** (Colab Resources) rendered through the canonical teal template: `.txt`/`.json`/`.html`/`.docx`/`.pdf` all present and non-empty, no `{{}}` tokens, HTTP 200 on `:9099/jobs/drafts/...`. `SAME_DAY_STRONG_COUNT 1` / `VERIFIED_COVER_LETTER_COUNT 1` / `MISSING_COVER_LETTER_COUNT 0`. - Handoff marked `consumed: true` at **16:08:59** (atomic write), so the 14:30 slot stayed silent; delivery at 16:12:22. - The run also caught **its own** drafting error: the first Colab letter misstated three resume facts (an invented "40% sales increase", Horizon Power programmes attributed to Pacific Energy, a stray en-dash). It was corrected against `current_resume_from_website.txt` and re-rendered - a live demonstration of the runbook's "verify against the canonical resume" rule. **So the 11-04 failure cost a Stage 1 run, not the day** - the first time the split pipeline's safety-net slot has had to carry the day alone. ## The cover-letter route is only alive by override `regenerate_cover_letter.py` line 82 reads `os.environ.get("CL_MODEL", "deepseek-paid-smart")`. During the 11-04 run that default returned **HTTP 402 (billing)**, so generation was pinned to **`gemini/gemini-3.5-flash`** for that process. `grep` finds **no `CL_MODEL` in `~/.hermes/.env`** - the override never persisted, so the next run inherits the default again. The same model name still serves other cron jobs (`cbfe434a3b5e`, this nightly audit, `last_status: ok`), so treat a 402 as **route/time-specific until re-tested** rather than as proof the route is dead. See [[runbooks/job-radar]]. ## Other in-window cron faults - `1a8e90ea2271` **wiki-stocktake-maintenance** ran **03:00:31 -> 08:42:17 (5 h 41 m 46 s)**, hit the turn ceiling, reported incomplete, delivery `suppressed` - the **third consecutive overrun** of this job. Its pin is `custom:omni-101` / **`free-stack`**. It overlapped the 07:00 window Stage 1 was running in. - `066bb5191280` **Uber Eats Monthly Spend Report** carries `last_status: delivery_failed` (streak 0) - part of the non-clean count, not a new failure. ## Open questions / next actions 1. Both 11-04 failures share a signature and a route with the 11-03 timeout. **No output file exists for either** - the fault lands upstream of the agent's first turn. 2. `failure_type` is stored as **`unknown`** for a stream fault, so neither new row groups with the 11-03 `timeout` row. The incident table undercounts what looks like one route-level pattern (same miscoding seen on the 2026-10-27 row, stored as `auth` for a 400). 3. All three rows are `alerted` and unclosed. Decide whether the two 11-04 rows are two incidents or one route incident - and whether the 11-03 row should be closed. 4. The stocktake's third overrun and its `free-stack` pin need a decision - see [[systems/scheduled-tasks]]. ## Related - [[incidents/2026-11-03-job-radar-stream-active-timeout]] - [[systems/scheduled-tasks]] - [[runbooks/job-radar]]