Files
agent-estate-wiki/incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
T

133 lines
7.7 KiB
Markdown

---
title: "Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)"
type: incident
status: resolved
created: 2026-11-05
updated: 2026-11-06
verified_on: 2026-11-06
confidence: high
tags: [cron, job-radar, newsletter-digest, omni-101, smart-route, stream-failure, incident]
sources: [live-executions-db, live-cron-incidents, live-deliveries-db, live-cron-output, live-jobs-json]
---
# Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)
**Window:** 2026-11-04 07:00:45 to 09:42:20 AWST (2 h 41 m 35 s) - **Status:** both incidents still open
(`state = alerted`); no data lost - the Job Radar's day was recovered by its Stage 2 safety-net slot.
## Summary
Two independent 07:00 jobs died on the **same upstream fault, 36 seconds apart** on 2026-11-04:
- `dca8482e4f76` **Daily Marketing Job Radar - Discovery** (`0 7 * * 1-5`)
- `85edf6772bca` **Tech & AI Newsletter Digest (Analytical)** (`0 7 * * *`)
Both ran **2 h 41 m** and ended with the identical error, `RuntimeError: Upstream stream failed before
completion.` Both are pinned to `custom:omni-101` / `smart-route`. Neither produced output. Two
`cron_incidents` rows were raised and alerted, and both notices were delivered.
Unlike the 2026-11-02 and 2026-11-03 incidents - where an aborted Stage 1 cost the day's candidate set -
**the Job Radar's day survived**: the 11:30 Stage 2 slot reconstructed the missing handoff, re-verified every
candidate live, and still wrote 5 roles and 1 cover letter. See *The recovery* below.
## Evidence
Live `executions.db`, `cron_incidents`, `deliveries.db` and `jobs.json`, read 2026-11-05 00:5x AWST:
| Job | Start | End | Duration | Status | Incident row |
|---|---|---|---|---|---|
| `dca8482e4f76` Daily Marketing Job Radar - Discovery | 07:00:45 | 09:42:20 | 2 h 41 m 35 s | `error` | `dca848_3b1bb1e9b003` (`unknown`, open) |
| `85edf6772bca` Tech & AI Newsletter Digest (Analytical) | 07:00:45 | 09:41:44 | 2 h 40 m 59 s | `error` | `85edf6_c52a11a62044` (`unknown`, open) |
Both error strings: `RuntimeError: Upstream stream failed before completion.`
All three open incident rows carry `state = alerted` with `closed_at` NULL - the 2026-11-03 `timeout` row
`dca848_dfd4bea84f2a` (first seen `2026-11-03T13:28:51`) plus these two (`09:41:43` and `09:42:19`).
**Unclosed incident rows 1 -> 3.**
Deliveries on 11-04: **8 rows, 8 `delivered`** - including both failure notices (09:41:43, 09:42:20). The
alerts were not silent.
## Why this is not simply the 2026-11-03 incident again
| | 2026-11-03 | 2026-11-04 |
|---|---|---|
| Error | `RuntimeError: stream active timeout` | `RuntimeError: Upstream stream failed before completion.` |
| Duration | 6 h 28 m 18 s | 2 h 41 m (both jobs) |
| Jobs hit | 1 (`dca8482e4f76`) | 2, in the same minute |
| Row | `dca848_dfd4bea84f2a` | `85edf6_c52a11a62044`, `dca848_3b1bb1e9b003` |
A timeout is a stall; "stream failed before completion" is an upstream drop. Both land in the stream layer
of the `smart-route` combo, on `custom:omni-101`. Two unrelated prompts - job discovery and a newsletter
digest - failing within 36 seconds of each other points at the route/provider, not at either job.
## The recovery
Stage 2 (`5fc292ff0f3b`) ran in its 11:30 safety-net slot and finished **15:58-16:09 AWST: Result PASS,
0 missing cover letters**:
- Stage 1 wrote **no handoff**. Stage 2 **reconstructed it from Stage 1's own candidate files** and
re-verified every surviving candidate **live in Camofox** before writing anything.
- **5 roles written to Notion** (board **479 -> 484**): Colab Resources (Group Marketing Manager, strong
match), Foundation for the WA Museum, Mindful Meditation Australia, HPH Solutions, RecruitShop.
- Discovery totals carried forward: **34 discovered / 10 duplicates avoided / 20 rejected**.
- **1 strong match -> 1 canonical cover letter** (Colab Resources) rendered through the canonical teal
template: `.txt`/`.json`/`.html`/`.docx`/`.pdf` all present and non-empty, no `{{}}` tokens, HTTP 200 on
`:9099/jobs/drafts/...`. `SAME_DAY_STRONG_COUNT 1` / `VERIFIED_COVER_LETTER_COUNT 1` /
`MISSING_COVER_LETTER_COUNT 0`.
- Handoff marked `consumed: true` at **16:08:59** (atomic write), so the 14:30 slot stayed silent; delivery
at 16:12:22.
- The run also caught **its own** drafting error: the first Colab letter misstated three resume facts (an
invented "40% sales increase", Horizon Power programmes attributed to Pacific Energy, a stray en-dash).
It was corrected against `current_resume_from_website.txt` and re-rendered - a live demonstration of the
runbook's "verify against the canonical resume" rule.
**So the 11-04 failure cost a Stage 1 run, not the day** - the first time the split pipeline's safety-net
slot has had to carry the day alone.
## The cover-letter route is only alive by override
`regenerate_cover_letter.py` line 82 reads `os.environ.get("CL_MODEL", "deepseek-paid-smart")`. During the
11-04 run that default returned **HTTP 402 (billing)**, so generation was pinned to
**`gemini/gemini-3.5-flash`** for that process. `grep` finds **no `CL_MODEL` in `~/.hermes/.env`** - the
override never persisted, so the next run inherits the default again. The same model name still serves other
cron jobs (`cbfe434a3b5e`, this nightly audit, `last_status: ok`), so treat a 402 as **route/time-specific
until re-tested** rather than as proof the route is dead. See [[runbooks/job-radar]].
## Other in-window cron faults
- `1a8e90ea2271` **wiki-stocktake-maintenance** ran **03:00:31 -> 08:42:17 (5 h 41 m 46 s)**, hit the turn
ceiling, reported incomplete, delivery `suppressed` - the **third consecutive overrun** of this job. Its
pin is `custom:omni-101` / **`free-stack`**. It overlapped the 07:00 window Stage 1 was running in.
- `066bb5191280` **Uber Eats Monthly Spend Report** carries `last_status: delivery_failed` (streak 0) - part
of the non-clean count, not a new failure.
## Open questions / next actions
1. Both 11-04 failures share a signature and a route with the 11-03 timeout. **No output file exists for
either** - the fault lands upstream of the agent's first turn.
2. `failure_type` is stored as **`unknown`** for a stream fault, so neither new row groups with the 11-03
`timeout` row. The incident table undercounts what looks like one route-level pattern (same miscoding
seen on the 2026-10-27 row, stored as `auth` for a 400).
3. All three rows are `alerted` and unclosed. Decide whether the two 11-04 rows are two incidents or one
route incident - and whether the 11-03 row should be closed.
4. The stocktake's third overrun and its `free-stack` pin need a decision - see [[systems/scheduled-tasks]].
## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)
- **Both affected jobs ran clean on 2026-11-05.** `85edf6772bca` (*Tech & AI Newsletter Digest (Analytical)*)
completed its 07:00 run in **7 m 01 s** (07:00:42 -> 07:07:43), its incident row closing at **07:07:42**;
`dca8482e4f76` (*Daily Marketing Job Radar - Discovery*) completed in **5 h 35 m 35 s** and its two rows
closed at **12:36:17**. **Fleet-wide unclosed incident rows: 3 -> 0** - the ledger was empty for the first
time since 2026-11-02, and 2026-11-05 was the first zero-`failed` ledger day since then too.
- The Stage 2 salvage documented above therefore stands as the whole recovery - no follow-up failure arrived.
- The `CL_MODEL` / HTTP 402 cover-letter caveat is **unchanged and still needs a decision**: the
`gemini/gemini-3.5-flash` pin was never written to `.env`, so it did not persist.
- **Status -> `resolved`.** A recurrence should be opened as a new incident with its own window.
## Related
- [[incidents/2026-11-03-job-radar-stream-active-timeout]]
- [[systems/scheduled-tasks]]
- [[runbooks/job-radar]]