121 lines
6.7 KiB
Markdown
121 lines
6.7 KiB
Markdown
---
|
|
title: "Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)"
|
|
type: incident
|
|
status: open
|
|
created: 2026-11-05
|
|
updated: 2026-11-05
|
|
verified_on: 2026-11-05
|
|
confidence: high
|
|
tags: [cron, job-radar, newsletter-digest, omni-101, smart-route, stream-failure, incident]
|
|
sources: [live-executions-db, live-cron-incidents, live-deliveries-db, live-cron-output, live-jobs-json]
|
|
---
|
|
|
|
# Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)
|
|
|
|
**Window:** 2026-11-04 07:00:45 to 09:42:20 AWST (2 h 41 m 35 s) - **Status:** both incidents still open
|
|
(`state = alerted`); no data lost - the Job Radar's day was recovered by its Stage 2 safety-net slot.
|
|
|
|
## Summary
|
|
|
|
Two independent 07:00 jobs died on the **same upstream fault, 36 seconds apart** on 2026-11-04:
|
|
|
|
- `dca8482e4f76` **Daily Marketing Job Radar - Discovery** (`0 7 * * 1-5`)
|
|
- `85edf6772bca` **Tech & AI Newsletter Digest (Analytical)** (`0 7 * * *`)
|
|
|
|
Both ran **2 h 41 m** and ended with the identical error, `RuntimeError: Upstream stream failed before
|
|
completion.` Both are pinned to `custom:omni-101` / `smart-route`. Neither produced output. Two
|
|
`cron_incidents` rows were raised and alerted, and both notices were delivered.
|
|
|
|
Unlike the 2026-11-02 and 2026-11-03 incidents - where an aborted Stage 1 cost the day's candidate set -
|
|
**the Job Radar's day survived**: the 11:30 Stage 2 slot reconstructed the missing handoff, re-verified every
|
|
candidate live, and still wrote 5 roles and 1 cover letter. See *The recovery* below.
|
|
|
|
## Evidence
|
|
|
|
Live `executions.db`, `cron_incidents`, `deliveries.db` and `jobs.json`, read 2026-11-05 00:5x AWST:
|
|
|
|
| Job | Start | End | Duration | Status | Incident row |
|
|
|---|---|---|---|---|---|
|
|
| `dca8482e4f76` Daily Marketing Job Radar - Discovery | 07:00:45 | 09:42:20 | 2 h 41 m 35 s | `error` | `dca848_3b1bb1e9b003` (`unknown`, open) |
|
|
| `85edf6772bca` Tech & AI Newsletter Digest (Analytical) | 07:00:45 | 09:41:44 | 2 h 40 m 59 s | `error` | `85edf6_c52a11a62044` (`unknown`, open) |
|
|
|
|
Both error strings: `RuntimeError: Upstream stream failed before completion.`
|
|
|
|
All three open incident rows carry `state = alerted` with `closed_at` NULL - the 2026-11-03 `timeout` row
|
|
`dca848_dfd4bea84f2a` (first seen `2026-11-03T13:28:51`) plus these two (`09:41:43` and `09:42:19`).
|
|
**Unclosed incident rows 1 -> 3.**
|
|
|
|
Deliveries on 11-04: **8 rows, 8 `delivered`** - including both failure notices (09:41:43, 09:42:20). The
|
|
alerts were not silent.
|
|
|
|
## Why this is not simply the 2026-11-03 incident again
|
|
|
|
| | 2026-11-03 | 2026-11-04 |
|
|
|---|---|---|
|
|
| Error | `RuntimeError: stream active timeout` | `RuntimeError: Upstream stream failed before completion.` |
|
|
| Duration | 6 h 28 m 18 s | 2 h 41 m (both jobs) |
|
|
| Jobs hit | 1 (`dca8482e4f76`) | 2, in the same minute |
|
|
| Row | `dca848_dfd4bea84f2a` | `85edf6_c52a11a62044`, `dca848_3b1bb1e9b003` |
|
|
|
|
A timeout is a stall; "stream failed before completion" is an upstream drop. Both land in the stream layer
|
|
of the `smart-route` combo, on `custom:omni-101`. Two unrelated prompts - job discovery and a newsletter
|
|
digest - failing within 36 seconds of each other points at the route/provider, not at either job.
|
|
|
|
## The recovery
|
|
|
|
Stage 2 (`5fc292ff0f3b`) ran in its 11:30 safety-net slot and finished **15:58-16:09 AWST: Result PASS,
|
|
0 missing cover letters**:
|
|
|
|
- Stage 1 wrote **no handoff**. Stage 2 **reconstructed it from Stage 1's own candidate files** and
|
|
re-verified every surviving candidate **live in Camofox** before writing anything.
|
|
- **5 roles written to Notion** (board **479 -> 484**): Colab Resources (Group Marketing Manager, strong
|
|
match), Foundation for the WA Museum, Mindful Meditation Australia, HPH Solutions, RecruitShop.
|
|
- Discovery totals carried forward: **34 discovered / 10 duplicates avoided / 20 rejected**.
|
|
- **1 strong match -> 1 canonical cover letter** (Colab Resources) rendered through the canonical teal
|
|
template: `.txt`/`.json`/`.html`/`.docx`/`.pdf` all present and non-empty, no `{{}}` tokens, HTTP 200 on
|
|
`:9099/jobs/drafts/...`. `SAME_DAY_STRONG_COUNT 1` / `VERIFIED_COVER_LETTER_COUNT 1` /
|
|
`MISSING_COVER_LETTER_COUNT 0`.
|
|
- Handoff marked `consumed: true` at **16:08:59** (atomic write), so the 14:30 slot stayed silent; delivery
|
|
at 16:12:22.
|
|
- The run also caught **its own** drafting error: the first Colab letter misstated three resume facts (an
|
|
invented "40% sales increase", Horizon Power programmes attributed to Pacific Energy, a stray en-dash).
|
|
It was corrected against `current_resume_from_website.txt` and re-rendered - a live demonstration of the
|
|
runbook's "verify against the canonical resume" rule.
|
|
|
|
**So the 11-04 failure cost a Stage 1 run, not the day** - the first time the split pipeline's safety-net
|
|
slot has had to carry the day alone.
|
|
|
|
## The cover-letter route is only alive by override
|
|
|
|
`regenerate_cover_letter.py` line 82 reads `os.environ.get("CL_MODEL", "deepseek-paid-smart")`. During the
|
|
11-04 run that default returned **HTTP 402 (billing)**, so generation was pinned to
|
|
**`gemini/gemini-3.5-flash`** for that process. `grep` finds **no `CL_MODEL` in `~/.hermes/.env`** - the
|
|
override never persisted, so the next run inherits the default again. The same model name still serves other
|
|
cron jobs (`cbfe434a3b5e`, this nightly audit, `last_status: ok`), so treat a 402 as **route/time-specific
|
|
until re-tested** rather than as proof the route is dead. See [[runbooks/job-radar]].
|
|
|
|
## Other in-window cron faults
|
|
|
|
- `1a8e90ea2271` **wiki-stocktake-maintenance** ran **03:00:31 -> 08:42:17 (5 h 41 m 46 s)**, hit the turn
|
|
ceiling, reported incomplete, delivery `suppressed` - the **third consecutive overrun** of this job. Its
|
|
pin is `custom:omni-101` / **`free-stack`**. It overlapped the 07:00 window Stage 1 was running in.
|
|
- `066bb5191280` **Uber Eats Monthly Spend Report** carries `last_status: delivery_failed` (streak 0) - part
|
|
of the non-clean count, not a new failure.
|
|
|
|
## Open questions / next actions
|
|
|
|
1. Both 11-04 failures share a signature and a route with the 11-03 timeout. **No output file exists for
|
|
either** - the fault lands upstream of the agent's first turn.
|
|
2. `failure_type` is stored as **`unknown`** for a stream fault, so neither new row groups with the 11-03
|
|
`timeout` row. The incident table undercounts what looks like one route-level pattern (same miscoding
|
|
seen on the 2026-10-27 row, stored as `auth` for a 400).
|
|
3. All three rows are `alerted` and unclosed. Decide whether the two 11-04 rows are two incidents or one
|
|
route incident - and whether the 11-03 row should be closed.
|
|
4. The stocktake's third overrun and its `free-stack` pin need a decision - see [[systems/scheduled-tasks]].
|
|
|
|
## Related
|
|
|
|
- [[incidents/2026-11-03-job-radar-stream-active-timeout]]
|
|
- [[systems/scheduled-tasks]]
|
|
- [[runbooks/job-radar]]
|