6.7 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04) | incident | open | 2026-11-05 | 2026-11-05 | 2026-11-05 | high |
|
|
Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)
Window: 2026-11-04 07:00:45 to 09:42:20 AWST (2 h 41 m 35 s) - Status: both incidents still open
(state = alerted); no data lost - the Job Radar's day was recovered by its Stage 2 safety-net slot.
Summary
Two independent 07:00 jobs died on the same upstream fault, 36 seconds apart on 2026-11-04:
dca8482e4f76Daily Marketing Job Radar - Discovery (0 7 * * 1-5)85edf6772bcaTech & AI Newsletter Digest (Analytical) (0 7 * * *)
Both ran 2 h 41 m and ended with the identical error, RuntimeError: Upstream stream failed before completion. Both are pinned to custom:omni-101 / smart-route. Neither produced output. Two
cron_incidents rows were raised and alerted, and both notices were delivered.
Unlike the 2026-11-02 and 2026-11-03 incidents - where an aborted Stage 1 cost the day's candidate set - the Job Radar's day survived: the 11:30 Stage 2 slot reconstructed the missing handoff, re-verified every candidate live, and still wrote 5 roles and 1 cover letter. See The recovery below.
Evidence
Live executions.db, cron_incidents, deliveries.db and jobs.json, read 2026-11-05 00:5x AWST:
| Job | Start | End | Duration | Status | Incident row |
|---|---|---|---|---|---|
dca8482e4f76 Daily Marketing Job Radar - Discovery |
07:00:45 | 09:42:20 | 2 h 41 m 35 s | error |
dca848_3b1bb1e9b003 (unknown, open) |
85edf6772bca Tech & AI Newsletter Digest (Analytical) |
07:00:45 | 09:41:44 | 2 h 40 m 59 s | error |
85edf6_c52a11a62044 (unknown, open) |
Both error strings: RuntimeError: Upstream stream failed before completion.
All three open incident rows carry state = alerted with closed_at NULL - the 2026-11-03 timeout row
dca848_dfd4bea84f2a (first seen 2026-11-03T13:28:51) plus these two (09:41:43 and 09:42:19).
Unclosed incident rows 1 -> 3.
Deliveries on 11-04: 8 rows, 8 delivered - including both failure notices (09:41:43, 09:42:20). The
alerts were not silent.
Why this is not simply the 2026-11-03 incident again
| 2026-11-03 | 2026-11-04 | |
|---|---|---|
| Error | RuntimeError: stream active timeout |
RuntimeError: Upstream stream failed before completion. |
| Duration | 6 h 28 m 18 s | 2 h 41 m (both jobs) |
| Jobs hit | 1 (dca8482e4f76) |
2, in the same minute |
| Row | dca848_dfd4bea84f2a |
85edf6_c52a11a62044, dca848_3b1bb1e9b003 |
A timeout is a stall; "stream failed before completion" is an upstream drop. Both land in the stream layer
of the smart-route combo, on custom:omni-101. Two unrelated prompts - job discovery and a newsletter
digest - failing within 36 seconds of each other points at the route/provider, not at either job.
The recovery
Stage 2 (5fc292ff0f3b) ran in its 11:30 safety-net slot and finished 15:58-16:09 AWST: Result PASS,
0 missing cover letters:
- Stage 1 wrote no handoff. Stage 2 reconstructed it from Stage 1's own candidate files and re-verified every surviving candidate live in Camofox before writing anything.
- 5 roles written to Notion (board 479 -> 484): Colab Resources (Group Marketing Manager, strong match), Foundation for the WA Museum, Mindful Meditation Australia, HPH Solutions, RecruitShop.
- Discovery totals carried forward: 34 discovered / 10 duplicates avoided / 20 rejected.
- 1 strong match -> 1 canonical cover letter (Colab Resources) rendered through the canonical teal
template:
.txt/.json/.html/.docx/.pdfall present and non-empty, no{{}}tokens, HTTP 200 on:9099/jobs/drafts/....SAME_DAY_STRONG_COUNT 1/VERIFIED_COVER_LETTER_COUNT 1/MISSING_COVER_LETTER_COUNT 0. - Handoff marked
consumed: trueat 16:08:59 (atomic write), so the 14:30 slot stayed silent; delivery at 16:12:22. - The run also caught its own drafting error: the first Colab letter misstated three resume facts (an
invented "40% sales increase", Horizon Power programmes attributed to Pacific Energy, a stray en-dash).
It was corrected against
current_resume_from_website.txtand re-rendered - a live demonstration of the runbook's "verify against the canonical resume" rule.
So the 11-04 failure cost a Stage 1 run, not the day - the first time the split pipeline's safety-net slot has had to carry the day alone.
The cover-letter route is only alive by override
regenerate_cover_letter.py line 82 reads os.environ.get("CL_MODEL", "deepseek-paid-smart"). During the
11-04 run that default returned HTTP 402 (billing), so generation was pinned to
gemini/gemini-3.5-flash for that process. grep finds no CL_MODEL in ~/.hermes/.env - the
override never persisted, so the next run inherits the default again. The same model name still serves other
cron jobs (cbfe434a3b5e, this nightly audit, last_status: ok), so treat a 402 as route/time-specific
until re-tested rather than as proof the route is dead. See runbooks/job-radar.
Other in-window cron faults
1a8e90ea2271wiki-stocktake-maintenance ran 03:00:31 -> 08:42:17 (5 h 41 m 46 s), hit the turn ceiling, reported incomplete, deliverysuppressed- the third consecutive overrun of this job. Its pin iscustom:omni-101/free-stack. It overlapped the 07:00 window Stage 1 was running in.066bb5191280Uber Eats Monthly Spend Report carrieslast_status: delivery_failed(streak 0) - part of the non-clean count, not a new failure.
Open questions / next actions
- Both 11-04 failures share a signature and a route with the 11-03 timeout. No output file exists for either - the fault lands upstream of the agent's first turn.
failure_typeis stored asunknownfor a stream fault, so neither new row groups with the 11-03timeoutrow. The incident table undercounts what looks like one route-level pattern (same miscoding seen on the 2026-10-27 row, stored asauthfor a 400).- All three rows are
alertedand unclosed. Decide whether the two 11-04 rows are two incidents or one route incident - and whether the 11-03 row should be closed. - The stocktake's third overrun and its
free-stackpin need a decision - see systems/scheduled-tasks.