Files
agent-estate-wiki/incidents/2026-11-04-upstream-stream-failure-0700-cohort.md

7.7 KiB

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04) incident resolved 2026-11-05 2026-11-06 2026-11-06 high
cron
job-radar
newsletter-digest
omni-101
smart-route
stream-failure
incident
live-executions-db
live-cron-incidents
live-deliveries-db
live-cron-output
live-jobs-json

Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)

Window: 2026-11-04 07:00:45 to 09:42:20 AWST (2 h 41 m 35 s) - Status: both incidents still open (state = alerted); no data lost - the Job Radar's day was recovered by its Stage 2 safety-net slot.

Summary

Two independent 07:00 jobs died on the same upstream fault, 36 seconds apart on 2026-11-04:

  • dca8482e4f76 Daily Marketing Job Radar - Discovery (0 7 * * 1-5)
  • 85edf6772bca Tech & AI Newsletter Digest (Analytical) (0 7 * * *)

Both ran 2 h 41 m and ended with the identical error, RuntimeError: Upstream stream failed before completion. Both are pinned to custom:omni-101 / smart-route. Neither produced output. Two cron_incidents rows were raised and alerted, and both notices were delivered.

Unlike the 2026-11-02 and 2026-11-03 incidents - where an aborted Stage 1 cost the day's candidate set - the Job Radar's day survived: the 11:30 Stage 2 slot reconstructed the missing handoff, re-verified every candidate live, and still wrote 5 roles and 1 cover letter. See The recovery below.

Evidence

Live executions.db, cron_incidents, deliveries.db and jobs.json, read 2026-11-05 00:5x AWST:

Job Start End Duration Status Incident row
dca8482e4f76 Daily Marketing Job Radar - Discovery 07:00:45 09:42:20 2 h 41 m 35 s error dca848_3b1bb1e9b003 (unknown, open)
85edf6772bca Tech & AI Newsletter Digest (Analytical) 07:00:45 09:41:44 2 h 40 m 59 s error 85edf6_c52a11a62044 (unknown, open)

Both error strings: RuntimeError: Upstream stream failed before completion.

All three open incident rows carry state = alerted with closed_at NULL - the 2026-11-03 timeout row dca848_dfd4bea84f2a (first seen 2026-11-03T13:28:51) plus these two (09:41:43 and 09:42:19). Unclosed incident rows 1 -> 3.

Deliveries on 11-04: 8 rows, 8 delivered - including both failure notices (09:41:43, 09:42:20). The alerts were not silent.

Why this is not simply the 2026-11-03 incident again

2026-11-03 2026-11-04
Error RuntimeError: stream active timeout RuntimeError: Upstream stream failed before completion.
Duration 6 h 28 m 18 s 2 h 41 m (both jobs)
Jobs hit 1 (dca8482e4f76) 2, in the same minute
Row dca848_dfd4bea84f2a 85edf6_c52a11a62044, dca848_3b1bb1e9b003

A timeout is a stall; "stream failed before completion" is an upstream drop. Both land in the stream layer of the smart-route combo, on custom:omni-101. Two unrelated prompts - job discovery and a newsletter digest - failing within 36 seconds of each other points at the route/provider, not at either job.

The recovery

Stage 2 (5fc292ff0f3b) ran in its 11:30 safety-net slot and finished 15:58-16:09 AWST: Result PASS, 0 missing cover letters:

  • Stage 1 wrote no handoff. Stage 2 reconstructed it from Stage 1's own candidate files and re-verified every surviving candidate live in Camofox before writing anything.
  • 5 roles written to Notion (board 479 -> 484): Colab Resources (Group Marketing Manager, strong match), Foundation for the WA Museum, Mindful Meditation Australia, HPH Solutions, RecruitShop.
  • Discovery totals carried forward: 34 discovered / 10 duplicates avoided / 20 rejected.
  • 1 strong match -> 1 canonical cover letter (Colab Resources) rendered through the canonical teal template: .txt/.json/.html/.docx/.pdf all present and non-empty, no {{}} tokens, HTTP 200 on :9099/jobs/drafts/.... SAME_DAY_STRONG_COUNT 1 / VERIFIED_COVER_LETTER_COUNT 1 / MISSING_COVER_LETTER_COUNT 0.
  • Handoff marked consumed: true at 16:08:59 (atomic write), so the 14:30 slot stayed silent; delivery at 16:12:22.
  • The run also caught its own drafting error: the first Colab letter misstated three resume facts (an invented "40% sales increase", Horizon Power programmes attributed to Pacific Energy, a stray en-dash). It was corrected against current_resume_from_website.txt and re-rendered - a live demonstration of the runbook's "verify against the canonical resume" rule.

So the 11-04 failure cost a Stage 1 run, not the day - the first time the split pipeline's safety-net slot has had to carry the day alone.

The cover-letter route is only alive by override

regenerate_cover_letter.py line 82 reads os.environ.get("CL_MODEL", "deepseek-paid-smart"). During the 11-04 run that default returned HTTP 402 (billing), so generation was pinned to gemini/gemini-3.5-flash for that process. grep finds no CL_MODEL in ~/.hermes/.env - the override never persisted, so the next run inherits the default again. The same model name still serves other cron jobs (cbfe434a3b5e, this nightly audit, last_status: ok), so treat a 402 as route/time-specific until re-tested rather than as proof the route is dead. See runbooks/job-radar.

Other in-window cron faults

  • 1a8e90ea2271 wiki-stocktake-maintenance ran 03:00:31 -> 08:42:17 (5 h 41 m 46 s), hit the turn ceiling, reported incomplete, delivery suppressed - the third consecutive overrun of this job. Its pin is custom:omni-101 / free-stack. It overlapped the 07:00 window Stage 1 was running in.
  • 066bb5191280 Uber Eats Monthly Spend Report carries last_status: delivery_failed (streak 0) - part of the non-clean count, not a new failure.

Open questions / next actions

  1. Both 11-04 failures share a signature and a route with the 11-03 timeout. No output file exists for either - the fault lands upstream of the agent's first turn.
  2. failure_type is stored as unknown for a stream fault, so neither new row groups with the 11-03 timeout row. The incident table undercounts what looks like one route-level pattern (same miscoding seen on the 2026-10-27 row, stored as auth for a 400).
  3. All three rows are alerted and unclosed. Decide whether the two 11-04 rows are two incidents or one route incident - and whether the 11-03 row should be closed.
  4. The stocktake's third overrun and its free-stack pin need a decision - see systems/scheduled-tasks.

Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)

  • Both affected jobs ran clean on 2026-11-05. 85edf6772bca (Tech & AI Newsletter Digest (Analytical)) completed its 07:00 run in 7 m 01 s (07:00:42 -> 07:07:43), its incident row closing at 07:07:42; dca8482e4f76 (Daily Marketing Job Radar - Discovery) completed in 5 h 35 m 35 s and its two rows closed at 12:36:17. Fleet-wide unclosed incident rows: 3 -> 0 - the ledger was empty for the first time since 2026-11-02, and 2026-11-05 was the first zero-failed ledger day since then too.
  • The Stage 2 salvage documented above therefore stands as the whole recovery - no follow-up failure arrived.
  • The CL_MODEL / HTTP 402 cover-letter caveat is unchanged and still needs a decision: the gemini/gemini-3.5-flash pin was never written to .env, so it did not persist.
  • Status -> resolved. A recurrence should be opened as a new incident with its own window.