131 lines
7.6 KiB
Markdown
131 lines
7.6 KiB
Markdown
---
|
|
title: Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
|
|
type: incident
|
|
status: resolved
|
|
created: 2026-11-04
|
|
updated: 2026-11-06
|
|
verified_on: 2026-11-06
|
|
review_after: 2026-11-11
|
|
confidence: high
|
|
tags: [cron, job-radar, stream-timeout, silent-failure, notion, context-compaction]
|
|
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-deliveries-db, live-jobs-json, live-state-db, live-gateway-log, live-radar-handoff-dir, session-history]
|
|
---
|
|
|
|
# Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
|
|
|
|
## Headline
|
|
|
|
**The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the
|
|
ledger went red.** On **2026-11-03** the 07:00 `dca8482e4f76` (*Daily Marketing Job Radar — Stage 1
|
|
(Discovery)*) ran **6 h 28 m 18 s** (`started 07:00:33.819190` → `finished 13:28:52.305995 +08:00`) and
|
|
closed:
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| `status` | **`failed`** |
|
|
| `error` | **`RuntimeError: stream active timeout`** |
|
|
| `delivery_outcome` | **`delivered`** (the failure notice, `for_failure: 1`) |
|
|
| `cron_incidents` row | **`dca848_dfd4bea84f2a`**, `failure_type: timeout`, **`alerted`**, still open |
|
|
|
|
No handoff artefact was written, so **nothing reached Notion, no cover letters were generated, and both
|
|
Stage 2 slots declined.** This is the **fourth Stage-1 failure in eight days** and a **new terminal
|
|
signature** — not the iteration cap that ended 10-27, 10-28 and 11-02.
|
|
|
|
## Timeline (all `+08:00`)
|
|
|
|
| Time | Event |
|
|
|---|---|
|
|
| 07:00:33.819190 | execution `dca8482e4f76` opens (07:00 weekday slot) |
|
|
| 07:01:17 | run session `local-7180cef7…` first message |
|
|
| 08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21 | **five context compactions** in one run |
|
|
| 11:30:36.617545 → 11:32:16.812897 | Stage 2 `5fc292ff0f3b` first slot: no handoff → `[SILENT]`, `suppressed` |
|
|
| 12:23:28 | **last tool result of the run** (a `terminal` call) |
|
|
| 13:28:49 | harness records *"This turn did not complete. Some actions may already have run; verify their effects before resending."* |
|
|
| 13:28:50,627 | gateway.log: `ERROR gateway.session_authority: Admitted turn 9bf9a296… failed` (traceback through `session_authority.py:1159 _execute_claimed` → `session_finite.py:80 execute_finite_admission` → `session_ingress.py:235 execute_admission`) |
|
|
| 13:28:51.463929 | incident `dca848_dfd4bea84f2a` opens |
|
|
| 13:28:51.741724 | **failure notice delivered** |
|
|
| 13:28:52.305995 | execution closes `failed` |
|
|
| 13:28:55.141618 | incident `alerted` |
|
|
| 14:30:32.261931 → 14:33:11.470879 | Stage 2 safety-net slot: same silent decline |
|
|
| in-session | Pulse report `p34c8e899-1` (covering Tue 03 Nov 13:01 onward) led with it: *"The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a stream active timeout."* |
|
|
|
|
## The 65-minute silence is the new information
|
|
|
|
The run's recorded activity stops at **12:23:28** and the harness does not declare the turn dead until
|
|
**13:28:49** — **65 m 21 s with no tool call, no model turn and no error written to the ledger**.
|
|
|
|
Contrast with the two cap failures: on **10-28** and **11-02** the runtime *injected* the text *"You've
|
|
reached the maximum number of tool-calling iterations allowed…"* into the session before it ended. This
|
|
session contains **no such message** — it stopped mid-verification, on a provider-stream liveness timeout,
|
|
with the run's own closing plan (*"Two final verification passes, then I'll assemble the handoff"*) never
|
|
executed.
|
|
|
|
**Therefore:** the agent iteration cap is not the only ceiling that ends a long Stage-1 run, and the
|
|
"2026-11-02 raised no incident at all" behaviour was a property of *that* terminal state (`completed`,
|
|
`error: NULL`), **not** of long runs in general. Nothing in the ledger or the gateway log records the
|
|
provider's socket state during the silent window.
|
|
|
|
## Why the ledger went red this time
|
|
|
|
| Date | `executions.status` | `cron_incidents` row | Notice to Anthony |
|
|
|---|---|---|---|
|
|
| 2026-10-27 | non-clean (`failed`) | `dca848_e93a538ade6b` — `failure_type: auth` (**wrong** for a budget fault) | yes — the run's INCOMPLETE report |
|
|
| 2026-10-28 | non-clean (`failed`) | `dca848_0a2f03b5952b` — `failure_type: timeout` | yes |
|
|
| 2026-11-02 | **`completed`, `error: NULL`** | **none raised** | **no — `suppressed`** |
|
|
| **2026-11-03** | **`failed`** | **`dca848_dfd4bea84f2a` — `timeout`, open** | **yes — 13:28:51.741724** |
|
|
|
|
Fleet-wide **unclosed incident rows 0 → 1**. A run the harness itself kills produces an honest red row and
|
|
a delivered notice; the cap produces a green row and silence. That is the distinction
|
|
[[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] could only predict, now observed.
|
|
|
|
## Downstream: both Stage 2 slots behaved correctly, and two days of candidates are lost
|
|
|
|
| Slot | Job | Ran | Result |
|
|
|---|---|---|---|
|
|
| 11:30 | `5fc292ff0f3b` (*Write and Letters*) | 11:30:36 → 11:32:16 | `[SILENT]` / `suppressed` |
|
|
| 14:30 (safety net) | `5fc292ff0f3b` | 14:30:32 → 14:33:11 | `[SILENT]` / `suppressed` |
|
|
|
|
`~/.hermes/radar-ref/handoff/` holds **only** `2026-10-27.json`, `2026-10-29.json` and `2026-10-30.json` —
|
|
nothing for 10-31, **11-02** or **11-03**. So **2026-11-02's 164 deduped candidates and 2026-11-03's
|
|
discovery are both unrecovered**, and Notion received nothing on either day. The last end-to-end success
|
|
remains **2026-10-29** (handoff written inside budget; Stage 2 landed **20 pages + 2 cover letters**,
|
|
`missing_cover_letter_count 0`).
|
|
|
|
## Context churn (observation, cause not established)
|
|
|
|
**363 messages** (6 `user` / 133 `assistant` / 224 `tool`), **288 compaction-flagged messages**, and **five
|
|
compactions** inside one 6 h run. The 11-02 run compacted repeatedly as well. Whether compaction churn
|
|
contributes to the stream stall is **not established** — this is recorded because both long failures share
|
|
the pattern, not because the link is proven.
|
|
|
|
## Open questions
|
|
|
|
1. Is the **65 m 21 s** silence the stream-active watchdog's own threshold, or did the provider socket die
|
|
silently long before the harness noticed?
|
|
2. Does repeated context compaction contribute to the stall? Two long runs, both compacting heavily.
|
|
3. The 07:00 run has now overrun past the 11:30 Stage-2 slot two days running — should the pairing be
|
|
re-slotted, or Stage 2 made to wait on a handoff?
|
|
4. Where does `radar-ref/handoff/` get its retention? The newest file is 10-30.
|
|
5. Should a stream timeout on a *file-deliverable* job trigger an automatic retry instead of leaving the
|
|
day's discovery to the next slot?
|
|
|
|
## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)
|
|
|
|
- The last open row raised here, `dca848_dfd4bea84f2a` (`timeout`, opened **2026-11-03 13:28:51**), **closed on
|
|
2026-11-05 at 12:36:17** - together with its 11-04 successor row `dca848_3b1bb1e9b003`.
|
|
- The job behind it, `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), **completed** its 2026-11-05
|
|
07:00 run in **5 h 35 m 35 s** (07:00:42 -> 12:36:18), with no repeat of
|
|
`RuntimeError: Upstream stream failed before completion.`
|
|
- **Status -> `resolved`.** The underlying questions (whether the 65 m 21 s stream silence is the watchdog
|
|
threshold or a silently dead socket; whether a stream timeout on a file-deliverable job should retry
|
|
automatically) remain open in [[systems/scheduled-tasks]] - they are no longer tracked as an incident
|
|
against this window.
|
|
|
|
## Related
|
|
|
|
- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]
|
|
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
|
|
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
|
|
- [[runbooks/job-radar]]
|
|
- [[systems/scheduled-tasks]]
|