Files
agent-estate-wiki/incidents/2026-11-03-job-radar-stream-active-timeout.md
T

7.6 KiB

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff incident resolved 2026-11-04 2026-11-06 2026-11-06 2026-11-11 high
cron
job-radar
stream-timeout
silent-failure
notion
context-compaction
live-sqlite-cron-executions
live-sqlite-cron-incidents
live-deliveries-db
live-jobs-json
live-state-db
live-gateway-log
live-radar-handoff-dir
session-history

Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff

Headline

The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the ledger went red. On 2026-11-03 the 07:00 dca8482e4f76 (Daily Marketing Job Radar — Stage 1 (Discovery)) ran 6 h 28 m 18 s (started 07:00:33.819190 → finished 13:28:52.305995 +08:00) and closed:

Field Value
status failed
error RuntimeError: stream active timeout
delivery_outcome delivered (the failure notice, for_failure: 1)
cron_incidents row dca848_dfd4bea84f2a, failure_type: timeout, alerted, still open

No handoff artefact was written, so nothing reached Notion, no cover letters were generated, and both Stage 2 slots declined. This is the fourth Stage-1 failure in eight days and a new terminal signature — not the iteration cap that ended 10-27, 10-28 and 11-02.

Timeline (all +08:00)

Time Event
07:00:33.819190 execution dca8482e4f76 opens (07:00 weekday slot)
07:01:17 run session local-7180cef7… first message
08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21 five context compactions in one run
11:30:36.617545 → 11:32:16.812897 Stage 2 5fc292ff0f3b first slot: no handoff → [SILENT], suppressed
12:23:28 last tool result of the run (a terminal call)
13:28:49 harness records "This turn did not complete. Some actions may already have run; verify their effects before resending."
13:28:50,627 gateway.log: ERROR gateway.session_authority: Admitted turn 9bf9a296… failed (traceback through session_authority.py:1159 _execute_claimed → session_finite.py:80 execute_finite_admission → session_ingress.py:235 execute_admission)
13:28:51.463929 incident dca848_dfd4bea84f2a opens
13:28:51.741724 failure notice delivered
13:28:52.305995 execution closes failed
13:28:55.141618 incident alerted
14:30:32.261931 → 14:33:11.470879 Stage 2 safety-net slot: same silent decline
in-session Pulse report p34c8e899-1 (covering Tue 03 Nov 13:01 onward) led with it: "The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a stream active timeout."

The 65-minute silence is the new information

The run's recorded activity stops at 12:23:28 and the harness does not declare the turn dead until 13:28:49 — 65 m 21 s with no tool call, no model turn and no error written to the ledger.

Contrast with the two cap failures: on 10-28 and 11-02 the runtime injected the text "You've reached the maximum number of tool-calling iterations allowed…" into the session before it ended. This session contains no such message — it stopped mid-verification, on a provider-stream liveness timeout, with the run's own closing plan ("Two final verification passes, then I'll assemble the handoff") never executed.

Therefore: the agent iteration cap is not the only ceiling that ends a long Stage-1 run, and the "2026-11-02 raised no incident at all" behaviour was a property of that terminal state (completed, error: NULL), not of long runs in general. Nothing in the ledger or the gateway log records the provider's socket state during the silent window.

Why the ledger went red this time

Date executions.status cron_incidents row Notice to Anthony
2026-10-27 non-clean (failed) dca848_e93a538ade6b — failure_type: auth (wrong for a budget fault) yes — the run's INCOMPLETE report
2026-10-28 non-clean (failed) dca848_0a2f03b5952b — failure_type: timeout yes
2026-11-02 completed, error: NULL none raised no — suppressed
2026-11-03 failed dca848_dfd4bea84f2a — timeout, open yes — 13:28:51.741724

Fleet-wide unclosed incident rows 0 → 1. A run the harness itself kills produces an honest red row and a delivered notice; the cap produces a green row and silence. That is the distinction incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence could only predict, now observed.

Downstream: both Stage 2 slots behaved correctly, and two days of candidates are lost

Slot Job Ran Result
11:30 5fc292ff0f3b (Write and Letters) 11:30:36 → 11:32:16 [SILENT] / suppressed
14:30 (safety net) 5fc292ff0f3b 14:30:32 → 14:33:11 [SILENT] / suppressed

~/.hermes/radar-ref/handoff/ holds only 2026-10-27.json, 2026-10-29.json and 2026-10-30.json — nothing for 10-31, 11-02 or 11-03. So 2026-11-02's 164 deduped candidates and 2026-11-03's discovery are both unrecovered, and Notion received nothing on either day. The last end-to-end success remains 2026-10-29 (handoff written inside budget; Stage 2 landed 20 pages + 2 cover letters, missing_cover_letter_count 0).

Context churn (observation, cause not established)

363 messages (6 user / 133 assistant / 224 tool), 288 compaction-flagged messages, and five compactions inside one 6 h run. The 11-02 run compacted repeatedly as well. Whether compaction churn contributes to the stream stall is not established — this is recorded because both long failures share the pattern, not because the link is proven.

Open questions

  1. Is the 65 m 21 s silence the stream-active watchdog's own threshold, or did the provider socket die silently long before the harness noticed?
  2. Does repeated context compaction contribute to the stall? Two long runs, both compacting heavily.
  3. The 07:00 run has now overrun past the 11:30 Stage-2 slot two days running — should the pairing be re-slotted, or Stage 2 made to wait on a handoff?
  4. Where does radar-ref/handoff/ get its retention? The newest file is 10-30.
  5. Should a stream timeout on a file-deliverable job trigger an automatic retry instead of leaving the day's discovery to the next slot?

Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)

  • The last open row raised here, dca848_dfd4bea84f2a (timeout, opened 2026-11-03 13:28:51), closed on 2026-11-05 at 12:36:17 - together with its 11-04 successor row dca848_3b1bb1e9b003.
  • The job behind it, dca8482e4f76 (Daily Marketing Job Radar - Discovery), completed its 2026-11-05 07:00 run in 5 h 35 m 35 s (07:00:42 -> 12:36:18), with no repeat of RuntimeError: Upstream stream failed before completion.
  • Status -> resolved. The underlying questions (whether the 65 m 21 s stream silence is the watchdog threshold or a silently dead socket; whether a stream timeout on a file-deliverable job should retry automatically) remain open in systems/scheduled-tasks - they are no longer tracked as an incident against this window.