wiki: nightly audit 2026-11-03 window — new Job Radar stream-timeout incident, hermes-hq-push system page; update 11-02 recurrence, scheduled-tasks, index, log
This commit is contained in:
@@ -0,0 +1,118 @@
|
||||
---
|
||||
title: Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-11-04
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
review_after: 2026-11-11
|
||||
confidence: high
|
||||
tags: [cron, job-radar, stream-timeout, silent-failure, notion, context-compaction]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-deliveries-db, live-jobs-json, live-state-db, live-gateway-log, live-radar-handoff-dir, session-history]
|
||||
---
|
||||
|
||||
# Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
|
||||
|
||||
## Headline
|
||||
|
||||
**The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the
|
||||
ledger went red.** On **2026-11-03** the 07:00 `dca8482e4f76` (*Daily Marketing Job Radar — Stage 1
|
||||
(Discovery)*) ran **6 h 28 m 18 s** (`started 07:00:33.819190` → `finished 13:28:52.305995 +08:00`) and
|
||||
closed:
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| `status` | **`failed`** |
|
||||
| `error` | **`RuntimeError: stream active timeout`** |
|
||||
| `delivery_outcome` | **`delivered`** (the failure notice, `for_failure: 1`) |
|
||||
| `cron_incidents` row | **`dca848_dfd4bea84f2a`**, `failure_type: timeout`, **`alerted`**, still open |
|
||||
|
||||
No handoff artefact was written, so **nothing reached Notion, no cover letters were generated, and both
|
||||
Stage 2 slots declined.** This is the **fourth Stage-1 failure in eight days** and a **new terminal
|
||||
signature** — not the iteration cap that ended 10-27, 10-28 and 11-02.
|
||||
|
||||
## Timeline (all `+08:00`)
|
||||
|
||||
| Time | Event |
|
||||
|---|---|
|
||||
| 07:00:33.819190 | execution `dca8482e4f76` opens (07:00 weekday slot) |
|
||||
| 07:01:17 | run session `local-7180cef7…` first message |
|
||||
| 08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21 | **five context compactions** in one run |
|
||||
| 11:30:36.617545 → 11:32:16.812897 | Stage 2 `5fc292ff0f3b` first slot: no handoff → `[SILENT]`, `suppressed` |
|
||||
| 12:23:28 | **last tool result of the run** (a `terminal` call) |
|
||||
| 13:28:49 | harness records *"This turn did not complete. Some actions may already have run; verify their effects before resending."* |
|
||||
| 13:28:50,627 | gateway.log: `ERROR gateway.session_authority: Admitted turn 9bf9a296… failed` (traceback through `session_authority.py:1159 _execute_claimed` → `session_finite.py:80 execute_finite_admission` → `session_ingress.py:235 execute_admission`) |
|
||||
| 13:28:51.463929 | incident `dca848_dfd4bea84f2a` opens |
|
||||
| 13:28:51.741724 | **failure notice delivered** |
|
||||
| 13:28:52.305995 | execution closes `failed` |
|
||||
| 13:28:55.141618 | incident `alerted` |
|
||||
| 14:30:32.261931 → 14:33:11.470879 | Stage 2 safety-net slot: same silent decline |
|
||||
| in-session | Pulse report `p34c8e899-1` (covering Tue 03 Nov 13:01 onward) led with it: *"The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a stream active timeout."* |
|
||||
|
||||
## The 65-minute silence is the new information
|
||||
|
||||
The run's recorded activity stops at **12:23:28** and the harness does not declare the turn dead until
|
||||
**13:28:49** — **65 m 21 s with no tool call, no model turn and no error written to the ledger**.
|
||||
|
||||
Contrast with the two cap failures: on **10-28** and **11-02** the runtime *injected* the text *"You've
|
||||
reached the maximum number of tool-calling iterations allowed…"* into the session before it ended. This
|
||||
session contains **no such message** — it stopped mid-verification, on a provider-stream liveness timeout,
|
||||
with the run's own closing plan (*"Two final verification passes, then I'll assemble the handoff"*) never
|
||||
executed.
|
||||
|
||||
**Therefore:** the agent iteration cap is not the only ceiling that ends a long Stage-1 run, and the
|
||||
"2026-11-02 raised no incident at all" behaviour was a property of *that* terminal state (`completed`,
|
||||
`error: NULL`), **not** of long runs in general. Nothing in the ledger or the gateway log records the
|
||||
provider's socket state during the silent window.
|
||||
|
||||
## Why the ledger went red this time
|
||||
|
||||
| Date | `executions.status` | `cron_incidents` row | Notice to Anthony |
|
||||
|---|---|---|---|
|
||||
| 2026-10-27 | non-clean (`failed`) | `dca848_e93a538ade6b` — `failure_type: auth` (**wrong** for a budget fault) | yes — the run's INCOMPLETE report |
|
||||
| 2026-10-28 | non-clean (`failed`) | `dca848_0a2f03b5952b` — `failure_type: timeout` | yes |
|
||||
| 2026-11-02 | **`completed`, `error: NULL`** | **none raised** | **no — `suppressed`** |
|
||||
| **2026-11-03** | **`failed`** | **`dca848_dfd4bea84f2a` — `timeout`, open** | **yes — 13:28:51.741724** |
|
||||
|
||||
Fleet-wide **unclosed incident rows 0 → 1**. A run the harness itself kills produces an honest red row and
|
||||
a delivered notice; the cap produces a green row and silence. That is the distinction
|
||||
[[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] could only predict, now observed.
|
||||
|
||||
## Downstream: both Stage 2 slots behaved correctly, and two days of candidates are lost
|
||||
|
||||
| Slot | Job | Ran | Result |
|
||||
|---|---|---|---|
|
||||
| 11:30 | `5fc292ff0f3b` (*Write and Letters*) | 11:30:36 → 11:32:16 | `[SILENT]` / `suppressed` |
|
||||
| 14:30 (safety net) | `5fc292ff0f3b` | 14:30:32 → 14:33:11 | `[SILENT]` / `suppressed` |
|
||||
|
||||
`~/.hermes/radar-ref/handoff/` holds **only** `2026-10-27.json`, `2026-10-29.json` and `2026-10-30.json` —
|
||||
nothing for 10-31, **11-02** or **11-03**. So **2026-11-02's 164 deduped candidates and 2026-11-03's
|
||||
discovery are both unrecovered**, and Notion received nothing on either day. The last end-to-end success
|
||||
remains **2026-10-29** (handoff written inside budget; Stage 2 landed **20 pages + 2 cover letters**,
|
||||
`missing_cover_letter_count 0`).
|
||||
|
||||
## Context churn (observation, cause not established)
|
||||
|
||||
**363 messages** (6 `user` / 133 `assistant` / 224 `tool`), **288 compaction-flagged messages**, and **five
|
||||
compactions** inside one 6 h run. The 11-02 run compacted repeatedly as well. Whether compaction churn
|
||||
contributes to the stream stall is **not established** — this is recorded because both long failures share
|
||||
the pattern, not because the link is proven.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Is the **65 m 21 s** silence the stream-active watchdog's own threshold, or did the provider socket die
|
||||
silently long before the harness noticed?
|
||||
2. Does repeated context compaction contribute to the stall? Two long runs, both compacting heavily.
|
||||
3. The 07:00 run has now overrun past the 11:30 Stage-2 slot two days running — should the pairing be
|
||||
re-slotted, or Stage 2 made to wait on a handoff?
|
||||
4. Where does `radar-ref/handoff/` get its retention? The newest file is 10-30.
|
||||
5. Should a stream timeout on a *file-deliverable* job trigger an automatic retry instead of leaving the
|
||||
day's discovery to the next slot?
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]
|
||||
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
|
||||
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
|
||||
- [[runbooks/job-radar]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
Reference in New Issue
Block a user