wiki: nightly audit 2026-11-03 window — new Job Radar stream-timeout incident, hermes-hq-push system page; update 11-02 recurrence, scheduled-tasks, index, log

This commit is contained in:
Hermes Wiki Audit
2026-11-04 00:16:23 +08:00
parent ca014efd63
commit 785fbc40b2
8 changed files with 534 additions and 13 deletions
@@ -3,11 +3,11 @@ title: Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete
type: incident
status: active
created: 2026-11-03
updated: 2026-11-03
verified_on: 2026-11-03
review_after: 2026-11-10
updated: 2026-11-04
verified_on: 2026-11-04
review_after: 2026-11-17
confidence: high
tags: [cron, job-radar, iteration-cap, silent-failure, notion, camofox]
tags: [cron, job-radar, iteration-cap, silent-failure, notion, camofox, stream-timeout]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-state-db, live-jobs-json, session-history]
---
@@ -72,9 +72,11 @@ gap is that Stage 1 cannot finish inside its budget.
| 2026-10-27 | [[incidents/2026-10-27-job-radar-budget-exhausted]] | Tool-call budget exhausted before the Notion write stage |
| 2026-10-28 | [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] | Stage 1 hit the agent iteration cap before the handoff |
| **2026-11-02** | **this page** | Same cap; **run recorded `completed` and its report suppressed** |
| 2026-11-03 | [[incidents/2026-11-03-job-radar-stream-active-timeout]] | **A different ceiling** — `RuntimeError: stream active timeout` after 6 h 28 m; `failed`, notice delivered |
Three hits in seven days, each ending in the same place: discovery complete, handoff absent, Notion write
stage never reached.
stage never reached. **A fourth run — 2026-11-03 — ended the same way (no handoff, both Stage 2 slots declined) but on a
different fault, and this time the ledger did go red; see the 2026-11-04 update below.**
## The "green row, no artefact" anti-pattern
@@ -98,6 +100,32 @@ The 2026-11-02 cap raised **nothing at all** (`cron_incidents` holds no row for
`cron_incidents` stayed quiet for a reason, before treating the slot as served. Silence from
`cron_incidents` is not the same as success — here it is the *symptom*.
## 2026-11-04 update — the missing incident row was a property of the cap, not of long runs
**Job Radar Stage 1 lost a second consecutive day on 2026-11-03 — and this time the harness killed the
turn, so the ledger went red.** The 07:00 `dca8482e4f76` ran **07:00:33.819190 → 13:28:52.305995
(6 h 28 m 18 s)** and closed **`status: failed` / `error: RuntimeError: stream active timeout` /
`delivery_outcome: delivered`**, raising incident **`dca848_dfd4bea84f2a`** (`failure_type: timeout`,
**`alerted` 13:28:55.141618, still open**) — fleet unclosed rows **0 → 1**.
→ [[incidents/2026-11-03-job-radar-stream-active-timeout]]
**This answers open question 3 below (partly): suppression tracked the terminal state, not the run length.**
When the run ends `completed` — as the cap produces — its report is suppressed; when the harness itself
kills the turn, the terminal state is `failed` and both the incident row and the notice appear. 11-02:
nothing delivered. 11-03: the failure notice went out at **13:28:51.741724** and Pulse surfaced it to
Anthony in the desktop session `20261031_190813_95fbde` (report `p34c8e899-1`, covering Tue 03 Nov 13:01
onward): *"The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a
stream active timeout."*
**It also narrows open question 2 below:** the ledger *can* record a non-`completed` state when a long run
dies — the iteration cap is the specific case that still escapes as `completed`.
**And it partially answers open question 4: the candidate set was lost again.** Both Stage 2 slots declined
silently (`5fc292ff0f3b` 11:30:36 → 11:32:16 and 14:30:32 → 14:33:11) and
`~/.hermes/radar-ref/handoff/` still holds **only** `2026-10-27.json`, `2026-10-29.json` and
`2026-10-30.json` — so **2026-11-02's 164 deduped candidates and 2026-11-03's discovery are both
unrecovered**, two days running.
## Open questions
1. **Is the iteration budget the right control, or is the task too large for one agent turn?** Three caps in
@@ -112,6 +140,7 @@ The 2026-11-02 cap raised **nothing at all** (`cron_incidents` holds no row for
## Related
- [[incidents/2026-11-03-job-radar-stream-active-timeout]]
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
- [[systems/scheduled-tasks]]
@@ -0,0 +1,118 @@
---
title: Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
type: incident
status: open
created: 2026-11-04
updated: 2026-11-04
verified_on: 2026-11-04
review_after: 2026-11-11
confidence: high
tags: [cron, job-radar, stream-timeout, silent-failure, notion, context-compaction]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-deliveries-db, live-jobs-json, live-state-db, live-gateway-log, live-radar-handoff-dir, session-history]
---
# Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
## Headline
**The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the
ledger went red.** On **2026-11-03** the 07:00 `dca8482e4f76` (*Daily Marketing Job Radar — Stage 1
(Discovery)*) ran **6 h 28 m 18 s** (`started 07:00:33.819190` → `finished 13:28:52.305995 +08:00`) and
closed:
| Field | Value |
|---|---|
| `status` | **`failed`** |
| `error` | **`RuntimeError: stream active timeout`** |
| `delivery_outcome` | **`delivered`** (the failure notice, `for_failure: 1`) |
| `cron_incidents` row | **`dca848_dfd4bea84f2a`**, `failure_type: timeout`, **`alerted`**, still open |
No handoff artefact was written, so **nothing reached Notion, no cover letters were generated, and both
Stage 2 slots declined.** This is the **fourth Stage-1 failure in eight days** and a **new terminal
signature** — not the iteration cap that ended 10-27, 10-28 and 11-02.
## Timeline (all `+08:00`)
| Time | Event |
|---|---|
| 07:00:33.819190 | execution `dca8482e4f76` opens (07:00 weekday slot) |
| 07:01:17 | run session `local-7180cef7…` first message |
| 08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21 | **five context compactions** in one run |
| 11:30:36.617545 → 11:32:16.812897 | Stage 2 `5fc292ff0f3b` first slot: no handoff → `[SILENT]`, `suppressed` |
| 12:23:28 | **last tool result of the run** (a `terminal` call) |
| 13:28:49 | harness records *"This turn did not complete. Some actions may already have run; verify their effects before resending."* |
| 13:28:50,627 | gateway.log: `ERROR gateway.session_authority: Admitted turn 9bf9a296… failed` (traceback through `session_authority.py:1159 _execute_claimed` → `session_finite.py:80 execute_finite_admission` → `session_ingress.py:235 execute_admission`) |
| 13:28:51.463929 | incident `dca848_dfd4bea84f2a` opens |
| 13:28:51.741724 | **failure notice delivered** |
| 13:28:52.305995 | execution closes `failed` |
| 13:28:55.141618 | incident `alerted` |
| 14:30:32.261931 → 14:33:11.470879 | Stage 2 safety-net slot: same silent decline |
| in-session | Pulse report `p34c8e899-1` (covering Tue 03 Nov 13:01 onward) led with it: *"The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a stream active timeout."* |
## The 65-minute silence is the new information
The run's recorded activity stops at **12:23:28** and the harness does not declare the turn dead until
**13:28:49** — **65 m 21 s with no tool call, no model turn and no error written to the ledger**.
Contrast with the two cap failures: on **10-28** and **11-02** the runtime *injected* the text *"You've
reached the maximum number of tool-calling iterations allowed…"* into the session before it ended. This
session contains **no such message** — it stopped mid-verification, on a provider-stream liveness timeout,
with the run's own closing plan (*"Two final verification passes, then I'll assemble the handoff"*) never
executed.
**Therefore:** the agent iteration cap is not the only ceiling that ends a long Stage-1 run, and the
"2026-11-02 raised no incident at all" behaviour was a property of *that* terminal state (`completed`,
`error: NULL`), **not** of long runs in general. Nothing in the ledger or the gateway log records the
provider's socket state during the silent window.
## Why the ledger went red this time
| Date | `executions.status` | `cron_incidents` row | Notice to Anthony |
|---|---|---|---|
| 2026-10-27 | non-clean (`failed`) | `dca848_e93a538ade6b` — `failure_type: auth` (**wrong** for a budget fault) | yes — the run's INCOMPLETE report |
| 2026-10-28 | non-clean (`failed`) | `dca848_0a2f03b5952b` — `failure_type: timeout` | yes |
| 2026-11-02 | **`completed`, `error: NULL`** | **none raised** | **no — `suppressed`** |
| **2026-11-03** | **`failed`** | **`dca848_dfd4bea84f2a` — `timeout`, open** | **yes — 13:28:51.741724** |
Fleet-wide **unclosed incident rows 0 → 1**. A run the harness itself kills produces an honest red row and
a delivered notice; the cap produces a green row and silence. That is the distinction
[[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] could only predict, now observed.
## Downstream: both Stage 2 slots behaved correctly, and two days of candidates are lost
| Slot | Job | Ran | Result |
|---|---|---|---|
| 11:30 | `5fc292ff0f3b` (*Write and Letters*) | 11:30:36 → 11:32:16 | `[SILENT]` / `suppressed` |
| 14:30 (safety net) | `5fc292ff0f3b` | 14:30:32 → 14:33:11 | `[SILENT]` / `suppressed` |
`~/.hermes/radar-ref/handoff/` holds **only** `2026-10-27.json`, `2026-10-29.json` and `2026-10-30.json` —
nothing for 10-31, **11-02** or **11-03**. So **2026-11-02's 164 deduped candidates and 2026-11-03's
discovery are both unrecovered**, and Notion received nothing on either day. The last end-to-end success
remains **2026-10-29** (handoff written inside budget; Stage 2 landed **20 pages + 2 cover letters**,
`missing_cover_letter_count 0`).
## Context churn (observation, cause not established)
**363 messages** (6 `user` / 133 `assistant` / 224 `tool`), **288 compaction-flagged messages**, and **five
compactions** inside one 6 h run. The 11-02 run compacted repeatedly as well. Whether compaction churn
contributes to the stream stall is **not established** — this is recorded because both long failures share
the pattern, not because the link is proven.
## Open questions
1. Is the **65 m 21 s** silence the stream-active watchdog's own threshold, or did the provider socket die
silently long before the harness noticed?
2. Does repeated context compaction contribute to the stall? Two long runs, both compacting heavily.
3. The 07:00 run has now overrun past the 11:30 Stage-2 slot two days running — should the pairing be
re-slotted, or Stage 2 made to wait on a handoff?
4. Where does `radar-ref/handoff/` get its retention? The newest file is 10-30.
5. Should a stream timeout on a *file-deliverable* job trigger an automatic retry instead of leaving the
day's discovery to the next slot?
## Related
- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
- [[runbooks/job-radar]]
- [[systems/scheduled-tasks]]