Nightly audit 2026-11-03 (window 2026-11-02): new incident — Job Radar Stage 1 iteration-cap recurrence (9h 01m, suppressed, no incident raised); Weekly Review incident resolved (reconciled 01:27 AWST); scheduled-tasks + browser-backend re-verified; index/log updated

This commit is contained in:
Tony0410
2026-11-03 00:22:13 +08:00
parent fda56e53fe
commit ca014efd63
8 changed files with 516 additions and 20 deletions
@@ -1,17 +1,22 @@
---
title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open
title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Recovered and Delivered 7 h 26 m Later
type: incident
status: active
status: resolved
created: 2026-11-02
updated: 2026-11-02
verified_on: 2026-11-02
review_after: 2026-11-09
updated: 2026-11-03
verified_on: 2026-11-03
review_after: 2026-11-17
confidence: high
tags: [cron, gateway, handoff, delivery, weekly-review, silent-failure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-jobs-json, live-logs, live-probe]
---
# Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open
# Weekly Review Lost at 18:00 on 2026-11-01 — Recovered and Delivered 7 h 26 m Later
> **Status: RESOLVED 2026-11-02T01:27:21+08:00.** The Sunday 18:00 run did *not* lose the review — it
> resumed, completed and delivered at **01:27:21**, and the incident row auto-closed. The body below is
> the record **as it stood at the 2026-11-02 audit**, before the recovery landed. See
> [Resolution](#resolution-2026-11-02) for the corrected outcome and the answers to the open questions.
## Headline
@@ -79,6 +84,8 @@ appearance of the "green/red row, no artefact" anti-pattern the wiki already doc
## Open questions
> ⚠ **Questions 1–3 are answered in [Resolution](#resolution-2026-11-02).** Question 4 remains open.
1. **What did "Gateway connection failed; no local fallback" mean?** Same message for two different
jobs 11 ms apart, both in the same handoff batch, with no gateway restart anywhere in the window.
2. **Why did `b63ed5` self-heal in 68 s and `79f9bb` never do so?** Both were in the same batch with the
@@ -88,6 +95,47 @@ appearance of the "green/red row, no artefact" anti-pattern the wiki already doc
4. Are the two unreconciled admission files (`141448f9…`, `69b5588f…`) the durable record needed to
decide whether either run had side effects? Reconcile before the next 18:00 Sunday slot.
## Resolution (2026-11-02)
**The run was not lost. It resumed, finished, and delivered — 7 h 26 m 43 s after it started.**
| Field | Value |
|---|---|
| Execution | `be6ec11a8e68428084af9ab18fb148db` — **the same execution record** the audit saw stuck at `unknown` |
| `scheduled_instant` | `2026-11-01T10:00:00+00:00` (18:00 AWST) |
| `started_at` | `2026-11-01T18:00:39.125776+08:00` |
| `finished_at` | **`2026-11-02T01:27:22.000881+08:00`** |
| `status` | **`completed`** |
| `error` | `NULL` |
| `delivery_outcome` | **`delivered`** |
| Delivery row | `2026-11-02T01:27:21.802246+08:00` — `delivered` |
| Incident row | `79f9bb_9c443b602eec` → `state: resolved`, `closed_at: 2026-11-02T01:27:21.606691+08:00` |
Anthony received the Weekly Review at 01:27 AWST on 2026-11-02. There is still exactly **one** execution
row for the 18:00:39 slot, so this is a reconciliation of the existing claim rather than a re-run — the
`unknown` terminal state at 18:05:13 was replaced when the run finally wrote its real outcome.
**Answers to the open questions:**
1. *(the "no local fallback" message)* — the handoff to the external worker did ultimately succeed; the
18:05:13 message recorded a **failed reconciliation attempt**, not a failed run. The run kept executing
for a further 7 h 26 m and produced its deliverable. The message is therefore a **false negative from
the reconciler**, not evidence that the work was abandoned.
2. **`b63ed5` closed in 68 s; `79f9bb` closed in 7 h 26 m — both eventually reached `completed`/`delivered`.**
The difference was run duration, not failure mode. The reconciler's window is shorter than the Weekly
Review's runtime, so any long-running job in a handoff batch will be reported `unknown` while it is
still working. **This is the general lesson: an `unknown` execution state on a long job is a timing
artefact, not a verdict.**
3. **No manual re-run was needed** — the run self-healed and delivered 12 minutes before the 2026-11-02
audit would have recommended one. Had the audit re-triggered the job, it would have duplicated the run.
### Consequences for the audit method
The 2026-11-02 audit read `status: unknown` + `closed_at: NULL` at 00:4x and concluded the deliverable was
lost. Both were true at the time of reading and both had inverted by 01:27. **A nightly audit that runs
~2 h after midnight will read every long-running job mid-flight and must not treat a null terminal state as
final until the job's expected runtime has elapsed** — see [[systems/scheduled-tasks]] §9.
## Related
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]]
@@ -0,0 +1,118 @@
---
title: Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded `completed`, Output Suppressed
type: incident
status: active
created: 2026-11-03
updated: 2026-11-03
verified_on: 2026-11-03
review_after: 2026-11-10
confidence: high
tags: [cron, job-radar, iteration-cap, silent-failure, notion, camofox]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-state-db, live-jobs-json, session-history]
---
# Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded `completed`, Output Suppressed
## Headline
**Third occurrence of the Stage-1 budget class, and the first that the cron ledger reports as healthy.**
On **2026-11-02** the 07:00 `dca8482e4f76` (*Daily Marketing Job Radar — Stage 1 (Discovery)*) ran for
**9 h 01 m 19 s** (`started 07:00:33.378239` → `finished 16:01:52.496335 +08:00`), consumed the harness's
entire tool-calling iteration budget, and closed with:
| Field | Value |
|---|---|
| `status` | **`completed`** |
| `error` | `NULL` |
| `delivery_outcome` | **`suppressed`** |
No handoff artefact was written. **Nothing reached Notion, no cover letters were generated, and Stage 2 was
never triggered.** Because the run's own status is `completed` with no error, a health check that inspects
`executions.status` alone reports this day as green — the failure is invisible in the ledger.
## What the run did and did not do
The session (`Daily Marketing Job Radar - Discovery · Nov 02 16:01`, `local-368c8fe9…`) shows the discovery
half genuinely worked:
- Camofox was restarted at run start after its HTTP server wedged (see [[systems/browser-backend]]).
- **SEEK 51 unique roles, LinkedIn 60 unique job IDs, Indeed live, WA Government 35 results** — 197 raw
candidates, **164 new after three-level dedup** against the 479-row Notion board.
- A **URL-canonicalisation bug was found and fixed mid-run**: `au.seek.com` vs `au.seek.com.au` had been
dropping **SEEK and Indeed entries** from the consolidated list; the fix was a job-ID-based
canonicalisation.
- Adverts were individually verified fresh and active; salaries, closing dates and JD bodies were pulled.
Then the ceiling hit. The runtime injected *"You've reached the maximum number of tool-calling iterations
allowed. Please provide a final response…"* while the agent was still reading JD text for the borderline
finalists, and the run terminated in the **scoring** phase.
The agent's own closing report was explicit — and it was **not delivered**:
> *"**Status: INCOMPLETE — handoff file NOT written, Stage 2 NOT triggered.** … Because the mandatory
> deliverable (the handoff JSON) was not produced, today's roles have **not** been handed to Stage 2."*
That text is `delivery_outcome: suppressed`, so Anthony saw nothing on 11-02 — not the roles and not the
warning that the roles were missing.
## Downstream: both Stage 2 slots behaved correctly
| Slot | Session | Ran | Result |
|---|---|---|---|
| 11:30 | `local-92665337…` (*Write and Letters*) | 11:30 → 12:25 | Exited `[SILENT]` after verifying Stage 1 was alive but had no handoff |
| 14:30 (safety net) | `local-05a3f00b…` (*Write and Letters*) | 14:30 → 14:30 | Exited `[SILENT]` — 6 messages, 2 tool calls |
Both declines were correct: no handoff existed to consume. The safety-net design worked as intended — the
gap is that Stage 1 cannot finish inside its budget.
## Recurrence pattern
| Date | Incident page | Failure shape |
|---|---|---|
| 2026-10-27 | [[incidents/2026-10-27-job-radar-budget-exhausted]] | Tool-call budget exhausted before the Notion write stage |
| 2026-10-28 | [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] | Stage 1 hit the agent iteration cap before the handoff |
| **2026-11-02** | **this page** | Same cap; **run recorded `completed` and its report suppressed** |
Three hits in seven days, each ending in the same place: discovery complete, handoff absent, Notion write
stage never reached.
## The "green row, no artefact" anti-pattern
This is the **fourth appearance** of the class [[systems/scheduled-tasks]] §9 of the maintenance playbook
already documents (a job whose durable ledger entry says success while the thing it was supposed to produce
does not exist). The new variant here is that the ledger flipped from red to green:
| Date | `cron_incidents` row for `dca8482e4f76` | `executions.status` |
|---|---|---|
| 2026-10-27 | `dca848_e93a538ade6b` (`failure_type: auth`) | non-clean |
| 2026-10-28 | `dca848_0a2f03b5952b` (`failure_type: timeout`) | non-clean |
| **2026-11-02** | **none raised** | **`completed`, `error: NULL`** |
The two earlier caps each raised an incident whose `error` text is literally the run's INCOMPLETE report.
The 2026-11-02 cap raised **nothing at all** (`cron_incidents` holds no row for `dca8482e4f76` with a
`first_seen_at` on 2026-11-02; fleet-wide unclosed rows were **0** at the 2026-11-03 audit). The run's
`delivery_outcome` is `suppressed`, so no message reached Anthony either.
**Detection rule:** for any job whose deliverable is a *file* (a handoff JSON, a report, a generated page),
`executions.status` is not sufficient evidence. Verify the artefact exists at its expected path, or that
`cron_incidents` stayed quiet for a reason, before treating the slot as served. Silence from
`cron_incidents` is not the same as success — here it is the *symptom*.
## Open questions
1. **Is the iteration budget the right control, or is the task too large for one agent turn?** Three caps in
a row suggest Stage 1's work (four job boards → dedup → individual advert verification → JD reading →
scoring) does not fit a single turn at any budget.
2. **Should the harness record a cap-hit as something other than `completed`?** A distinct terminal status
(or a non-null `error`) would make this visible to every existing health check.
3. **Should the run's final response be delivered even when the job "succeeded"?** Suppressing an INCOMPLETE
report converts a loud failure into silence.
4. **Did the 11-02 roles get picked up on the next day's run**, or is 2026-11-02's candidate set lost?
The 164 deduped candidates were never persisted.
## Related
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
- [[systems/scheduled-tasks]]
- [[systems/browser-backend]]