Nightly audit 2026-11-03 (window 2026-11-02): new incident — Job Radar Stage 1 iteration-cap recurrence (9h 01m, suppressed, no incident raised); Weekly Review incident resolved (reconciled 01:27 AWST); scheduled-tasks + browser-backend re-verified; index/log updated
This commit is contained in:
@@ -1,17 +1,22 @@
|
||||
---
|
||||
title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open
|
||||
title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Recovered and Delivered 7 h 26 m Later
|
||||
type: incident
|
||||
status: active
|
||||
status: resolved
|
||||
created: 2026-11-02
|
||||
updated: 2026-11-02
|
||||
verified_on: 2026-11-02
|
||||
review_after: 2026-11-09
|
||||
updated: 2026-11-03
|
||||
verified_on: 2026-11-03
|
||||
review_after: 2026-11-17
|
||||
confidence: high
|
||||
tags: [cron, gateway, handoff, delivery, weekly-review, silent-failure]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-jobs-json, live-logs, live-probe]
|
||||
---
|
||||
|
||||
# Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open
|
||||
# Weekly Review Lost at 18:00 on 2026-11-01 — Recovered and Delivered 7 h 26 m Later
|
||||
|
||||
> **Status: RESOLVED 2026-11-02T01:27:21+08:00.** The Sunday 18:00 run did *not* lose the review — it
|
||||
> resumed, completed and delivered at **01:27:21**, and the incident row auto-closed. The body below is
|
||||
> the record **as it stood at the 2026-11-02 audit**, before the recovery landed. See
|
||||
> [Resolution](#resolution-2026-11-02) for the corrected outcome and the answers to the open questions.
|
||||
|
||||
## Headline
|
||||
|
||||
@@ -79,6 +84,8 @@ appearance of the "green/red row, no artefact" anti-pattern the wiki already doc
|
||||
|
||||
## Open questions
|
||||
|
||||
> ⚠ **Questions 1–3 are answered in [Resolution](#resolution-2026-11-02).** Question 4 remains open.
|
||||
|
||||
1. **What did "Gateway connection failed; no local fallback" mean?** Same message for two different
|
||||
jobs 11 ms apart, both in the same handoff batch, with no gateway restart anywhere in the window.
|
||||
2. **Why did `b63ed5` self-heal in 68 s and `79f9bb` never do so?** Both were in the same batch with the
|
||||
@@ -88,6 +95,47 @@ appearance of the "green/red row, no artefact" anti-pattern the wiki already doc
|
||||
4. Are the two unreconciled admission files (`141448f9…`, `69b5588f…`) the durable record needed to
|
||||
decide whether either run had side effects? Reconcile before the next 18:00 Sunday slot.
|
||||
|
||||
## Resolution (2026-11-02)
|
||||
|
||||
**The run was not lost. It resumed, finished, and delivered — 7 h 26 m 43 s after it started.**
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Execution | `be6ec11a8e68428084af9ab18fb148db` — **the same execution record** the audit saw stuck at `unknown` |
|
||||
| `scheduled_instant` | `2026-11-01T10:00:00+00:00` (18:00 AWST) |
|
||||
| `started_at` | `2026-11-01T18:00:39.125776+08:00` |
|
||||
| `finished_at` | **`2026-11-02T01:27:22.000881+08:00`** |
|
||||
| `status` | **`completed`** |
|
||||
| `error` | `NULL` |
|
||||
| `delivery_outcome` | **`delivered`** |
|
||||
| Delivery row | `2026-11-02T01:27:21.802246+08:00` — `delivered` |
|
||||
| Incident row | `79f9bb_9c443b602eec` → `state: resolved`, `closed_at: 2026-11-02T01:27:21.606691+08:00` |
|
||||
|
||||
Anthony received the Weekly Review at 01:27 AWST on 2026-11-02. There is still exactly **one** execution
|
||||
row for the 18:00:39 slot, so this is a reconciliation of the existing claim rather than a re-run — the
|
||||
`unknown` terminal state at 18:05:13 was replaced when the run finally wrote its real outcome.
|
||||
|
||||
**Answers to the open questions:**
|
||||
|
||||
1. *(the "no local fallback" message)* — the handoff to the external worker did ultimately succeed; the
|
||||
18:05:13 message recorded a **failed reconciliation attempt**, not a failed run. The run kept executing
|
||||
for a further 7 h 26 m and produced its deliverable. The message is therefore a **false negative from
|
||||
the reconciler**, not evidence that the work was abandoned.
|
||||
2. **`b63ed5` closed in 68 s; `79f9bb` closed in 7 h 26 m — both eventually reached `completed`/`delivered`.**
|
||||
The difference was run duration, not failure mode. The reconciler's window is shorter than the Weekly
|
||||
Review's runtime, so any long-running job in a handoff batch will be reported `unknown` while it is
|
||||
still working. **This is the general lesson: an `unknown` execution state on a long job is a timing
|
||||
artefact, not a verdict.**
|
||||
3. **No manual re-run was needed** — the run self-healed and delivered 12 minutes before the 2026-11-02
|
||||
audit would have recommended one. Had the audit re-triggered the job, it would have duplicated the run.
|
||||
|
||||
### Consequences for the audit method
|
||||
|
||||
The 2026-11-02 audit read `status: unknown` + `closed_at: NULL` at 00:4x and concluded the deliverable was
|
||||
lost. Both were true at the time of reading and both had inverted by 01:27. **A nightly audit that runs
|
||||
~2 h after midnight will read every long-running job mid-flight and must not treat a null terminal state as
|
||||
final until the job's expected runtime has elapsed** — see [[systems/scheduled-tasks]] §9.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]]
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
---
|
||||
title: Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded `completed`, Output Suppressed
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-03
|
||||
updated: 2026-11-03
|
||||
verified_on: 2026-11-03
|
||||
review_after: 2026-11-10
|
||||
confidence: high
|
||||
tags: [cron, job-radar, iteration-cap, silent-failure, notion, camofox]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-state-db, live-jobs-json, session-history]
|
||||
---
|
||||
|
||||
# Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded `completed`, Output Suppressed
|
||||
|
||||
## Headline
|
||||
|
||||
**Third occurrence of the Stage-1 budget class, and the first that the cron ledger reports as healthy.**
|
||||
On **2026-11-02** the 07:00 `dca8482e4f76` (*Daily Marketing Job Radar — Stage 1 (Discovery)*) ran for
|
||||
**9 h 01 m 19 s** (`started 07:00:33.378239` → `finished 16:01:52.496335 +08:00`), consumed the harness's
|
||||
entire tool-calling iteration budget, and closed with:
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| `status` | **`completed`** |
|
||||
| `error` | `NULL` |
|
||||
| `delivery_outcome` | **`suppressed`** |
|
||||
|
||||
No handoff artefact was written. **Nothing reached Notion, no cover letters were generated, and Stage 2 was
|
||||
never triggered.** Because the run's own status is `completed` with no error, a health check that inspects
|
||||
`executions.status` alone reports this day as green — the failure is invisible in the ledger.
|
||||
|
||||
## What the run did and did not do
|
||||
|
||||
The session (`Daily Marketing Job Radar - Discovery · Nov 02 16:01`, `local-368c8fe9…`) shows the discovery
|
||||
half genuinely worked:
|
||||
|
||||
- Camofox was restarted at run start after its HTTP server wedged (see [[systems/browser-backend]]).
|
||||
- **SEEK 51 unique roles, LinkedIn 60 unique job IDs, Indeed live, WA Government 35 results** — 197 raw
|
||||
candidates, **164 new after three-level dedup** against the 479-row Notion board.
|
||||
- A **URL-canonicalisation bug was found and fixed mid-run**: `au.seek.com` vs `au.seek.com.au` had been
|
||||
dropping **SEEK and Indeed entries** from the consolidated list; the fix was a job-ID-based
|
||||
canonicalisation.
|
||||
- Adverts were individually verified fresh and active; salaries, closing dates and JD bodies were pulled.
|
||||
|
||||
Then the ceiling hit. The runtime injected *"You've reached the maximum number of tool-calling iterations
|
||||
allowed. Please provide a final response…"* while the agent was still reading JD text for the borderline
|
||||
finalists, and the run terminated in the **scoring** phase.
|
||||
|
||||
The agent's own closing report was explicit — and it was **not delivered**:
|
||||
|
||||
> *"**Status: INCOMPLETE — handoff file NOT written, Stage 2 NOT triggered.** … Because the mandatory
|
||||
> deliverable (the handoff JSON) was not produced, today's roles have **not** been handed to Stage 2."*
|
||||
|
||||
That text is `delivery_outcome: suppressed`, so Anthony saw nothing on 11-02 — not the roles and not the
|
||||
warning that the roles were missing.
|
||||
|
||||
## Downstream: both Stage 2 slots behaved correctly
|
||||
|
||||
| Slot | Session | Ran | Result |
|
||||
|---|---|---|---|
|
||||
| 11:30 | `local-92665337…` (*Write and Letters*) | 11:30 → 12:25 | Exited `[SILENT]` after verifying Stage 1 was alive but had no handoff |
|
||||
| 14:30 (safety net) | `local-05a3f00b…` (*Write and Letters*) | 14:30 → 14:30 | Exited `[SILENT]` — 6 messages, 2 tool calls |
|
||||
|
||||
Both declines were correct: no handoff existed to consume. The safety-net design worked as intended — the
|
||||
gap is that Stage 1 cannot finish inside its budget.
|
||||
|
||||
## Recurrence pattern
|
||||
|
||||
| Date | Incident page | Failure shape |
|
||||
|---|---|---|
|
||||
| 2026-10-27 | [[incidents/2026-10-27-job-radar-budget-exhausted]] | Tool-call budget exhausted before the Notion write stage |
|
||||
| 2026-10-28 | [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] | Stage 1 hit the agent iteration cap before the handoff |
|
||||
| **2026-11-02** | **this page** | Same cap; **run recorded `completed` and its report suppressed** |
|
||||
|
||||
Three hits in seven days, each ending in the same place: discovery complete, handoff absent, Notion write
|
||||
stage never reached.
|
||||
|
||||
## The "green row, no artefact" anti-pattern
|
||||
|
||||
This is the **fourth appearance** of the class [[systems/scheduled-tasks]] §9 of the maintenance playbook
|
||||
already documents (a job whose durable ledger entry says success while the thing it was supposed to produce
|
||||
does not exist). The new variant here is that the ledger flipped from red to green:
|
||||
|
||||
| Date | `cron_incidents` row for `dca8482e4f76` | `executions.status` |
|
||||
|---|---|---|
|
||||
| 2026-10-27 | `dca848_e93a538ade6b` (`failure_type: auth`) | non-clean |
|
||||
| 2026-10-28 | `dca848_0a2f03b5952b` (`failure_type: timeout`) | non-clean |
|
||||
| **2026-11-02** | **none raised** | **`completed`, `error: NULL`** |
|
||||
|
||||
The two earlier caps each raised an incident whose `error` text is literally the run's INCOMPLETE report.
|
||||
The 2026-11-02 cap raised **nothing at all** (`cron_incidents` holds no row for `dca8482e4f76` with a
|
||||
`first_seen_at` on 2026-11-02; fleet-wide unclosed rows were **0** at the 2026-11-03 audit). The run's
|
||||
`delivery_outcome` is `suppressed`, so no message reached Anthony either.
|
||||
|
||||
**Detection rule:** for any job whose deliverable is a *file* (a handoff JSON, a report, a generated page),
|
||||
`executions.status` is not sufficient evidence. Verify the artefact exists at its expected path, or that
|
||||
`cron_incidents` stayed quiet for a reason, before treating the slot as served. Silence from
|
||||
`cron_incidents` is not the same as success — here it is the *symptom*.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Is the iteration budget the right control, or is the task too large for one agent turn?** Three caps in
|
||||
a row suggest Stage 1's work (four job boards → dedup → individual advert verification → JD reading →
|
||||
scoring) does not fit a single turn at any budget.
|
||||
2. **Should the harness record a cap-hit as something other than `completed`?** A distinct terminal status
|
||||
(or a non-null `error`) would make this visible to every existing health check.
|
||||
3. **Should the run's final response be delivered even when the job "succeeded"?** Suppressing an INCOMPLETE
|
||||
report converts a loud failure into silence.
|
||||
4. **Did the 11-02 roles get picked up on the next day's run**, or is 2026-11-02's candidate set lost?
|
||||
The 164 deduped candidates were never persisted.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
|
||||
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/browser-backend]]
|
||||
Reference in New Issue
Block a user