--- title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Recovered and Delivered 7 h 26 m Later type: incident status: resolved created: 2026-11-02 updated: 2026-11-03 verified_on: 2026-11-03 review_after: 2026-11-17 confidence: high tags: [cron, gateway, handoff, delivery, weekly-review, silent-failure] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-jobs-json, live-logs, live-probe] --- # Weekly Review Lost at 18:00 on 2026-11-01 — Recovered and Delivered 7 h 26 m Later > **Status: RESOLVED 2026-11-02T01:27:21+08:00.** The Sunday 18:00 run did *not* lose the review — it > resumed, completed and delivered at **01:27:21**, and the incident row auto-closed. The body below is > the record **as it stood at the 2026-11-02 audit**, before the recovery landed. See > [Resolution](#resolution-2026-11-02) for the corrected outcome and the answers to the open questions. ## Headline The **Sunday 18:00 Weekly Review (`79f9bb41ecf5`, `0 18 * * 0`) did not run and nothing was delivered on 2026-11-01.** At **18:05:13** its execution closed with `status: unknown`, `delivery_outcome: NULL` and the error *"Cron execution unverified; reconcile `…/cron/admissions/141448f9…json`: **Gateway connection failed; no local fallback**"*. Its incident row **`79f9bb_9c443b602eec` is still `alerted` at the time of this audit** (`first_seen` 18:05:13, `alerted_at` 18:05:32, `closed_at: NULL`) and the job reads `last_status: error`, **`failure_streak: 1`**. There is **no `deliveries.db` row for any slot after 11:01:12 on 11-01** — the last delivery of the day was the Tech & AI Digest at 11:01:12. ## Timeline (AWST, 2026-11-01) | Time | Event | |---|---| | 18:00:39 | **Four jobs handed to external workers in one batch**: `2fd50c87b0d0`, `96eb95ce12f2`, `b63ed5817212`, `79f9bb41ecf5` | | 18:00:57 / 18:01:04 | `96eb95ce12f2` and `2fd50c87b0d0` run fine (`completed`) | | **18:05:13.697679** | `b63ed5_6f45aeea97f0` opens — *"Cron execution unverified; reconcile `…/admissions/69b5588f…json`: Gateway connection failed; no local fallback"* (`alerted_at: None` — **never alerted**) | | **18:05:13.708153** | `79f9bb_9c443b602eec` opens with the **identical** message, pointing at `…/admissions/141448f9…json` | | 18:05:32 | `79f9bb` incident **alerted** | | 18:06:21 | `b63ed5817212` execution closes `completed` (18:00:39 → 18:06:21) → its incident auto-resolves **68 s later** | | — | `79f9bb41ecf5` closes at 18:05:13 with **no terminal state, no delivery, no recovery** | | 18:04:24 → 18:10 | `agent.log`: `[reaper] Active gateway work detected … deferring restart`, then *"Reaper: Plan A restart completed successfully (5m02s)"* — the **Plan A/B path is RSS compaction, not a process restart** (no `SIGTERM`/`SIGUSR1` and no relay-counter reset) | ## What actually reached the user **Nothing.** Across the whole of 11-01 `deliveries.db` holds 8 rows: | Time | Status | Note | |---|---|---| | 02:39:33 | delivered | Mnemosyne-era failure notice (closes the 10-26 and 10-31 rows) | | 03:00:37 | delivered | stocktake | | **07:18:50** | **failed** | `configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id` — job `066bb5191280` (see [[systems/messaging-integrations]]) | | 07:34:49 | delivered | — | | 08:36:51 | delivered | Gmail digest | | 10:44:12 | delivered | Facebook scrape | | 11:01:12 | delivered | Tech & AI Digest | | — | — | **no row after 11:01:12** | So of the day's eight delivery attempts, **seven reached their target and one (`066bb5191280`) landed in the DM thread-less while being recorded `failed`**. The Weekly Review's absence is not a suppressed delivery — it is an absent run. ## Second silent gap: a job that reported `error` with no execution row at all `ecc1fc13655a` (FreshRSS Midday News Briefing) reads **`last_status: error`, `failure_streak: 1`, `last_run_at: 2026-11-01T12:01:00.374089+08:00`** — yet **no `executions` row exists for it on 11-01** (the last is 2026-10-31T12:01:07, `completed`/`delivered`) and **no `cron_incidents` row was raised.** That 12:00 slot overlapped the same `18:00:39`-style handoff batch pattern (three jobs claimed at 12:00:53). This is the **second job of the day whose failure left no durable record**, and the third appearance of the "green/red row, no artefact" anti-pattern the wiki already documents ([[systems/scheduled-tasks]], §9 of the maintenance playbook). ## Fleet state at this audit - **Fleet 29 jobs — 26 enabled / 3 disabled** (`jobs.json`), unchanged from 10-31. - **Unclosed incident rows: 1 → 1, but a different row.** The 10-31 audit left `ea79a2_a4661554a3c2` / `ea79a2_9197df32320b` / `0e753e_e7f8194a7d06` open; **all three closed on 11-01** (02:39:33 for the two `ea79a2` rows, **10:44:12** for `0e753e_e7f8194a7d06` — the Facebook-repin row that had been open since **2026-10-01 09:00:55**, i.e. 31 days). The new sole open row is `79f9bb_9c443b602eec`. - **Non-clean `last_status` 4 → 2** (`ea79a2107e36`, `79f9bb41ecf5`); **failing outright 1 → 1** (`79f9bb41ecf5`). - **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`.** ## Open questions > ⚠ **Questions 1–3 are answered in [Resolution](#resolution-2026-11-02).** Question 4 remains open. 1. **What did "Gateway connection failed; no local fallback" mean?** Same message for two different jobs 11 ms apart, both in the same handoff batch, with no gateway restart anywhere in the window. 2. **Why did `b63ed5` self-heal in 68 s and `79f9bb` never do so?** Both were in the same batch with the same error string; one closed `completed`, the other never wrote a terminal state. 3. **Should `79f9bb41ecf5` be manually re-run for 2026-11-01?** Anthony did not receive his weekly review. 4. Are the two unreconciled admission files (`141448f9…`, `69b5588f…`) the durable record needed to decide whether either run had side effects? Reconcile before the next 18:00 Sunday slot. ## Resolution (2026-11-02) **The run was not lost. It resumed, finished, and delivered — 7 h 26 m 43 s after it started.** | Field | Value | |---|---| | Execution | `be6ec11a8e68428084af9ab18fb148db` — **the same execution record** the audit saw stuck at `unknown` | | `scheduled_instant` | `2026-11-01T10:00:00+00:00` (18:00 AWST) | | `started_at` | `2026-11-01T18:00:39.125776+08:00` | | `finished_at` | **`2026-11-02T01:27:22.000881+08:00`** | | `status` | **`completed`** | | `error` | `NULL` | | `delivery_outcome` | **`delivered`** | | Delivery row | `2026-11-02T01:27:21.802246+08:00` — `delivered` | | Incident row | `79f9bb_9c443b602eec` → `state: resolved`, `closed_at: 2026-11-02T01:27:21.606691+08:00` | Anthony received the Weekly Review at 01:27 AWST on 2026-11-02. There is still exactly **one** execution row for the 18:00:39 slot, so this is a reconciliation of the existing claim rather than a re-run — the `unknown` terminal state at 18:05:13 was replaced when the run finally wrote its real outcome. **Answers to the open questions:** 1. *(the "no local fallback" message)* — the handoff to the external worker did ultimately succeed; the 18:05:13 message recorded a **failed reconciliation attempt**, not a failed run. The run kept executing for a further 7 h 26 m and produced its deliverable. The message is therefore a **false negative from the reconciler**, not evidence that the work was abandoned. 2. **`b63ed5` closed in 68 s; `79f9bb` closed in 7 h 26 m — both eventually reached `completed`/`delivered`.** The difference was run duration, not failure mode. The reconciler's window is shorter than the Weekly Review's runtime, so any long-running job in a handoff batch will be reported `unknown` while it is still working. **This is the general lesson: an `unknown` execution state on a long job is a timing artefact, not a verdict.** 3. **No manual re-run was needed** — the run self-healed and delivered 12 minutes before the 2026-11-02 audit would have recommended one. Had the audit re-triggered the job, it would have duplicated the run. ### Consequences for the audit method The 2026-11-02 audit read `status: unknown` + `closed_at: NULL` at 00:4x and concluded the deliverable was lost. Both were true at the time of reading and both had inverted by 01:27. **A nightly audit that runs ~2 h after midnight will read every long-running job mid-flight and must not treat a null terminal state as final until the job's expected runtime has elapsed** — see [[systems/scheduled-tasks]] §9. ## Related - [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] - [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]] - [[systems/scheduled-tasks]] - [[systems/messaging-integrations]]