145 lines
8.6 KiB
Markdown
145 lines
8.6 KiB
Markdown
---
|
||
title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Recovered and Delivered 7 h 26 m Later
|
||
type: incident
|
||
status: resolved
|
||
created: 2026-11-02
|
||
updated: 2026-11-03
|
||
verified_on: 2026-11-03
|
||
review_after: 2026-11-17
|
||
confidence: high
|
||
tags: [cron, gateway, handoff, delivery, weekly-review, silent-failure]
|
||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-jobs-json, live-logs, live-probe]
|
||
---
|
||
|
||
# Weekly Review Lost at 18:00 on 2026-11-01 — Recovered and Delivered 7 h 26 m Later
|
||
|
||
> **Status: RESOLVED 2026-11-02T01:27:21+08:00.** The Sunday 18:00 run did *not* lose the review — it
|
||
> resumed, completed and delivered at **01:27:21**, and the incident row auto-closed. The body below is
|
||
> the record **as it stood at the 2026-11-02 audit**, before the recovery landed. See
|
||
> [Resolution](#resolution-2026-11-02) for the corrected outcome and the answers to the open questions.
|
||
|
||
## Headline
|
||
|
||
The **Sunday 18:00 Weekly Review (`79f9bb41ecf5`, `0 18 * * 0`) did not run and nothing was delivered on
|
||
2026-11-01.** At **18:05:13** its execution closed with `status: unknown`, `delivery_outcome: NULL` and
|
||
the error *"Cron execution unverified; reconcile `…/cron/admissions/141448f9…json`: **Gateway connection
|
||
failed; no local fallback**"*. Its incident row **`79f9bb_9c443b602eec` is still `alerted` at the time of
|
||
this audit** (`first_seen` 18:05:13, `alerted_at` 18:05:32, `closed_at: NULL`) and the job reads
|
||
`last_status: error`, **`failure_streak: 1`**. There is **no `deliveries.db` row for any slot after
|
||
11:01:12 on 11-01** — the last delivery of the day was the Tech & AI Digest at 11:01:12.
|
||
|
||
## Timeline (AWST, 2026-11-01)
|
||
|
||
| Time | Event |
|
||
|---|---|
|
||
| 18:00:39 | **Four jobs handed to external workers in one batch**: `2fd50c87b0d0`, `96eb95ce12f2`, `b63ed5817212`, `79f9bb41ecf5` |
|
||
| 18:00:57 / 18:01:04 | `96eb95ce12f2` and `2fd50c87b0d0` run fine (`completed`) |
|
||
| **18:05:13.697679** | `b63ed5_6f45aeea97f0` opens — *"Cron execution unverified; reconcile `…/admissions/69b5588f…json`: Gateway connection failed; no local fallback"* (`alerted_at: None` — **never alerted**) |
|
||
| **18:05:13.708153** | `79f9bb_9c443b602eec` opens with the **identical** message, pointing at `…/admissions/141448f9…json` |
|
||
| 18:05:32 | `79f9bb` incident **alerted** |
|
||
| 18:06:21 | `b63ed5817212` execution closes `completed` (18:00:39 → 18:06:21) → its incident auto-resolves **68 s later** |
|
||
| — | `79f9bb41ecf5` closes at 18:05:13 with **no terminal state, no delivery, no recovery** |
|
||
| 18:04:24 → 18:10 | `agent.log`: `[reaper] Active gateway work detected … deferring restart`, then *"Reaper: Plan A restart completed successfully (5m02s)"* — the **Plan A/B path is RSS compaction, not a process restart** (no `SIGTERM`/`SIGUSR1` and no relay-counter reset) |
|
||
|
||
## What actually reached the user
|
||
|
||
**Nothing.** Across the whole of 11-01 `deliveries.db` holds 8 rows:
|
||
|
||
| Time | Status | Note |
|
||
|---|---|---|
|
||
| 02:39:33 | delivered | Mnemosyne-era failure notice (closes the 10-26 and 10-31 rows) |
|
||
| 03:00:37 | delivered | stocktake |
|
||
| **07:18:50** | **failed** | `configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id` — job `066bb5191280` (see [[systems/messaging-integrations]]) |
|
||
| 07:34:49 | delivered | — |
|
||
| 08:36:51 | delivered | Gmail digest |
|
||
| 10:44:12 | delivered | Facebook scrape |
|
||
| 11:01:12 | delivered | Tech & AI Digest |
|
||
| — | — | **no row after 11:01:12** |
|
||
|
||
So of the day's eight delivery attempts, **seven reached their target and one (`066bb5191280`) landed in
|
||
the DM thread-less while being recorded `failed`**. The Weekly Review's absence is not a suppressed
|
||
delivery — it is an absent run.
|
||
|
||
## Second silent gap: a job that reported `error` with no execution row at all
|
||
|
||
`ecc1fc13655a` (FreshRSS Midday News Briefing) reads **`last_status: error`, `failure_streak: 1`,
|
||
`last_run_at: 2026-11-01T12:01:00.374089+08:00`** — yet **no `executions` row exists for it on 11-01**
|
||
(the last is 2026-10-31T12:01:07, `completed`/`delivered`) and **no `cron_incidents` row was raised.**
|
||
That 12:00 slot overlapped the same `18:00:39`-style handoff batch pattern (three jobs claimed at
|
||
12:00:53). This is the **second job of the day whose failure left no durable record**, and the third
|
||
appearance of the "green/red row, no artefact" anti-pattern the wiki already documents
|
||
([[systems/scheduled-tasks]], §9 of the maintenance playbook).
|
||
|
||
## Fleet state at this audit
|
||
|
||
- **Fleet 29 jobs — 26 enabled / 3 disabled** (`jobs.json`), unchanged from 10-31.
|
||
- **Unclosed incident rows: 1 → 1, but a different row.** The 10-31 audit left
|
||
`ea79a2_a4661554a3c2` / `ea79a2_9197df32320b` / `0e753e_e7f8194a7d06` open; **all three closed on
|
||
11-01** (02:39:33 for the two `ea79a2` rows, **10:44:12** for `0e753e_e7f8194a7d06` — the
|
||
Facebook-repin row that had been open since **2026-10-01 09:00:55**, i.e. 31 days). The new sole
|
||
open row is `79f9bb_9c443b602eec`.
|
||
- **Non-clean `last_status` 4 → 2** (`ea79a2107e36`, `79f9bb41ecf5`); **failing outright 1 → 1**
|
||
(`79f9bb41ecf5`).
|
||
- **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`.**
|
||
|
||
## Open questions
|
||
|
||
> ⚠ **Questions 1–3 are answered in [Resolution](#resolution-2026-11-02).** Question 4 remains open.
|
||
|
||
1. **What did "Gateway connection failed; no local fallback" mean?** Same message for two different
|
||
jobs 11 ms apart, both in the same handoff batch, with no gateway restart anywhere in the window.
|
||
2. **Why did `b63ed5` self-heal in 68 s and `79f9bb` never do so?** Both were in the same batch with the
|
||
same error string; one closed `completed`, the other never wrote a terminal state.
|
||
3. **Should `79f9bb41ecf5` be manually re-run for 2026-11-01?** Anthony did not receive his weekly
|
||
review.
|
||
4. Are the two unreconciled admission files (`141448f9…`, `69b5588f…`) the durable record needed to
|
||
decide whether either run had side effects? Reconcile before the next 18:00 Sunday slot.
|
||
|
||
## Resolution (2026-11-02)
|
||
|
||
**The run was not lost. It resumed, finished, and delivered — 7 h 26 m 43 s after it started.**
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Execution | `be6ec11a8e68428084af9ab18fb148db` — **the same execution record** the audit saw stuck at `unknown` |
|
||
| `scheduled_instant` | `2026-11-01T10:00:00+00:00` (18:00 AWST) |
|
||
| `started_at` | `2026-11-01T18:00:39.125776+08:00` |
|
||
| `finished_at` | **`2026-11-02T01:27:22.000881+08:00`** |
|
||
| `status` | **`completed`** |
|
||
| `error` | `NULL` |
|
||
| `delivery_outcome` | **`delivered`** |
|
||
| Delivery row | `2026-11-02T01:27:21.802246+08:00` — `delivered` |
|
||
| Incident row | `79f9bb_9c443b602eec` → `state: resolved`, `closed_at: 2026-11-02T01:27:21.606691+08:00` |
|
||
|
||
Anthony received the Weekly Review at 01:27 AWST on 2026-11-02. There is still exactly **one** execution
|
||
row for the 18:00:39 slot, so this is a reconciliation of the existing claim rather than a re-run — the
|
||
`unknown` terminal state at 18:05:13 was replaced when the run finally wrote its real outcome.
|
||
|
||
**Answers to the open questions:**
|
||
|
||
1. *(the "no local fallback" message)* — the handoff to the external worker did ultimately succeed; the
|
||
18:05:13 message recorded a **failed reconciliation attempt**, not a failed run. The run kept executing
|
||
for a further 7 h 26 m and produced its deliverable. The message is therefore a **false negative from
|
||
the reconciler**, not evidence that the work was abandoned.
|
||
2. **`b63ed5` closed in 68 s; `79f9bb` closed in 7 h 26 m — both eventually reached `completed`/`delivered`.**
|
||
The difference was run duration, not failure mode. The reconciler's window is shorter than the Weekly
|
||
Review's runtime, so any long-running job in a handoff batch will be reported `unknown` while it is
|
||
still working. **This is the general lesson: an `unknown` execution state on a long job is a timing
|
||
artefact, not a verdict.**
|
||
3. **No manual re-run was needed** — the run self-healed and delivered 12 minutes before the 2026-11-02
|
||
audit would have recommended one. Had the audit re-triggered the job, it would have duplicated the run.
|
||
|
||
### Consequences for the audit method
|
||
|
||
The 2026-11-02 audit read `status: unknown` + `closed_at: NULL` at 00:4x and concluded the deliverable was
|
||
lost. Both were true at the time of reading and both had inverted by 01:27. **A nightly audit that runs
|
||
~2 h after midnight will read every long-running job mid-flight and must not treat a null terminal state as
|
||
final until the job's expected runtime has elapsed** — see [[systems/scheduled-tasks]] §9.
|
||
|
||
## Related
|
||
|
||
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]]
|
||
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
|
||
- [[systems/scheduled-tasks]]
|
||
- [[systems/messaging-integrations]]
|