Files
agent-estate-wiki/incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
T

145 lines
8.6 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Recovered and Delivered 7 h 26 m Later
type: incident
status: resolved
created: 2026-11-02
updated: 2026-11-03
verified_on: 2026-11-03
review_after: 2026-11-17
confidence: high
tags: [cron, gateway, handoff, delivery, weekly-review, silent-failure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-jobs-json, live-logs, live-probe]
---
# Weekly Review Lost at 18:00 on 2026-11-01 — Recovered and Delivered 7 h 26 m Later
> **Status: RESOLVED 2026-11-02T01:27:21+08:00.** The Sunday 18:00 run did *not* lose the review — it
> resumed, completed and delivered at **01:27:21**, and the incident row auto-closed. The body below is
> the record **as it stood at the 2026-11-02 audit**, before the recovery landed. See
> [Resolution](#resolution-2026-11-02) for the corrected outcome and the answers to the open questions.
## Headline
The **Sunday 18:00 Weekly Review (`79f9bb41ecf5`, `0 18 * * 0`) did not run and nothing was delivered on
2026-11-01.** At **18:05:13** its execution closed with `status: unknown`, `delivery_outcome: NULL` and
the error *"Cron execution unverified; reconcile `…/cron/admissions/141448f9…json`: **Gateway connection
failed; no local fallback**"*. Its incident row **`79f9bb_9c443b602eec` is still `alerted` at the time of
this audit** (`first_seen` 18:05:13, `alerted_at` 18:05:32, `closed_at: NULL`) and the job reads
`last_status: error`, **`failure_streak: 1`**. There is **no `deliveries.db` row for any slot after
11:01:12 on 11-01** — the last delivery of the day was the Tech & AI Digest at 11:01:12.
## Timeline (AWST, 2026-11-01)
| Time | Event |
|---|---|
| 18:00:39 | **Four jobs handed to external workers in one batch**: `2fd50c87b0d0`, `96eb95ce12f2`, `b63ed5817212`, `79f9bb41ecf5` |
| 18:00:57 / 18:01:04 | `96eb95ce12f2` and `2fd50c87b0d0` run fine (`completed`) |
| **18:05:13.697679** | `b63ed5_6f45aeea97f0` opens — *"Cron execution unverified; reconcile `…/admissions/69b5588f…json`: Gateway connection failed; no local fallback"* (`alerted_at: None` — **never alerted**) |
| **18:05:13.708153** | `79f9bb_9c443b602eec` opens with the **identical** message, pointing at `…/admissions/141448f9…json` |
| 18:05:32 | `79f9bb` incident **alerted** |
| 18:06:21 | `b63ed5817212` execution closes `completed` (18:00:39 → 18:06:21) → its incident auto-resolves **68 s later** |
| — | `79f9bb41ecf5` closes at 18:05:13 with **no terminal state, no delivery, no recovery** |
| 18:04:24 → 18:10 | `agent.log`: `[reaper] Active gateway work detected … deferring restart`, then *"Reaper: Plan A restart completed successfully (5m02s)"* — the **Plan A/B path is RSS compaction, not a process restart** (no `SIGTERM`/`SIGUSR1` and no relay-counter reset) |
## What actually reached the user
**Nothing.** Across the whole of 11-01 `deliveries.db` holds 8 rows:
| Time | Status | Note |
|---|---|---|
| 02:39:33 | delivered | Mnemosyne-era failure notice (closes the 10-26 and 10-31 rows) |
| 03:00:37 | delivered | stocktake |
| **07:18:50** | **failed** | `configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id` — job `066bb5191280` (see [[systems/messaging-integrations]]) |
| 07:34:49 | delivered | — |
| 08:36:51 | delivered | Gmail digest |
| 10:44:12 | delivered | Facebook scrape |
| 11:01:12 | delivered | Tech & AI Digest |
| — | — | **no row after 11:01:12** |
So of the day's eight delivery attempts, **seven reached their target and one (`066bb5191280`) landed in
the DM thread-less while being recorded `failed`**. The Weekly Review's absence is not a suppressed
delivery — it is an absent run.
## Second silent gap: a job that reported `error` with no execution row at all
`ecc1fc13655a` (FreshRSS Midday News Briefing) reads **`last_status: error`, `failure_streak: 1`,
`last_run_at: 2026-11-01T12:01:00.374089+08:00`** — yet **no `executions` row exists for it on 11-01**
(the last is 2026-10-31T12:01:07, `completed`/`delivered`) and **no `cron_incidents` row was raised.**
That 12:00 slot overlapped the same `18:00:39`-style handoff batch pattern (three jobs claimed at
12:00:53). This is the **second job of the day whose failure left no durable record**, and the third
appearance of the "green/red row, no artefact" anti-pattern the wiki already documents
([[systems/scheduled-tasks]], §9 of the maintenance playbook).
## Fleet state at this audit
- **Fleet 29 jobs — 26 enabled / 3 disabled** (`jobs.json`), unchanged from 10-31.
- **Unclosed incident rows: 1 → 1, but a different row.** The 10-31 audit left
`ea79a2_a4661554a3c2` / `ea79a2_9197df32320b` / `0e753e_e7f8194a7d06` open; **all three closed on
11-01** (02:39:33 for the two `ea79a2` rows, **10:44:12** for `0e753e_e7f8194a7d06` — the
Facebook-repin row that had been open since **2026-10-01 09:00:55**, i.e. 31 days). The new sole
open row is `79f9bb_9c443b602eec`.
- **Non-clean `last_status` 4 → 2** (`ea79a2107e36`, `79f9bb41ecf5`); **failing outright 1 → 1**
(`79f9bb41ecf5`).
- **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`.**
## Open questions
> ⚠ **Questions 1–3 are answered in [Resolution](#resolution-2026-11-02).** Question 4 remains open.
1. **What did "Gateway connection failed; no local fallback" mean?** Same message for two different
jobs 11 ms apart, both in the same handoff batch, with no gateway restart anywhere in the window.
2. **Why did `b63ed5` self-heal in 68 s and `79f9bb` never do so?** Both were in the same batch with the
same error string; one closed `completed`, the other never wrote a terminal state.
3. **Should `79f9bb41ecf5` be manually re-run for 2026-11-01?** Anthony did not receive his weekly
review.
4. Are the two unreconciled admission files (`141448f9…`, `69b5588f…`) the durable record needed to
decide whether either run had side effects? Reconcile before the next 18:00 Sunday slot.
## Resolution (2026-11-02)
**The run was not lost. It resumed, finished, and delivered — 7 h 26 m 43 s after it started.**
| Field | Value |
|---|---|
| Execution | `be6ec11a8e68428084af9ab18fb148db` — **the same execution record** the audit saw stuck at `unknown` |
| `scheduled_instant` | `2026-11-01T10:00:00+00:00` (18:00 AWST) |
| `started_at` | `2026-11-01T18:00:39.125776+08:00` |
| `finished_at` | **`2026-11-02T01:27:22.000881+08:00`** |
| `status` | **`completed`** |
| `error` | `NULL` |
| `delivery_outcome` | **`delivered`** |
| Delivery row | `2026-11-02T01:27:21.802246+08:00` — `delivered` |
| Incident row | `79f9bb_9c443b602eec` → `state: resolved`, `closed_at: 2026-11-02T01:27:21.606691+08:00` |
Anthony received the Weekly Review at 01:27 AWST on 2026-11-02. There is still exactly **one** execution
row for the 18:00:39 slot, so this is a reconciliation of the existing claim rather than a re-run — the
`unknown` terminal state at 18:05:13 was replaced when the run finally wrote its real outcome.
**Answers to the open questions:**
1. *(the "no local fallback" message)* — the handoff to the external worker did ultimately succeed; the
18:05:13 message recorded a **failed reconciliation attempt**, not a failed run. The run kept executing
for a further 7 h 26 m and produced its deliverable. The message is therefore a **false negative from
the reconciler**, not evidence that the work was abandoned.
2. **`b63ed5` closed in 68 s; `79f9bb` closed in 7 h 26 m — both eventually reached `completed`/`delivered`.**
The difference was run duration, not failure mode. The reconciler's window is shorter than the Weekly
Review's runtime, so any long-running job in a handoff batch will be reported `unknown` while it is
still working. **This is the general lesson: an `unknown` execution state on a long job is a timing
artefact, not a verdict.**
3. **No manual re-run was needed** — the run self-healed and delivered 12 minutes before the 2026-11-02
audit would have recommended one. Had the audit re-triggered the job, it would have duplicated the run.
### Consequences for the audit method
The 2026-11-02 audit read `status: unknown` + `closed_at: NULL` at 00:4x and concluded the deliverable was
lost. Both were true at the time of reading and both had inverted by 01:27. **A nightly audit that runs
~2 h after midnight will read every long-running job mid-flight and must not treat a null terminal state as
final until the job's expected runtime has elapsed** — see [[systems/scheduled-tasks]] §9.
## Related
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]]
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
- [[systems/scheduled-tasks]]
- [[systems/messaging-integrations]]