8.6 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Recovered and Delivered 7 h 26 m Later | incident | resolved | 2026-11-02 | 2026-11-03 | 2026-11-03 | 2026-11-17 | high |
|
|
Weekly Review Lost at 18:00 on 2026-11-01 — Recovered and Delivered 7 h 26 m Later
Status: RESOLVED 2026-11-02T01:27:21+08:00. The Sunday 18:00 run did not lose the review — it resumed, completed and delivered at 01:27:21, and the incident row auto-closed. The body below is the record as it stood at the 2026-11-02 audit, before the recovery landed. See Resolution for the corrected outcome and the answers to the open questions.
Headline
The Sunday 18:00 Weekly Review (79f9bb41ecf5, 0 18 * * 0) did not run and nothing was delivered on
2026-11-01. At 18:05:13 its execution closed with status: unknown, delivery_outcome: NULL and
the error "Cron execution unverified; reconcile …/cron/admissions/141448f9…json: Gateway connection
failed; no local fallback". Its incident row 79f9bb_9c443b602eec is still alerted at the time of
this audit (first_seen 18:05:13, alerted_at 18:05:32, closed_at: NULL) and the job reads
last_status: error, failure_streak: 1. There is no deliveries.db row for any slot after
11:01:12 on 11-01 — the last delivery of the day was the Tech & AI Digest at 11:01:12.
Timeline (AWST, 2026-11-01)
| Time | Event |
|---|---|
| 18:00:39 | Four jobs handed to external workers in one batch: 2fd50c87b0d0, 96eb95ce12f2, b63ed5817212, 79f9bb41ecf5 |
| 18:00:57 / 18:01:04 | 96eb95ce12f2 and 2fd50c87b0d0 run fine (completed) |
| 18:05:13.697679 | b63ed5_6f45aeea97f0 opens — "Cron execution unverified; reconcile …/admissions/69b5588f…json: Gateway connection failed; no local fallback" (alerted_at: None — never alerted) |
| 18:05:13.708153 | 79f9bb_9c443b602eec opens with the identical message, pointing at …/admissions/141448f9…json |
| 18:05:32 | 79f9bb incident alerted |
| 18:06:21 | b63ed5817212 execution closes completed (18:00:39 → 18:06:21) → its incident auto-resolves 68 s later |
| — | 79f9bb41ecf5 closes at 18:05:13 with no terminal state, no delivery, no recovery |
| 18:04:24 → 18:10 | agent.log: [reaper] Active gateway work detected … deferring restart, then "Reaper: Plan A restart completed successfully (5m02s)" — the Plan A/B path is RSS compaction, not a process restart (no SIGTERM/SIGUSR1 and no relay-counter reset) |
What actually reached the user
Nothing. Across the whole of 11-01 deliveries.db holds 8 rows:
| Time | Status | Note |
|---|---|---|
| 02:39:33 | delivered | Mnemosyne-era failure notice (closes the 10-26 and 10-31 rows) |
| 03:00:37 | delivered | stocktake |
| 07:18:50 | failed | configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id — job 066bb5191280 (see systems/messaging-integrations) |
| 07:34:49 | delivered | — |
| 08:36:51 | delivered | Gmail digest |
| 10:44:12 | delivered | Facebook scrape |
| 11:01:12 | delivered | Tech & AI Digest |
| — | — | no row after 11:01:12 |
So of the day's eight delivery attempts, seven reached their target and one (066bb5191280) landed in
the DM thread-less while being recorded failed. The Weekly Review's absence is not a suppressed
delivery — it is an absent run.
Second silent gap: a job that reported error with no execution row at all
ecc1fc13655a (FreshRSS Midday News Briefing) reads last_status: error, failure_streak: 1,
last_run_at: 2026-11-01T12:01:00.374089+08:00 — yet no executions row exists for it on 11-01
(the last is 2026-10-31T12:01:07, completed/delivered) and no cron_incidents row was raised.
That 12:00 slot overlapped the same 18:00:39-style handoff batch pattern (three jobs claimed at
12:00:53). This is the second job of the day whose failure left no durable record, and the third
appearance of the "green/red row, no artefact" anti-pattern the wiki already documents
(systems/scheduled-tasks, §9 of the maintenance playbook).
Fleet state at this audit
- Fleet 29 jobs — 26 enabled / 3 disabled (
jobs.json), unchanged from 10-31. - Unclosed incident rows: 1 → 1, but a different row. The 10-31 audit left
ea79a2_a4661554a3c2/ea79a2_9197df32320b/0e753e_e7f8194a7d06open; all three closed on 11-01 (02:39:33 for the twoea79a2rows, 10:44:12 for0e753e_e7f8194a7d06— the Facebook-repin row that had been open since 2026-10-01 09:00:55, i.e. 31 days). The new sole open row is79f9bb_9c443b602eec. - Non-clean
last_status4 → 2 (ea79a2107e36,79f9bb41ecf5); failing outright 1 → 1 (79f9bb41ecf5). - 2026-11-01 settled 167
completed/ 1running/ 2unknown.
Open questions
⚠ Questions 1–3 are answered in Resolution. Question 4 remains open.
- What did "Gateway connection failed; no local fallback" mean? Same message for two different jobs 11 ms apart, both in the same handoff batch, with no gateway restart anywhere in the window.
- Why did
b63ed5self-heal in 68 s and79f9bbnever do so? Both were in the same batch with the same error string; one closedcompleted, the other never wrote a terminal state. - Should
79f9bb41ecf5be manually re-run for 2026-11-01? Anthony did not receive his weekly review. - Are the two unreconciled admission files (
141448f9…,69b5588f…) the durable record needed to decide whether either run had side effects? Reconcile before the next 18:00 Sunday slot.
Resolution (2026-11-02)
The run was not lost. It resumed, finished, and delivered — 7 h 26 m 43 s after it started.
| Field | Value |
|---|---|
| Execution | be6ec11a8e68428084af9ab18fb148db — the same execution record the audit saw stuck at unknown |
scheduled_instant |
2026-11-01T10:00:00+00:00 (18:00 AWST) |
started_at |
2026-11-01T18:00:39.125776+08:00 |
finished_at |
2026-11-02T01:27:22.000881+08:00 |
status |
completed |
error |
NULL |
delivery_outcome |
delivered |
| Delivery row | 2026-11-02T01:27:21.802246+08:00 — delivered |
| Incident row | 79f9bb_9c443b602eec → state: resolved, closed_at: 2026-11-02T01:27:21.606691+08:00 |
Anthony received the Weekly Review at 01:27 AWST on 2026-11-02. There is still exactly one execution
row for the 18:00:39 slot, so this is a reconciliation of the existing claim rather than a re-run — the
unknown terminal state at 18:05:13 was replaced when the run finally wrote its real outcome.
Answers to the open questions:
- (the "no local fallback" message) — the handoff to the external worker did ultimately succeed; the 18:05:13 message recorded a failed reconciliation attempt, not a failed run. The run kept executing for a further 7 h 26 m and produced its deliverable. The message is therefore a false negative from the reconciler, not evidence that the work was abandoned.
b63ed5closed in 68 s;79f9bbclosed in 7 h 26 m — both eventually reachedcompleted/delivered. The difference was run duration, not failure mode. The reconciler's window is shorter than the Weekly Review's runtime, so any long-running job in a handoff batch will be reportedunknownwhile it is still working. This is the general lesson: anunknownexecution state on a long job is a timing artefact, not a verdict.- No manual re-run was needed — the run self-healed and delivered 12 minutes before the 2026-11-02 audit would have recommended one. Had the audit re-triggered the job, it would have duplicated the run.
Consequences for the audit method
The 2026-11-02 audit read status: unknown + closed_at: NULL at 00:4x and concluded the deliverable was
lost. Both were true at the time of reading and both had inverted by 01:27. A nightly audit that runs
~2 h after midnight will read every long-running job mid-flight and must not treat a null terminal state as
final until the job's expected runtime has elapsed — see systems/scheduled-tasks §9.