Files
agent-estate-wiki/incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
T
Tony0410 fda56e53fe audit 2026-11-02: Weekly Review lost to gateway handoff failure; three cron workers SIGTERM'd
Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown.

NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active)
NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved)
UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section)
UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h)
UPDATED index.md, log.md
2026-11-02 00:27:09 +08:00

5.6 KiB

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open incident active 2026-11-02 2026-11-02 2026-11-02 2026-11-09 high
cron
gateway
handoff
delivery
weekly-review
silent-failure
live-sqlite-cron-executions
live-sqlite-cron-incidents
live-sqlite-deliveries
live-jobs-json
live-logs
live-probe

Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open

Headline

The Sunday 18:00 Weekly Review (79f9bb41ecf5, 0 18 * * 0) did not run and nothing was delivered on 2026-11-01. At 18:05:13 its execution closed with status: unknown, delivery_outcome: NULL and the error "Cron execution unverified; reconcile …/cron/admissions/141448f9…json: Gateway connection failed; no local fallback". Its incident row 79f9bb_9c443b602eec is still alerted at the time of this audit (first_seen 18:05:13, alerted_at 18:05:32, closed_at: NULL) and the job reads last_status: error, failure_streak: 1. There is no deliveries.db row for any slot after 11:01:12 on 11-01 — the last delivery of the day was the Tech & AI Digest at 11:01:12.

Timeline (AWST, 2026-11-01)

Time Event
18:00:39 Four jobs handed to external workers in one batch: 2fd50c87b0d0, 96eb95ce12f2, b63ed5817212, 79f9bb41ecf5
18:00:57 / 18:01:04 96eb95ce12f2 and 2fd50c87b0d0 run fine (completed)
18:05:13.697679 b63ed5_6f45aeea97f0 opens — "Cron execution unverified; reconcile …/admissions/69b5588f…json: Gateway connection failed; no local fallback" (alerted_at: None — never alerted)
18:05:13.708153 79f9bb_9c443b602eec opens with the identical message, pointing at …/admissions/141448f9…json
18:05:32 79f9bb incident alerted
18:06:21 b63ed5817212 execution closes completed (18:00:39 → 18:06:21) → its incident auto-resolves 68 s later
— 79f9bb41ecf5 closes at 18:05:13 with no terminal state, no delivery, no recovery
18:04:24 → 18:10 agent.log: [reaper] Active gateway work detected … deferring restart, then "Reaper: Plan A restart completed successfully (5m02s)" — the Plan A/B path is RSS compaction, not a process restart (no SIGTERM/SIGUSR1 and no relay-counter reset)

What actually reached the user

Nothing. Across the whole of 11-01 deliveries.db holds 8 rows:

Time Status Note
02:39:33 delivered Mnemosyne-era failure notice (closes the 10-26 and 10-31 rows)
03:00:37 delivered stocktake
07:18:50 failed configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id — job 066bb5191280 (see systems/messaging-integrations)
07:34:49 delivered —
08:36:51 delivered Gmail digest
10:44:12 delivered Facebook scrape
11:01:12 delivered Tech & AI Digest
— — no row after 11:01:12

So of the day's eight delivery attempts, seven reached their target and one (066bb5191280) landed in the DM thread-less while being recorded failed. The Weekly Review's absence is not a suppressed delivery — it is an absent run.

Second silent gap: a job that reported error with no execution row at all

ecc1fc13655a (FreshRSS Midday News Briefing) reads last_status: error, failure_streak: 1, last_run_at: 2026-11-01T12:01:00.374089+08:00 — yet no executions row exists for it on 11-01 (the last is 2026-10-31T12:01:07, completed/delivered) and no cron_incidents row was raised. That 12:00 slot overlapped the same 18:00:39-style handoff batch pattern (three jobs claimed at 12:00:53). This is the second job of the day whose failure left no durable record, and the third appearance of the "green/red row, no artefact" anti-pattern the wiki already documents (systems/scheduled-tasks, §9 of the maintenance playbook).

Fleet state at this audit

  • Fleet 29 jobs — 26 enabled / 3 disabled (jobs.json), unchanged from 10-31.
  • Unclosed incident rows: 1 → 1, but a different row. The 10-31 audit left ea79a2_a4661554a3c2 / ea79a2_9197df32320b / 0e753e_e7f8194a7d06 open; all three closed on 11-01 (02:39:33 for the two ea79a2 rows, 10:44:12 for 0e753e_e7f8194a7d06 — the Facebook-repin row that had been open since 2026-10-01 09:00:55, i.e. 31 days). The new sole open row is 79f9bb_9c443b602eec.
  • Non-clean last_status 4 → 2 (ea79a2107e36, 79f9bb41ecf5); failing outright 1 → 1 (79f9bb41ecf5).
  • 2026-11-01 settled 167 completed / 1 running / 2 unknown.

Open questions

  1. What did "Gateway connection failed; no local fallback" mean? Same message for two different jobs 11 ms apart, both in the same handoff batch, with no gateway restart anywhere in the window.
  2. Why did b63ed5 self-heal in 68 s and 79f9bb never do so? Both were in the same batch with the same error string; one closed completed, the other never wrote a terminal state.
  3. Should 79f9bb41ecf5 be manually re-run for 2026-11-01? Anthony did not receive his weekly review.
  4. Are the two unreconciled admission files (141448f9…, 69b5588f…) the durable record needed to decide whether either run had side effects? Reconcile before the next 18:00 Sunday slot.