audit 2026-11-02: Weekly Review lost to gateway handoff failure; three cron workers SIGTERM'd
Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown. NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active) NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved) UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section) UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h) UPDATED index.md, log.md
This commit is contained in:
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
review_after: 2026-11-06
|
||||
updated: 2026-11-02
|
||||
verified_on: 2026-11-02
|
||||
review_after: 2026-11-09
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
@@ -14,7 +14,33 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-11-01 00:2x AWST (live `jobs.json` + `executions.db` + `cron_incidents` + gateway
|
||||
- **Current as of 2026-11-02 00:4x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
|
||||
`deliveries.db` + gateway log) — the fleet ran clean all day, then failed in three different ways
|
||||
and missed a delivery.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged). **Window 2026-11-01
|
||||
settled 167 `completed` / 1 `running` / 2 `unknown`** (+ this audit `running`, started 23:59:28)
|
||||
against **161 / 4** on 10-31, **138 / 0** on 10-30 and **177 / 0** on 10-29 — the first **non-clean**
|
||||
day in four. **The two `unknown` rows are the whole story of the day:** the Sunday 18:00 **Weekly
|
||||
Review never ran and never delivered** (*"Gateway connection failed; no local fallback"*, incident
|
||||
`79f9bb_9c443b602eec` **still `alerted`** at audit time, job reads `error` / `failure_streak: 1`),
|
||||
and one `96eb95ce12f2` watchdog slot. **Three other jobs were `SIGTERM`ed (`-15`) mid-adoption without
|
||||
writing a durable terminal state** (`85edf6772bca` @07:30:46, `96eb95ce12f2` and `0e753e9720fd`
|
||||
@09:02:18) — with **no gateway restart or reload anywhere in the window** (PID `175269` up since
|
||||
10-31 19:24:40; the relay counter ran unbroken through both signals) →
|
||||
[[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] and
|
||||
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]. **Delivery health:** 8
|
||||
delivery rows on 11-01, **7 `delivered` / 1 `failed`** — the failure is `066bb5191280`'s
|
||||
`thread_id 34795` DM-thread miss (the thread-less fallback still reaches the DM; the run is recorded
|
||||
`failed` anyway). **Non-clean `last_status` 4 → 2`** (`ea79a2107e36`, `79f9bb41ecf5`).
|
||||
⚠ **`ecc1fc13655a` reads `error` with `failure_streak: 1` and `last_run 2026-11-01T12:01:00` while
|
||||
owning NO `executions` row for that slot and NO `cron_incidents` row** — a status field with no
|
||||
artefact behind it. **Unclosed incident rows 1 → 1, but a different row:** all three rows the 10-31
|
||||
audit left open closed on 11-01 (02:39:33 ×2 for `ea79a2`, **10:44:12** for
|
||||
`0e753e_e7f8194a7d06` — the Facebook-repin row open since **2026-10-01**, 31 days). ⚠ **Telegram's
|
||||
transport flapped all day** (dual-stack `api.telegram.org` ConnectTimeouts, IPv4-literal fallbacks,
|
||||
a disconnect at 20:51:53, polling restarts at 22:33:30 / 23:12:23) and the **relay adapter has been
|
||||
absent ~29 h** (`Reconnecting relay` attempt **336** at 11-02 00:10:47) →
|
||||
[[systems/messaging-integrations]].
|
||||
- **Prior window (2026-11-01 00:2x AWST, live `jobs.json` + `executions.db` + `cron_incidents` + gateway
|
||||
log) — the last five hours of 10-31 ran clean, but an 18:06 gateway reload force-drained at its full
|
||||
1 800 s cap and killed a live cron run.** Fleet **29 jobs — 26 enabled / 3 disabled**
|
||||
(`jobs.json.updated_at 2026-11-01T00:05:42+08:00`). **Window 2026-10-31 settled 160 `completed` /
|
||||
@@ -669,3 +695,62 @@ obsolete, no decision reversed in the window.
|
||||
Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255
|
||||
stopped; the Conduit patch uncommitted and **five releases behind HEAD**; the **569.9 MB** Conduit
|
||||
health store with no retention; and a **call that has never been placed** since the memory fix.
|
||||
|
||||
## Re-verified 2026-11-02 (audit window 2026-11-01 00:00 → 2026-11-02 00:4x AWST; live probes 00:0x–00:5x)
|
||||
|
||||
**Settled: 167 `completed` / 1 `running` / 2 `unknown`** — the first non-clean day after three
|
||||
consecutive clean-or-nearly-clean days (10-31 was 161 / 4). Fleet unchanged at **29 jobs, 26 enabled,
|
||||
3 disabled**. Nothing was added, removed, disabled or repinned in the window.
|
||||
|
||||
### The day's three failure modes, none of which is a repeat of 10-31
|
||||
|
||||
| # | Fault | Jobs | Durable record | Page |
|
||||
|---|---|---|---|---|
|
||||
| 1 | **`SIGTERM` (`-15`) after adoption, no terminal state** — *"whether side effects ran is unknown"* | `85edf6772bca` 07:30:46, `96eb95ce12f2` + `0e753e9720fd` 09:02:18 | 3 `cron_incidents` rows, all `resolved`; one execution left `unknown` | [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] |
|
||||
| 2 | **Gateway handoff failure** — *"Cron execution unverified … Gateway connection failed; no local fallback"* | `b63ed5817212` (self-healed 68 s) and `79f9bb41ecf5` | 2 rows; **`79f9bb_9c443b602eec` still `alerted`** | [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] |
|
||||
| 3 | **`last_status: error` with no execution row and no incident row** | `ecc1fc13655a`, `last_run 2026-11-01T12:01:00` | **none** | same page, "Second silent gap" |
|
||||
|
||||
Fault 1 is a **fourth variant** of the orphan class (after 10-20 crash, 10-26 drained-`SIGTERM`, 10-31
|
||||
cap force-drain) and the first with **no gateway restart anywhere near it** — so it cannot be explained
|
||||
by reload/drain behaviour. Fault 3 is the inverse of the usual silent failure: instead of a green row
|
||||
over a dead job, there is a **red status with no row at all**.
|
||||
|
||||
### The 09:00 pair — cold-start cost meets a 30 s ack window
|
||||
|
||||
At **09:00:49.708** the spawner logged `Cron external worker for job '96eb95ce12f2' did not acknowledge
|
||||
within 30s; leaving the durable execution claim untouched`; the worker then spent **09:00:49 → 09:01:02**
|
||||
doing a full cold boot (`Plugin discovery complete: 83 found, 69 enabled`) before starting its job at
|
||||
09:01:13 and being signalled at ~09:02:13. **The 30 s window is smaller than this host's cold-start
|
||||
time.** Not yet formalised as an Open item because the signal source is unidentified — see the incident
|
||||
page's open questions.
|
||||
|
||||
### Delivery health: 8 rows, 7 delivered
|
||||
|
||||
`deliveries.db` for 11-01: 02:39:33 ✔, 03:00:37 ✔, **07:18:50 ✘** (`066bb5191280`,
|
||||
`configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id`),
|
||||
07:34:49 ✔, 08:36:51 ✔, 10:44:12 ✔, 11:01:12 ✔ — **and nothing after 11:01:12**, which is how the
|
||||
missing Weekly Review was detected. The `066bb5191280` case is the **second variant** documented in
|
||||
[[systems/messaging-integrations]] (thread-less fallback send, run still marked `failed`).
|
||||
|
||||
### Incident ledger movement
|
||||
|
||||
- **Closed on 11-01:** `ea79a2_a4661554a3c2` + `ea79a2_9197df32320b` (02:39:33 — the Mnemosyne Health
|
||||
Monitor pair from 10-26 / 10-31) and **`0e753e_e7f8194a7d06`** (10:44:12) — the Facebook birthday
|
||||
repin row that had been open since **2026-10-01 09:00:55**, the oldest row on the board. That job's
|
||||
`delivery_outcome: delivered` on the same run.
|
||||
- **Opened on 11-01:** 5 rows; **4 resolved, 1 open**.
|
||||
- **Now unclosed: exactly one** — `79f9bb_9c443b602eec` (`state: alerted`, `acked_at: None`).
|
||||
|
||||
### Deliberately not done
|
||||
|
||||
- No full vault mirror pass (drift counts recorded in `log.md`; a complete pass is bigger than an
|
||||
audit's mandate).
|
||||
- No edits to `### Open` beyond the `066bb5191280` note above: the 10-13 GitHub-token and Camofox
|
||||
containment items were not re-probed this window, so their lines are left as last verified.
|
||||
- Nothing removed or archived — no job retired, no system obsoleted, no decision reversed.
|
||||
|
||||
**Still open (carried):** the single-node no-failover topology behind every model-calling job; the
|
||||
GitHub Trending Radar token; Camofox containment under multi-tab load; the Conduit health store with no
|
||||
retention; **plus new:** the unreconciled admission files
|
||||
`~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json` (both 11-01) and the unidentified `-15`
|
||||
signal source.
|
||||
|
||||
Reference in New Issue
Block a user