audit: 2026-10-26 nightly — planned restart orphaned a live cron run (new incident page); 503 burst fully closed; radar cover-letter PDF pipeline moved to Chromium-over-HTML

This commit is contained in:
Tony0410
2026-10-27 00:11:50 +08:00
parent 3f796dc3b9
commit a2dd0bd650
8 changed files with 236 additions and 20 deletions
+46
View File
@@ -1408,3 +1408,49 @@ The two top-level **stale** files are `current-state.md` and
`reports/communities-wa-social-media-audit-2026.md` — both dated historical records
rather than drift. The vault repo is **local-only** (no remotes); it was committed,
not pushed.
## [2026-10-27] audit | Nightly wiki audit — window 2026-10-26 00:00 → 2026-10-27 00:5x AWST: a planned restart orphaned a live cron run
- **A *planned* gateway restart orphaned a cron run — the second instance of the 2026-10-20 orphan class, and
this time it was loud.** `gateway.log`: `Received SIGTERM as a planned gateway stop` at **14:33:35**
(`under_systemd=yes`, `parent_name=systemd`, `loadavg_1m=2.15`). The drain then logged
`Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True` → `interrupting remaining work`,
and at 14:34:29 `Skipping .clean_shutdown marker — drain timed out with interrupted agents`. The victim was
**`ea79a2107e36` (Mnemosyne Health Monitor)**, claimed **12:01:13** — **2 h 34 m** in. It closed **14:35:59
`failed`** on a string this wiki had not seen: **`RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its
durable fire claim ownership`**. Unlike the 10-20 orphan it raised **two `cron_incidents` rows** (14:33:37
`ea79a2_c3ab2d784ba3` *"Cron execution unverified; reconcile `admissions/724a69108b…json`"*; 14:35:59
`ea79a2_a4661554a3c2`, the RuntimeError) and **delivered** the failure. → new
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md`
- **The Mnemosyne Watchdog did its job.** `96eb95ce12f2` red-flagged twice (14:46:12, 15:01:15,
`Script exited with code 1`) reacting to the same log `ERROR`, then **self-resolved at 15:16:12** when the
line aged out of its 15-minute window. Both rows (`96eb95_0c89bd373a0a`, `96eb95_e32174134546`) are closed.
- **Window tally and fleet:** 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are
this one interruption** (Health Monitor + the two watchdog reds); 2026-10-25 settled **172 / 2**. Fleet
unchanged: **28 jobs, 25 enabled / 3 disabled**. **Non-clean `last_status` 5 → 4** (`c0a9fa314542` and
`08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as `error`, streak 2). **Unclosed incident rows 2 → 3**.
**Pins unchanged.** One gateway start (14:35:18, teardown 54.06 s). **17 sessions in window** (15 `cron`,
1 `ios`, 1 `telegram`).
- **One incident row closed — the 2026-10-16 503 burst is fully shut.** `c0a9fa_764558b5ae83` **auto-closed
09:13:00** when the radar's 09:00 slot ran clean; no execution row carries the resource-pressure string on
10-25 or 10-26. Status stays `monitoring` (the single-node exposure is unchanged). Updated
`incidents/2026-10-16-omniroute-503-resource-pressure.md`.
- **⚠ Two quiet degradations: a completed run that delivered nothing useful, and a broken delivery thread.**
(a) The radar's 07:00 slot closed **`completed` at 09:10**, but its delivered text read **“RUN INCOMPLETE —
tool-call budget exhausted”** — no Notion writes, no cover letters — so the day's backlog was executed by hand.
(b) `fad5a86979d4`'s 03:08 slot completed but ended `delivery_failed`: **`Thread 34795 not found`** — the same
dead thread the gateway's own 14:33:37 shutdown notice hit. → `runbooks/job-radar.md`
- **The radar's cover-letter pipeline changed the same morning.** The PDF is now printed from the **canonical
rendered HTML** via Chromium (`build_cover_letter_pdf_from_html.py`, created 10:37) instead of ReportLab, so a
download matches the on-screen letter; the generation default moved to **`deepseek-paid-smart`** (10:47) and
`job_dashboard_server.py`'s regenerate timeout went **120 s → 600 s** (10:48, server restarted).
→ `runbooks/job-radar.md`
- **Files:** new `incidents/2026-10-26-planned-restart-orphaned-cron-run.md`; updated
`incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md`,
`incidents/2026-10-16-omniroute-503-resource-pressure.md`, `runbooks/job-radar.md`,
`systems/scheduled-tasks.md`, `current-state.md`, `index.md`, `log.md`. **Nothing removed or archived.**
No secrets written.
- **Unresolved:** why a *planned* stop bypasses the drain deadline (`timed_out=True` at 1.35 s of a 30 s budget);
whether `failure_streak` should count the *unverified* row (one interruption currently reads streak 2); and the
**`Thread 34795`** delivery target needs re-pointing, or the Monday 03:08 digest will keep falling back to the
unthreaded chat.