audit: 2026-10-26 nightly — planned restart orphaned a live cron run (new incident page); 503 burst fully closed; radar cover-letter PDF pipeline moved to Chromium-over-HTML
This commit is contained in:
@@ -1408,3 +1408,49 @@ The two top-level **stale** files are `current-state.md` and
|
||||
`reports/communities-wa-social-media-audit-2026.md` — both dated historical records
|
||||
rather than drift. The vault repo is **local-only** (no remotes); it was committed,
|
||||
not pushed.
|
||||
|
||||
## [2026-10-27] audit | Nightly wiki audit — window 2026-10-26 00:00 → 2026-10-27 00:5x AWST: a planned restart orphaned a live cron run
|
||||
|
||||
- **A *planned* gateway restart orphaned a cron run — the second instance of the 2026-10-20 orphan class, and
|
||||
this time it was loud.** `gateway.log`: `Received SIGTERM as a planned gateway stop` at **14:33:35**
|
||||
(`under_systemd=yes`, `parent_name=systemd`, `loadavg_1m=2.15`). The drain then logged
|
||||
`Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True` → `interrupting remaining work`,
|
||||
and at 14:34:29 `Skipping .clean_shutdown marker — drain timed out with interrupted agents`. The victim was
|
||||
**`ea79a2107e36` (Mnemosyne Health Monitor)**, claimed **12:01:13** — **2 h 34 m** in. It closed **14:35:59
|
||||
`failed`** on a string this wiki had not seen: **`RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its
|
||||
durable fire claim ownership`**. Unlike the 10-20 orphan it raised **two `cron_incidents` rows** (14:33:37
|
||||
`ea79a2_c3ab2d784ba3` *"Cron execution unverified; reconcile `admissions/724a69108b…json`"*; 14:35:59
|
||||
`ea79a2_a4661554a3c2`, the RuntimeError) and **delivered** the failure. → new
|
||||
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md`
|
||||
- **The Mnemosyne Watchdog did its job.** `96eb95ce12f2` red-flagged twice (14:46:12, 15:01:15,
|
||||
`Script exited with code 1`) reacting to the same log `ERROR`, then **self-resolved at 15:16:12** when the
|
||||
line aged out of its 15-minute window. Both rows (`96eb95_0c89bd373a0a`, `96eb95_e32174134546`) are closed.
|
||||
- **Window tally and fleet:** 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are
|
||||
this one interruption** (Health Monitor + the two watchdog reds); 2026-10-25 settled **172 / 2**. Fleet
|
||||
unchanged: **28 jobs, 25 enabled / 3 disabled**. **Non-clean `last_status` 5 → 4** (`c0a9fa314542` and
|
||||
`08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as `error`, streak 2). **Unclosed incident rows 2 → 3**.
|
||||
**Pins unchanged.** One gateway start (14:35:18, teardown 54.06 s). **17 sessions in window** (15 `cron`,
|
||||
1 `ios`, 1 `telegram`).
|
||||
- **One incident row closed — the 2026-10-16 503 burst is fully shut.** `c0a9fa_764558b5ae83` **auto-closed
|
||||
09:13:00** when the radar's 09:00 slot ran clean; no execution row carries the resource-pressure string on
|
||||
10-25 or 10-26. Status stays `monitoring` (the single-node exposure is unchanged). Updated
|
||||
`incidents/2026-10-16-omniroute-503-resource-pressure.md`.
|
||||
- **⚠ Two quiet degradations: a completed run that delivered nothing useful, and a broken delivery thread.**
|
||||
(a) The radar's 07:00 slot closed **`completed` at 09:10**, but its delivered text read **“RUN INCOMPLETE —
|
||||
tool-call budget exhausted”** — no Notion writes, no cover letters — so the day's backlog was executed by hand.
|
||||
(b) `fad5a86979d4`'s 03:08 slot completed but ended `delivery_failed`: **`Thread 34795 not found`** — the same
|
||||
dead thread the gateway's own 14:33:37 shutdown notice hit. → `runbooks/job-radar.md`
|
||||
- **The radar's cover-letter pipeline changed the same morning.** The PDF is now printed from the **canonical
|
||||
rendered HTML** via Chromium (`build_cover_letter_pdf_from_html.py`, created 10:37) instead of ReportLab, so a
|
||||
download matches the on-screen letter; the generation default moved to **`deepseek-paid-smart`** (10:47) and
|
||||
`job_dashboard_server.py`'s regenerate timeout went **120 s → 600 s** (10:48, server restarted).
|
||||
→ `runbooks/job-radar.md`
|
||||
- **Files:** new `incidents/2026-10-26-planned-restart-orphaned-cron-run.md`; updated
|
||||
`incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md`,
|
||||
`incidents/2026-10-16-omniroute-503-resource-pressure.md`, `runbooks/job-radar.md`,
|
||||
`systems/scheduled-tasks.md`, `current-state.md`, `index.md`, `log.md`. **Nothing removed or archived.**
|
||||
No secrets written.
|
||||
- **Unresolved:** why a *planned* stop bypasses the drain deadline (`timed_out=True` at 1.35 s of a 30 s budget);
|
||||
whether `failure_streak` should count the *unverified* row (one interruption currently reads streak 2); and the
|
||||
**`Thread 34795`** delivery target needs re-pointing, or the Monday 03:08 digest will keep falling back to the
|
||||
unthreaded chat.
|
||||
|
||||
Reference in New Issue
Block a user