audit: 2026-10-26 nightly — planned restart orphaned a live cron run (new incident page); 503 burst fully closed; radar cover-letter PDF pipeline moved to Chromium-over-HTML

This commit is contained in:
Tony0410
2026-10-27 00:11:50 +08:00
parent 3f796dc3b9
commit a2dd0bd650
8 changed files with 236 additions and 20 deletions
+32 -5
View File
@@ -3,18 +3,45 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-10-25
verified_on: 2026-10-25
review_after: 2026-11-01
updated: 2026-10-27
verified_on: 2026-10-27
review_after: 2026-11-03
confidence: high
tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
---
# Current State
Updated: 2026-10-25
Updated: 2026-10-27
## Recent Changes (2026-10-25, nightly audit; live probes 2026-10-25 00:1x–00:5x AWST)
## Recent Changes (2026-10-27, nightly audit; live probes 2026-10-27 00:1x–00:5x AWST)
- **A *planned* gateway restart still orphaned a live cron run — and this time it alerted loudly.** A planned
`SIGTERM` at **2026-10-26 14:33:35** produced a shutdown drain that reported **`timed_out=True` after
just 1.35 s** of its 30 s budget and interrupted `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot)
**2 h 34 m** in. It closed **`failed`** on a **new** string — `lost its durable fire claim ownership` —
against the 2026-10-20 orphan's silent `unknown`, and raised **two** `cron_incidents` rows (14:33:37
*unverified / reconcile admission*, 14:35:59 *fire claim*) that are still `alerted`; the failure **was
delivered** at 14:35:59. Two Mnemosyne Watchdog reds followed and **self-resolved at 15:16:12**. Window
2026-10-26 settled **169 / 3** and **all three failures are this one fault** — so the fleet's failure
count is one interruption, not a systemic turn. Fleet unchanged at **28 jobs (25 enabled / 3 disabled)**;
**non-clean `last_status` 5 → 4**; **unclosed incident rows 2 → 3**; **pins unchanged**. →
new page [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]
- **One incident row closed, and the 2026-10-16 503 burst is fully shut.** `c0a9fa_764558b5ae83`
(GitHub Trending Radar) **auto-closed 09:13:00** when its 09:00 slot ran clean — the last open row of the
resource-pressure fault → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. No execution row
carries the 503 string on 10-25 or 10-26. The job stays **`monitoring`**: single-node exposure unchanged.
- **Two quiet degradations worth naming.** `fad5a86979d4`'s 03:08 slot **completed but delivered
`delivery_failed`** — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead
thread at 14:33:37); and the **radar's 07:00 slot closed `completed` at 09:10 while delivering
“RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters, batch finished by
hand. That morning the cover-letter PDF pipeline moved to **Chromium over the canonical HTML** with model
**`deepseek-paid-smart`** and a **600 s** dashboard timeout → [[runbooks/job-radar]]
- **Also updated:** `incidents/2026-10-26-planned-restart-orphaned-cron-run.md` (new),
`incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md` (recurrence), `incidents/2026-10-16-omniroute-503-resource-pressure.md`,
`runbooks/job-radar.md`, `systems/scheduled-tasks.md`, `index.md`, `log.md`. **Nothing removed or archived.**
## Recent Changes (2026-10-25, nightly audit; live probes 2026-10-25 00:1x–00:5x AWST) *(superseded by the 2026-10-27 block above)*
- **A third consecutive clean cron day — and an off-cron fault that is not the fleet's.** Window
2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at