audit: 2026-10-26 nightly — planned restart orphaned a live cron run (new incident page); 503 burst fully closed; radar cover-letter PDF pipeline moved to Chromium-over-HTML

This commit is contained in:
Tony0410
2026-10-27 00:11:50 +08:00
parent 3f796dc3b9
commit a2dd0bd650
8 changed files with 236 additions and 20 deletions
+33 -3
View File
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery
type: runbook
status: active
created: 2026-07-27
updated: 2026-10-13
review_after: 2026-10-25
verified_on: 2026-10-13
updated: 2026-10-27
review_after: 2026-11-10
verified_on: 2026-10-27
version_applies_to: Hermes Agent cron dca8482e4f76
confidence: high
tags: [automation, cron, jobs, notion, dashboard, runbook]
@@ -262,6 +262,36 @@ hermes cron run dca8482e4f76 # reproduce and capture the 403
Do **not** allow a silent fallback — the radar must surface a model failure rather than emit an empty "no jobs found" digest.
## Cover-letter PDF pipeline — printed from the dashboard HTML (changed 2026-10-26)
The PDF is now produced from the **canonical rendered HTML**, so a download matches the letter on screen.
Previously ReportLab (`build_cover_letter_pdf.py`) laid the letter out independently of the template and
produced a visibly different document — Anthony rejected exactly that mismatch.
- **Renderer:** `~/.hermes/scripts/build_cover_letter_pdf_from_html.py` (created 2026-10-26 10:37) drives
**Chromium `--print-to-pdf`** over the canonical HTML and writes the PDF into
`~/.hermes/radar-ref/drafts/`. **Do not go back to ReportLab.**
- **Model:** the cover-letter generation default moved from the `cover-letter-writer` combo to
**`deepseek-paid-smart`** on 2026-10-26 10:47 (`regenerate_cover_letter.py`, env override `CL_MODEL`).
- **Dashboard timeout:** `job_dashboard_server.py` must allow **`timeout=600`** on the regenerate
subprocess (set 2026-10-26 10:48, up from 120 s). One run is an LLM call + DOCX assembly + a Chromium
print; it can exceed two minutes. The dashboard server was restarted to pick the change up.
- All five artefacts (`.txt .json .html .docx .pdf`) must still exist and be nonempty, and the HTML must
contain no unresolved `{{...}}` tokens — the HTML is now the source of truth the PDF follows.
### 2026-10-26 07:00 slot — `RUN INCOMPLETE`, and why the backlog was finished by hand
The 07:00 run started on schedule and closed **`completed` at 09:10**, but the report it delivered read
**“RUN INCOMPLETE — tool-call budget exhausted”**: discovery and scoring consumed the run's tool budget, so
the **Notion writes and the entire cover-letter stage never executed**. The same-day backlog was then run
interactively (rows written for the outstanding strong matches, cover letters generated and verified).
- **A `completed` execution row is not a completed radar run.** Read the delivered text for the
`RUN INCOMPLETE` marker before treating a slot as done.
- Budget the browser-heavy stages (one tab in flight, close before opening the next — see the
2026-10-12 hazard below) and keep headroom for the Notion + cover-letter writes, which sit last in the
pipeline.
## Browser-layer hazard inside the run (2026-10-12)
This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove