audit: 2026-10-26 nightly — planned restart orphaned a live cron run (new incident page); 503 burst fully closed; radar cover-letter PDF pipeline moved to Chromium-over-HTML
This commit is contained in:
+33
-3
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-27
|
||||
updated: 2026-10-13
|
||||
review_after: 2026-10-25
|
||||
verified_on: 2026-10-13
|
||||
updated: 2026-10-27
|
||||
review_after: 2026-11-10
|
||||
verified_on: 2026-10-27
|
||||
version_applies_to: Hermes Agent cron dca8482e4f76
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, dashboard, runbook]
|
||||
@@ -262,6 +262,36 @@ hermes cron run dca8482e4f76 # reproduce and capture the 403
|
||||
|
||||
Do **not** allow a silent fallback — the radar must surface a model failure rather than emit an empty "no jobs found" digest.
|
||||
|
||||
## Cover-letter PDF pipeline — printed from the dashboard HTML (changed 2026-10-26)
|
||||
|
||||
The PDF is now produced from the **canonical rendered HTML**, so a download matches the letter on screen.
|
||||
Previously ReportLab (`build_cover_letter_pdf.py`) laid the letter out independently of the template and
|
||||
produced a visibly different document — Anthony rejected exactly that mismatch.
|
||||
|
||||
- **Renderer:** `~/.hermes/scripts/build_cover_letter_pdf_from_html.py` (created 2026-10-26 10:37) drives
|
||||
**Chromium `--print-to-pdf`** over the canonical HTML and writes the PDF into
|
||||
`~/.hermes/radar-ref/drafts/`. **Do not go back to ReportLab.**
|
||||
- **Model:** the cover-letter generation default moved from the `cover-letter-writer` combo to
|
||||
**`deepseek-paid-smart`** on 2026-10-26 10:47 (`regenerate_cover_letter.py`, env override `CL_MODEL`).
|
||||
- **Dashboard timeout:** `job_dashboard_server.py` must allow **`timeout=600`** on the regenerate
|
||||
subprocess (set 2026-10-26 10:48, up from 120 s). One run is an LLM call + DOCX assembly + a Chromium
|
||||
print; it can exceed two minutes. The dashboard server was restarted to pick the change up.
|
||||
- All five artefacts (`.txt .json .html .docx .pdf`) must still exist and be nonempty, and the HTML must
|
||||
contain no unresolved `{{...}}` tokens — the HTML is now the source of truth the PDF follows.
|
||||
|
||||
### 2026-10-26 07:00 slot — `RUN INCOMPLETE`, and why the backlog was finished by hand
|
||||
|
||||
The 07:00 run started on schedule and closed **`completed` at 09:10**, but the report it delivered read
|
||||
**“RUN INCOMPLETE — tool-call budget exhausted”**: discovery and scoring consumed the run's tool budget, so
|
||||
the **Notion writes and the entire cover-letter stage never executed**. The same-day backlog was then run
|
||||
interactively (rows written for the outstanding strong matches, cover letters generated and verified).
|
||||
|
||||
- **A `completed` execution row is not a completed radar run.** Read the delivered text for the
|
||||
`RUN INCOMPLETE` marker before treating a slot as done.
|
||||
- Budget the browser-heavy stages (one tab in flight, close before opening the next — see the
|
||||
2026-10-12 hazard below) and keep headroom for the Notion + cover-letter writes, which sit last in the
|
||||
pipeline.
|
||||
|
||||
## Browser-layer hazard inside the run (2026-10-12)
|
||||
|
||||
This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove
|
||||
|
||||
Reference in New Issue
Block a user