wiki: nightly audit 2026-11-04 window — new incident (Job Radar + Newsletter Digest both killed by upstream stream failure; Stage 2 rebuilt handoff and delivered 5 roles + 1 letter); CT299 second freeze documented; CL_MODEL 402 caveat in job-radar runbook; scheduled-tasks/current-state/index/log updated

This commit is contained in:
hermes
2026-11-05 01:04:31 +08:00
parent 785fbc40b2
commit e4086ec18e
8 changed files with 361 additions and 35 deletions
+15 -4
View File
@@ -3,13 +3,13 @@ title: Daily Marketing Job Radar Operations and Recovery
type: runbook
status: active
created: 2026-07-27
updated: 2026-10-30
updated: 2026-11-05
review_after: 2026-11-13
verified_on: 2026-10-30
verified_on: 2026-11-05
version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2)
confidence: high
tags: [automation, cron, jobs, notion, dashboard, runbook]
sources: [live-cron-output, live-notion-api, live-dashboard-api, hermes-docs]
tags: [automation, cron, jobs, notion, dashboard, runbook, cover-letter, model-route]
sources: [live-cron-output, live-notion-api, live-dashboard-api, hermes-docs, live-script-inspection, live-endpoint-probe]
---
# Daily Marketing Job Radar Operations and Recovery
@@ -39,6 +39,17 @@ Cron jobs inherit the profile-level `fallback_providers` chain; Hermes does not
## Cover letters are non-negotiable
> **Model route caveat - verified 2026-11-05.** `regenerate_cover_letter.py` (line 82) reads
> `os.environ.get("CL_MODEL", "deepseek-paid-smart")`. On the 2026-11-04 run that default returned **HTTP 402
> (billing)**, and the run pinned **`gemini/gemini-3.5-flash`** via `CL_MODEL` to get the strong-match letter out.
> `grep` finds **no `CL_MODEL` in `~/.hermes/.env`** - the override existed only inside that process, so **the
> next run inherits the default again**. If a run logs 402/503 while generating letters, re-pin `CL_MODEL` for the
> process and confirm the letters rendered - do **not** ship a run without its cover letters. Note the same model
> name still serves other cron jobs successfully (e.g. the 23:59 nightly audit), and the endpoint itself is live:
> `omniroute:20128/v1/models` -> **401** and `100.78.63.20:20128/v1/models` -> **401** on 2026-11-05 (key-gated,
> not down). Treat a 402 as **route/time-specific until re-tested**, not as proof the route is dead.
A successful radar run includes the cover-letter stage.
- Audit **every Strong Match added today**, including pages created by an interrupted earlier run.