audit(2026-10-13): 10-12 clean (174/0, 3rd day) but GitHub Trending Radar token DEAD (getMe 401) - supersedes the wrong 2026-10-06 'refuted' entry

Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline.

NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450).

Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
This commit is contained in:
hermes-audit
2026-10-13 00:19:15 +08:00
parent 0eee5f101e
commit ebf57146f1
10 changed files with 304 additions and 65 deletions
+21 -2
View File
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery
type: runbook
status: active
created: 2026-07-27
updated: 2026-07-28
updated: 2026-10-13
review_after: 2026-10-25
verified_on: 2026-07-28
verified_on: 2026-10-13
version_applies_to: Hermes Agent cron dca8482e4f76
confidence: high
tags: [automation, cron, jobs, notion, dashboard, runbook]
@@ -262,8 +262,27 @@ hermes cron run dca8482e4f76 # reproduce and capture the 403
Do **not** allow a silent fallback — the radar must surface a model failure rather than emit an empty "no jobs found" digest.
## Browser-layer hazard inside the run (2026-10-12)
This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove
`camofox-browser.service` to its memory ceiling — `2.5G`, swap fully consumed (`available: 0B`), 383 tasks —
after which REST `/tabs` and page evals hung in a way indistinguishable from ordinary tool timeouts. The
run recovered by restarting the service at **07:23:50** and finishing **one tab at a time with an immediate
close**, and it delivered normally.
Operational implications for this runbook:
- A **green cron row is not proof the browser layer was healthy** — check
`systemctl --user status camofox-browser.service` (peak / swap / task count) after any long run.
- Extraction must be bounded: one tab in flight, close before opening the next. Multi-tab pre-fetch is what
pushed it over.
- Limits live in `~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`
(`MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G`, `OOMPolicy=kill`). Raise them only if the host has
real headroom — full detail in [[systems/browser-backend]].
## Related
- [[systems/scheduled-tasks]]
- [[systems/browser-backend]]
- [[infrastructure/tailscale]]
- [[systems/model-providers]]