audit(2026-10-13): 10-12 clean (174/0, 3rd day) but GitHub Trending Radar token DEAD (getMe 401) - supersedes the wrong 2026-10-06 'refuted' entry
Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline. NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450). Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
This commit is contained in:
+21
-2
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-27
|
||||
updated: 2026-07-28
|
||||
updated: 2026-10-13
|
||||
review_after: 2026-10-25
|
||||
verified_on: 2026-07-28
|
||||
verified_on: 2026-10-13
|
||||
version_applies_to: Hermes Agent cron dca8482e4f76
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, dashboard, runbook]
|
||||
@@ -262,8 +262,27 @@ hermes cron run dca8482e4f76 # reproduce and capture the 403
|
||||
|
||||
Do **not** allow a silent fallback — the radar must surface a model failure rather than emit an empty "no jobs found" digest.
|
||||
|
||||
## Browser-layer hazard inside the run (2026-10-12)
|
||||
|
||||
This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove
|
||||
`camofox-browser.service` to its memory ceiling — `2.5G`, swap fully consumed (`available: 0B`), 383 tasks —
|
||||
after which REST `/tabs` and page evals hung in a way indistinguishable from ordinary tool timeouts. The
|
||||
run recovered by restarting the service at **07:23:50** and finishing **one tab at a time with an immediate
|
||||
close**, and it delivered normally.
|
||||
|
||||
Operational implications for this runbook:
|
||||
|
||||
- A **green cron row is not proof the browser layer was healthy** — check
|
||||
`systemctl --user status camofox-browser.service` (peak / swap / task count) after any long run.
|
||||
- Extraction must be bounded: one tab in flight, close before opening the next. Multi-tab pre-fetch is what
|
||||
pushed it over.
|
||||
- Limits live in `~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`
|
||||
(`MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G`, `OOMPolicy=kill`). Raise them only if the host has
|
||||
real headroom — full detail in [[systems/browser-backend]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/browser-backend]]
|
||||
- [[infrastructure/tailscale]]
|
||||
- [[systems/model-providers]]
|
||||
|
||||
Reference in New Issue
Block a user