audit(2026-10-13): 10-12 clean (174/0, 3rd day) but GitHub Trending Radar token DEAD (getMe 401) - supersedes the wrong 2026-10-06 'refuted' entry

Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline.

NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450).

Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
This commit is contained in:
hermes-audit
2026-10-13 00:19:15 +08:00
parent 0eee5f101e
commit ebf57146f1
10 changed files with 304 additions and 65 deletions
+66 -17
View File
@@ -3,8 +3,8 @@ title: Browser Backend
type: system
status: active
created: 2026-07-22
updated: 2026-08-02
verified_on: 2026-08-02
updated: 2026-10-13
verified_on: 2026-10-13
confidence: high
tags: [browser, camofox, automation]
sources: [live-probe, config, session-history]
@@ -15,36 +15,77 @@ sources: [live-probe, config, session-history]
## Purpose
Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.
## Current Configuration (as of 2026-08-02)
- **Primary backend:** Camoufox (Firefox-based, anti-fingerprint) on CT450
## Current Configuration (re-verified 2026-10-13)
- **Primary backend:** Camofox (Firefox-based, anti-fingerprint) — `systemd --user` unit `camofox-browser.service` **on the Hermes host itself** (the unit path is `/home/hermes/.config/systemd/user/…`, and the VNC route is named after CT460). The older `CT450` attribution below was not reproducible on 2026-10-13 — see the correction note.
- **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox`
- **REST API:** `localhost:9093` (profile `anthony`, sessionKey `anthony`)
- **CDP WebSocket:** `localhost:9377` (via Camoufox's Playwright bridge)
- **VNC viewer:** `https://ct460.kangaroo-eel.ts.net/vnc-camofox` (port 6080)
- **REST API:** **`:9377`** — live-verified 2026-10-13 (`node server.js`, `CAMOFOX_PORT=9377`; `LISTEN *:9377`). **The `:9093` figure in earlier revisions of this page is stale** — nothing was listening on 9093 on 2026-10-13.
- **VNC viewer:** NovNC `websockify` on `127.0.0.1:6080` → `127.0.0.1:5900`, published as `https://ct460.kangaroo-eel.ts.net/vnc-camofox`
- **Steel Browser (CT201):** Still running but Hermes not connected — plugin disabled
## Camoufox (CT450)
- **Host:** CT450 (same host as Hermes — no network hop)
## Camofox (local `systemd --user` service)
> ⚠ Earlier revisions titled this section "Camoufox (CT450)". Live-verified 2026-10-13: the service Hermes
> talks to is a `systemd --user` unit on the Hermes host (`/home/hermes/.config/systemd/user/camofox-browser.service`),
> reached at `localhost:9377`, with its VNC route published as `…/vnc-camofox`. Treat any "runs on CT450 /
> CT201" claim as unverified until re-probed.
- **Host:** the Hermes host itself (CT460) — `systemd --user` unit resolved on `hostname = hermes`, no network hop
- **Deployment:** Systemd service `camofox-browser.service`
- **Engine:** Camoufox (custom Firefox build with anti-detection fingerprinting)
- **Display:** Headed via Xvfb `:99` (1600×1200×24) + fluxbox window manager
- **Auth:** Persistent profile `anthony` — authenticated into Uber Eats, LinkedIn, etc.
- **Connection:** REST API at `localhost:9093` for session management, CDP WebSocket at `localhost:9377` for browser tools
- **Connection:** REST API **and** WebSocket bridge both at `localhost:9377` (see the port correction above)
- **Last verified:** Jul 30 — `browserConnected: true`, `sessions: 1`, `tabs: 1`, active 200s on scroll/navigate
### Access
| Endpoint | URL | Used by |
|---|---|---|
| REST API | `http://localhost:9093` | Hermes browser tools (navigate, click, snapshot) |
| CDP WebSocket | `ws://localhost:9377` | Direct CDP control |
| VNC | `https://ct460.kangaroo-eel.ts.net/vnc-camofox` | Visual session viewer |
| Camoufox UI | `http://localhost:9378` | Internal management |
| REST API | `http://localhost:9377` | Hermes browser tools + direct REST (`/tabs`, `/start`, `/sessions`) |
| VNC (NovNC) | `https://ct460.kangaroo-eel.ts.net/vnc-camofox` | Visual session viewer (`websockify` `127.0.0.1:6080` → `127.0.0.1:5900`) |
> ⚠ **Port correction (2026-10-13):** earlier revisions listed the REST API on `:9093` and a separate CDP
> WebSocket on `:9377`. Live probe: the only Camofox listener is `node server.js` on `*:9377` (with
> `CAMOFOX_URL=http://localhost:9377` in the unit). `:9093` and `:9378` were both closed. `:9377` is the
> REST API **and** the WebSocket bridge — there is no separate 9093 service.
### Config
- `cloud_provider: camofox` in `~/.hermes/config.yaml`
- `CAMOFOX_URL=http://localhost:9093` in `~/.hermes/.env`
- `CAMOFOX_API_KEY` in `~/.hermes/.env`
- Profile: `anthony` (502913bf) — NOT `persist-anthony`
- `CAMOFOX_URL=http://localhost:9377` and `CAMOFOX_API_KEY` in `~/.hermes/.env`
- Unit environment (`~/.config/systemd/user/camofox-browser.service`): `CAMOFOX_PORT=9377`,
`CAMOFOX_PROFILE_DIR=/home/hermes/.camofox/profiles`, `ENABLE_VNC=1`, `HANDLER_TIMEOUT_MS=90000`,
`NAVIGATE_TIMEOUT_MS=60000`, `SESSION_TIMEOUT_MS=600000`, `TAB_INACTIVITY_MS=300000`,
`BROWSER_IDLE_TIMEOUT_MS=300000`, `Restart=always`, `RestartSec=5`. *(The unit also carries an API key
value — deliberately not reproduced here.)*
- Profile: `anthony` (502913bf) — NOT `persist-anthony`. A clean public profile `jobradar` is used for job-board scraping.
## Resource containment (live-verified 2026-10-13)
The unit carries a drop-in **`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`**:
```
[Service]
MemoryHigh=2G
MemoryMax=3G
MemorySwapMax=1G
OOMPolicy=kill
```
**Observed ceiling event — 2026-10-12 07:01–07:23 (Daily Marketing Job Radar).** A long browser run drove
the service to the top of that containment window:
- Mid-run: `Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M,
swap peak: 1G)`, **383 tasks**, uptime 9 days. Swap fully consumed, no reclaimable headroom.
- Symptom: the REST `/tabs` endpoint and page evals began hanging (they look like ordinary tool timeouts).
- Recovery: restart `camofox-browser.service`, then work **one tab at a time, closing each before opening
the next**. Post-restart the host sat at ~2.1 GB of 14 GB.
**Diagnosis:** the limits held and `OOMPolicy=kill` never had to fire — the *workload* exceeded the cgroup,
not a leak. A green cron row therefore does **not** mean the browser layer was healthy; check
`systemctl --user status camofox-browser.service` (peak/swap/tasks) whenever a browser-heavy job runs long.
Restart (`systemctl --user restart camofox-browser.service`) is measurable roughly 16 h later as a healthy
baseline: `Memory: 16.5M` cgroup, `peak: 1.4G`, `swap peak: 234.5M`, 16 tasks, `browserConnected: false`
until the next session starts.
## Steel Browser (CT201) — Disabled
- **Status:** Still running on CT201, Hermes not connected since 2026-07-30
@@ -59,6 +100,14 @@ Browser automation for Hermes — web search, form filling, screenshot, CDP cont
- **2026-07-29:** Steel Browser deployed on CT201, became primary
- **2026-07-30:** Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary.
- **2026-08-02:** Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API)
- **2026-10-12 — containment ceiling hit under a browser-heavy cron run.** Daily Marketing Job Radar drove
`camofox-browser.service` to `2.5G` with swap fully consumed (`available: 0B`, 383 tasks, 9-day uptime);
`/tabs` and evals hung until the service was restarted at 07:23:50 and the job finished one tab at a time.
Workload problem, not a leak — limits (`MemoryHigh=2G`/`MemoryMax=3G`/`MemorySwapMax=1G`) held.
- **2026-10-13 — page corrected against a live probe.** REST port `:9093` → **`:9377`**; `:9378` (claimed
"Camoufox UI") does not exist; `CT450` host attribution replaced with the local `systemd --user` unit on
the Hermes host (CT460). Containment drop-in documented. Same day, the 07:01 Job Radar session was audited
end-to-end and its memory-pressure evidence added above.
## Related
- [[comparisons/steel-vs-camoufox]]