audit(2026-10-13): 10-12 clean (174/0, 3rd day) but GitHub Trending Radar token DEAD (getMe 401) - supersedes the wrong 2026-10-06 'refuted' entry
Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline. NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450). Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
This commit is contained in:
+66
-17
@@ -3,8 +3,8 @@ title: Browser Backend
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-08-02
|
||||
verified_on: 2026-08-02
|
||||
updated: 2026-10-13
|
||||
verified_on: 2026-10-13
|
||||
confidence: high
|
||||
tags: [browser, camofox, automation]
|
||||
sources: [live-probe, config, session-history]
|
||||
@@ -15,36 +15,77 @@ sources: [live-probe, config, session-history]
|
||||
## Purpose
|
||||
Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.
|
||||
|
||||
## Current Configuration (as of 2026-08-02)
|
||||
- **Primary backend:** Camoufox (Firefox-based, anti-fingerprint) on CT450
|
||||
## Current Configuration (re-verified 2026-10-13)
|
||||
- **Primary backend:** Camofox (Firefox-based, anti-fingerprint) — `systemd --user` unit `camofox-browser.service` **on the Hermes host itself** (the unit path is `/home/hermes/.config/systemd/user/…`, and the VNC route is named after CT460). The older `CT450` attribution below was not reproducible on 2026-10-13 — see the correction note.
|
||||
- **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox`
|
||||
- **REST API:** `localhost:9093` (profile `anthony`, sessionKey `anthony`)
|
||||
- **CDP WebSocket:** `localhost:9377` (via Camoufox's Playwright bridge)
|
||||
- **VNC viewer:** `https://ct460.kangaroo-eel.ts.net/vnc-camofox` (port 6080)
|
||||
- **REST API:** **`:9377`** — live-verified 2026-10-13 (`node server.js`, `CAMOFOX_PORT=9377`; `LISTEN *:9377`). **The `:9093` figure in earlier revisions of this page is stale** — nothing was listening on 9093 on 2026-10-13.
|
||||
- **VNC viewer:** NovNC `websockify` on `127.0.0.1:6080` → `127.0.0.1:5900`, published as `https://ct460.kangaroo-eel.ts.net/vnc-camofox`
|
||||
- **Steel Browser (CT201):** Still running but Hermes not connected — plugin disabled
|
||||
|
||||
## Camoufox (CT450)
|
||||
- **Host:** CT450 (same host as Hermes — no network hop)
|
||||
## Camofox (local `systemd --user` service)
|
||||
|
||||
> ⚠ Earlier revisions titled this section "Camoufox (CT450)". Live-verified 2026-10-13: the service Hermes
|
||||
> talks to is a `systemd --user` unit on the Hermes host (`/home/hermes/.config/systemd/user/camofox-browser.service`),
|
||||
> reached at `localhost:9377`, with its VNC route published as `…/vnc-camofox`. Treat any "runs on CT450 /
|
||||
> CT201" claim as unverified until re-probed.
|
||||
|
||||
- **Host:** the Hermes host itself (CT460) — `systemd --user` unit resolved on `hostname = hermes`, no network hop
|
||||
- **Deployment:** Systemd service `camofox-browser.service`
|
||||
- **Engine:** Camoufox (custom Firefox build with anti-detection fingerprinting)
|
||||
- **Display:** Headed via Xvfb `:99` (1600×1200×24) + fluxbox window manager
|
||||
- **Auth:** Persistent profile `anthony` — authenticated into Uber Eats, LinkedIn, etc.
|
||||
- **Connection:** REST API at `localhost:9093` for session management, CDP WebSocket at `localhost:9377` for browser tools
|
||||
- **Connection:** REST API **and** WebSocket bridge both at `localhost:9377` (see the port correction above)
|
||||
- **Last verified:** Jul 30 — `browserConnected: true`, `sessions: 1`, `tabs: 1`, active 200s on scroll/navigate
|
||||
|
||||
### Access
|
||||
| Endpoint | URL | Used by |
|
||||
|---|---|---|
|
||||
| REST API | `http://localhost:9093` | Hermes browser tools (navigate, click, snapshot) |
|
||||
| CDP WebSocket | `ws://localhost:9377` | Direct CDP control |
|
||||
| VNC | `https://ct460.kangaroo-eel.ts.net/vnc-camofox` | Visual session viewer |
|
||||
| Camoufox UI | `http://localhost:9378` | Internal management |
|
||||
| REST API | `http://localhost:9377` | Hermes browser tools + direct REST (`/tabs`, `/start`, `/sessions`) |
|
||||
| VNC (NovNC) | `https://ct460.kangaroo-eel.ts.net/vnc-camofox` | Visual session viewer (`websockify` `127.0.0.1:6080` → `127.0.0.1:5900`) |
|
||||
|
||||
> ⚠ **Port correction (2026-10-13):** earlier revisions listed the REST API on `:9093` and a separate CDP
|
||||
> WebSocket on `:9377`. Live probe: the only Camofox listener is `node server.js` on `*:9377` (with
|
||||
> `CAMOFOX_URL=http://localhost:9377` in the unit). `:9093` and `:9378` were both closed. `:9377` is the
|
||||
> REST API **and** the WebSocket bridge — there is no separate 9093 service.
|
||||
|
||||
### Config
|
||||
- `cloud_provider: camofox` in `~/.hermes/config.yaml`
|
||||
- `CAMOFOX_URL=http://localhost:9093` in `~/.hermes/.env`
|
||||
- `CAMOFOX_API_KEY` in `~/.hermes/.env`
|
||||
- Profile: `anthony` (502913bf) — NOT `persist-anthony`
|
||||
- `CAMOFOX_URL=http://localhost:9377` and `CAMOFOX_API_KEY` in `~/.hermes/.env`
|
||||
- Unit environment (`~/.config/systemd/user/camofox-browser.service`): `CAMOFOX_PORT=9377`,
|
||||
`CAMOFOX_PROFILE_DIR=/home/hermes/.camofox/profiles`, `ENABLE_VNC=1`, `HANDLER_TIMEOUT_MS=90000`,
|
||||
`NAVIGATE_TIMEOUT_MS=60000`, `SESSION_TIMEOUT_MS=600000`, `TAB_INACTIVITY_MS=300000`,
|
||||
`BROWSER_IDLE_TIMEOUT_MS=300000`, `Restart=always`, `RestartSec=5`. *(The unit also carries an API key
|
||||
value — deliberately not reproduced here.)*
|
||||
- Profile: `anthony` (502913bf) — NOT `persist-anthony`. A clean public profile `jobradar` is used for job-board scraping.
|
||||
|
||||
## Resource containment (live-verified 2026-10-13)
|
||||
|
||||
The unit carries a drop-in **`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`**:
|
||||
|
||||
```
|
||||
[Service]
|
||||
MemoryHigh=2G
|
||||
MemoryMax=3G
|
||||
MemorySwapMax=1G
|
||||
OOMPolicy=kill
|
||||
```
|
||||
|
||||
**Observed ceiling event — 2026-10-12 07:01–07:23 (Daily Marketing Job Radar).** A long browser run drove
|
||||
the service to the top of that containment window:
|
||||
|
||||
- Mid-run: `Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M,
|
||||
swap peak: 1G)`, **383 tasks**, uptime 9 days. Swap fully consumed, no reclaimable headroom.
|
||||
- Symptom: the REST `/tabs` endpoint and page evals began hanging (they look like ordinary tool timeouts).
|
||||
- Recovery: restart `camofox-browser.service`, then work **one tab at a time, closing each before opening
|
||||
the next**. Post-restart the host sat at ~2.1 GB of 14 GB.
|
||||
|
||||
**Diagnosis:** the limits held and `OOMPolicy=kill` never had to fire — the *workload* exceeded the cgroup,
|
||||
not a leak. A green cron row therefore does **not** mean the browser layer was healthy; check
|
||||
`systemctl --user status camofox-browser.service` (peak/swap/tasks) whenever a browser-heavy job runs long.
|
||||
|
||||
Restart (`systemctl --user restart camofox-browser.service`) is measurable roughly 16 h later as a healthy
|
||||
baseline: `Memory: 16.5M` cgroup, `peak: 1.4G`, `swap peak: 234.5M`, 16 tasks, `browserConnected: false`
|
||||
until the next session starts.
|
||||
|
||||
## Steel Browser (CT201) — Disabled
|
||||
- **Status:** Still running on CT201, Hermes not connected since 2026-07-30
|
||||
@@ -59,6 +100,14 @@ Browser automation for Hermes — web search, form filling, screenshot, CDP cont
|
||||
- **2026-07-29:** Steel Browser deployed on CT201, became primary
|
||||
- **2026-07-30:** Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary.
|
||||
- **2026-08-02:** Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API)
|
||||
- **2026-10-12 — containment ceiling hit under a browser-heavy cron run.** Daily Marketing Job Radar drove
|
||||
`camofox-browser.service` to `2.5G` with swap fully consumed (`available: 0B`, 383 tasks, 9-day uptime);
|
||||
`/tabs` and evals hung until the service was restarted at 07:23:50 and the job finished one tab at a time.
|
||||
Workload problem, not a leak — limits (`MemoryHigh=2G`/`MemoryMax=3G`/`MemorySwapMax=1G`) held.
|
||||
- **2026-10-13 — page corrected against a live probe.** REST port `:9093` → **`:9377`**; `:9378` (claimed
|
||||
"Camoufox UI") does not exist; `CT450` host attribution replaced with the local `systemd --user` unit on
|
||||
the Hermes host (CT460). Containment drop-in documented. Same day, the 07:01 Job Radar session was audited
|
||||
end-to-end and its memory-pressure evidence added above.
|
||||
|
||||
## Related
|
||||
- [[comparisons/steel-vs-camoufox]]
|
||||
|
||||
Reference in New Issue
Block a user