Files
agent-estate-wiki/systems/browser-backend.md
T
hermes-audit ebf57146f1 audit(2026-10-13): 10-12 clean (174/0, 3rd day) but GitHub Trending Radar token DEAD (getMe 401) - supersedes the wrong 2026-10-06 'refuted' entry
Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline.

NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450).

Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
2026-10-13 00:19:15 +08:00

117 lines
6.8 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Browser Backend
type: system
status: active
created: 2026-07-22
updated: 2026-10-13
verified_on: 2026-10-13
confidence: high
tags: [browser, camofox, automation]
sources: [live-probe, config, session-history]
---
# Browser Backend
## Purpose
Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.
## Current Configuration (re-verified 2026-10-13)
- **Primary backend:** Camofox (Firefox-based, anti-fingerprint) — `systemd --user` unit `camofox-browser.service` **on the Hermes host itself** (the unit path is `/home/hermes/.config/systemd/user/…`, and the VNC route is named after CT460). The older `CT450` attribution below was not reproducible on 2026-10-13 — see the correction note.
- **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox`
- **REST API:** **`:9377`** — live-verified 2026-10-13 (`node server.js`, `CAMOFOX_PORT=9377`; `LISTEN *:9377`). **The `:9093` figure in earlier revisions of this page is stale** — nothing was listening on 9093 on 2026-10-13.
- **VNC viewer:** NovNC `websockify` on `127.0.0.1:6080` → `127.0.0.1:5900`, published as `https://ct460.kangaroo-eel.ts.net/vnc-camofox`
- **Steel Browser (CT201):** Still running but Hermes not connected — plugin disabled
## Camofox (local `systemd --user` service)
> ⚠ Earlier revisions titled this section "Camoufox (CT450)". Live-verified 2026-10-13: the service Hermes
> talks to is a `systemd --user` unit on the Hermes host (`/home/hermes/.config/systemd/user/camofox-browser.service`),
> reached at `localhost:9377`, with its VNC route published as `…/vnc-camofox`. Treat any "runs on CT450 /
> CT201" claim as unverified until re-probed.
- **Host:** the Hermes host itself (CT460) — `systemd --user` unit resolved on `hostname = hermes`, no network hop
- **Deployment:** Systemd service `camofox-browser.service`
- **Engine:** Camoufox (custom Firefox build with anti-detection fingerprinting)
- **Display:** Headed via Xvfb `:99` (1600×1200×24) + fluxbox window manager
- **Auth:** Persistent profile `anthony` — authenticated into Uber Eats, LinkedIn, etc.
- **Connection:** REST API **and** WebSocket bridge both at `localhost:9377` (see the port correction above)
- **Last verified:** Jul 30 — `browserConnected: true`, `sessions: 1`, `tabs: 1`, active 200s on scroll/navigate
### Access
| Endpoint | URL | Used by |
|---|---|---|
| REST API | `http://localhost:9377` | Hermes browser tools + direct REST (`/tabs`, `/start`, `/sessions`) |
| VNC (NovNC) | `https://ct460.kangaroo-eel.ts.net/vnc-camofox` | Visual session viewer (`websockify` `127.0.0.1:6080` → `127.0.0.1:5900`) |
> ⚠ **Port correction (2026-10-13):** earlier revisions listed the REST API on `:9093` and a separate CDP
> WebSocket on `:9377`. Live probe: the only Camofox listener is `node server.js` on `*:9377` (with
> `CAMOFOX_URL=http://localhost:9377` in the unit). `:9093` and `:9378` were both closed. `:9377` is the
> REST API **and** the WebSocket bridge — there is no separate 9093 service.
### Config
- `cloud_provider: camofox` in `~/.hermes/config.yaml`
- `CAMOFOX_URL=http://localhost:9377` and `CAMOFOX_API_KEY` in `~/.hermes/.env`
- Unit environment (`~/.config/systemd/user/camofox-browser.service`): `CAMOFOX_PORT=9377`,
`CAMOFOX_PROFILE_DIR=/home/hermes/.camofox/profiles`, `ENABLE_VNC=1`, `HANDLER_TIMEOUT_MS=90000`,
`NAVIGATE_TIMEOUT_MS=60000`, `SESSION_TIMEOUT_MS=600000`, `TAB_INACTIVITY_MS=300000`,
`BROWSER_IDLE_TIMEOUT_MS=300000`, `Restart=always`, `RestartSec=5`. *(The unit also carries an API key
value — deliberately not reproduced here.)*
- Profile: `anthony` (502913bf) — NOT `persist-anthony`. A clean public profile `jobradar` is used for job-board scraping.
## Resource containment (live-verified 2026-10-13)
The unit carries a drop-in **`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`**:
```
[Service]
MemoryHigh=2G
MemoryMax=3G
MemorySwapMax=1G
OOMPolicy=kill
```
**Observed ceiling event — 2026-10-12 07:01–07:23 (Daily Marketing Job Radar).** A long browser run drove
the service to the top of that containment window:
- Mid-run: `Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M,
swap peak: 1G)`, **383 tasks**, uptime 9 days. Swap fully consumed, no reclaimable headroom.
- Symptom: the REST `/tabs` endpoint and page evals began hanging (they look like ordinary tool timeouts).
- Recovery: restart `camofox-browser.service`, then work **one tab at a time, closing each before opening
the next**. Post-restart the host sat at ~2.1 GB of 14 GB.
**Diagnosis:** the limits held and `OOMPolicy=kill` never had to fire — the *workload* exceeded the cgroup,
not a leak. A green cron row therefore does **not** mean the browser layer was healthy; check
`systemctl --user status camofox-browser.service` (peak/swap/tasks) whenever a browser-heavy job runs long.
Restart (`systemctl --user restart camofox-browser.service`) is measurable roughly 16 h later as a healthy
baseline: `Memory: 16.5M` cgroup, `peak: 1.4G`, `swap peak: 234.5M`, 16 tasks, `browserConnected: false`
until the next session starts.
## Steel Browser (CT201) — Disabled
- **Status:** Still running on CT201, Hermes not connected since 2026-07-30
- **Plugin:** `browser-steel` at `/home/hermes/.hermes/plugins/browser-steel/` — disabled
- **CDP:** `http://100.96.244.39:9223`
- **REST API:** `http://100.96.244.39:3000`
- **UI:** `https://runtipi.kangaroo-eel.ts.net/steel`
- **See:** [[comparisons/steel-vs-camoufox]] for the full experiment story
## History
- **2026-07-22:** Camoufox primary, cua-driver bridge
- **2026-07-29:** Steel Browser deployed on CT201, became primary
- **2026-07-30:** Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary.
- **2026-08-02:** Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API)
- **2026-10-12 — containment ceiling hit under a browser-heavy cron run.** Daily Marketing Job Radar drove
`camofox-browser.service` to `2.5G` with swap fully consumed (`available: 0B`, 383 tasks, 9-day uptime);
`/tabs` and evals hung until the service was restarted at 07:23:50 and the job finished one tab at a time.
Workload problem, not a leak — limits (`MemoryHigh=2G`/`MemoryMax=3G`/`MemorySwapMax=1G`) held.
- **2026-10-13 — page corrected against a live probe.** REST port `:9093` → **`:9377`**; `:9378` (claimed
"Camoufox UI") does not exist; `CT450` host attribution replaced with the local `systemd --user` unit on
the Hermes host (CT460). Containment drop-in documented. Same day, the 07:01 Job Radar session was audited
end-to-end and its memory-pressure evidence added above.
## Related
- [[comparisons/steel-vs-camoufox]]
- [[concepts/browser-automation]]
- [[decisions/rollback-steel-to-camofox]]
- [[runbooks/restart-browser]]