--- title: Browser Backend type: system status: active created: 2026-07-22 updated: 2026-11-08 verified_on: 2026-11-08 confidence: high tags: [browser, camofox, automation] sources: [live-probe, config, session-history] --- # Browser Backend ## Purpose Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping. ## Current Configuration (re-verified 2026-10-13) > **2026-11-07/08 — the browser backend now ships as a plugin, not a core feature.** The Hermes > `0.22.5 → 0.23.0` update installed **`camofox` v1.1.0** as an **enabled plugin** at > `~/.hermes/plugins/camofox/` (`plugin.yaml`, `camofox_tools.py`, `camofox_policy.py`, > `camofox_vault.py`, `camofox_cli.py`), confirmed live by `hermes plugins list` on 2026-11-07. > No `browser.*` key was changed in this window — the sections below still describe the running > configuration ([[systems/hermes-agent]]). - **Primary backend:** Camofox (Firefox-based, anti-fingerprint) — `systemd --user` unit `camofox-browser.service` **on the Hermes host itself** (the unit path is `/home/hermes/.config/systemd/user/…`, and the VNC route is named after CT460). The older `CT450` attribution below was not reproducible on 2026-10-13 — see the correction note. - **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox` - **REST API:** **`:9377`** — live-verified 2026-10-13 (`node server.js`, `CAMOFOX_PORT=9377`; `LISTEN *:9377`). **The `:9093` figure in earlier revisions of this page is stale** — nothing was listening on 9093 on 2026-10-13. - **VNC viewer:** NovNC `websockify` on `127.0.0.1:6080` → `127.0.0.1:5900`, published as `https://ct460.kangaroo-eel.ts.net/vnc-camofox` - **Steel Browser (CT201):** **Fully removed on 2026-10-29** — plugin deleted, CT201 deployment torn down, Serve routes deleted. **Camofox is the sole backend.** See the 2026-10-29 correction below. ## Camofox (local `systemd --user` service) > ⚠ Earlier revisions titled this section "Camoufox (CT450)". Live-verified 2026-10-13: the service Hermes > talks to is a `systemd --user` unit on the Hermes host (`/home/hermes/.config/systemd/user/camofox-browser.service`), > reached at `localhost:9377`, with its VNC route published as `…/vnc-camofox`. Treat any "runs on CT450 / > CT201" claim as unverified until re-probed. - **Host:** the Hermes host itself (CT460) — `systemd --user` unit resolved on `hostname = hermes`, no network hop - **Deployment:** Systemd service `camofox-browser.service` - **Engine:** Camoufox (custom Firefox build with anti-detection fingerprinting) - **Display:** Headed via Xvfb `:99` (1600×1200×24) + fluxbox window manager - **Auth:** Persistent profile `anthony` — authenticated into Uber Eats, LinkedIn, etc. - **Connection:** REST API **and** WebSocket bridge both at `localhost:9377` (see the port correction above) - **Last verified:** Jul 30 — `browserConnected: true`, `sessions: 1`, `tabs: 1`, active 200s on scroll/navigate ### Access | Endpoint | URL | Used by | |---|---|---| | REST API | `http://localhost:9377` | Hermes browser tools + direct REST (`/tabs`, `/start`, `/sessions`) | | VNC (NovNC) | `https://ct460.kangaroo-eel.ts.net/vnc-camofox` | Visual session viewer (`websockify` `127.0.0.1:6080` → `127.0.0.1:5900`) | > ⚠ **Port correction (2026-10-13):** earlier revisions listed the REST API on `:9093` and a separate CDP > WebSocket on `:9377`. Live probe: the only Camofox listener is `node server.js` on `*:9377` (with > `CAMOFOX_URL=http://localhost:9377` in the unit). `:9093` and `:9378` were both closed. `:9377` is the > REST API **and** the WebSocket bridge — there is no separate 9093 service. ### Config - `cloud_provider: camofox` in `~/.hermes/config.yaml` - `CAMOFOX_URL=http://localhost:9377` and `CAMOFOX_API_KEY` in `~/.hermes/.env` - Unit environment (`~/.config/systemd/user/camofox-browser.service`): `CAMOFOX_PORT=9377`, `CAMOFOX_PROFILE_DIR=/home/hermes/.camofox/profiles`, `ENABLE_VNC=1`, `HANDLER_TIMEOUT_MS=90000`, `NAVIGATE_TIMEOUT_MS=60000`, `SESSION_TIMEOUT_MS=600000`, `TAB_INACTIVITY_MS=300000`, `BROWSER_IDLE_TIMEOUT_MS=300000`, `Restart=always`, `RestartSec=5`. *(The unit also carries an API key value — deliberately not reproduced here.)* - Profile: `anthony` (502913bf) — NOT `persist-anthony`. A clean public profile `jobradar` is used for job-board scraping. ## Resource containment (live-verified 2026-10-13) The unit carries a drop-in **`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`**: ``` [Service] MemoryHigh=2G MemoryMax=3G MemorySwapMax=1G OOMPolicy=kill ``` **Observed ceiling event — 2026-10-12 07:01–07:23 (Daily Marketing Job Radar).** A long browser run drove the service to the top of that containment window: - Mid-run: `Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M, swap peak: 1G)`, **383 tasks**, uptime 9 days. Swap fully consumed, no reclaimable headroom. - Symptom: the REST `/tabs` endpoint and page evals began hanging (they look like ordinary tool timeouts). - Recovery: restart `camofox-browser.service`, then work **one tab at a time, closing each before opening the next**. Post-restart the host sat at ~2.1 GB of 14 GB. **Diagnosis:** the limits held and `OOMPolicy=kill` never had to fire — the *workload* exceeded the cgroup, not a leak. A green cron row therefore does **not** mean the browser layer was healthy; check `systemctl --user status camofox-browser.service` (peak/swap/tasks) whenever a browser-heavy job runs long. Restart (`systemctl --user restart camofox-browser.service`) is measurable roughly 16 h later as a healthy baseline: `Memory: 16.5M` cgroup, `peak: 1.4G`, `swap peak: 234.5M`, 16 tasks, `browserConnected: false` until the next session starts. ## Steel Browser (CT201) — Removed (2026-10-29) - **Status:** **Fully removed.** Nothing Steel-related survives on CT201 or CT333 — containers, images, network, project dir and Serve routes are all gone. Camofox is the only browser backend. - **Plugin:** `browser-steel` — **deleted** (`~/.hermes/plugins/browser-steel/` no longer exists; no `browser-steel` key in `plugins.disabled` or `plugins.entries`) - **CDP / REST / UI:** **retired** — the former CDP `:9223`, REST `:3000` and `/steel` UI addresses no longer serve anything - **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, and the images remain re-pullable from ghcr.io - **See:** [[comparisons/steel-vs-camoufox]] for the full experiment story ## History - **2026-07-22:** Camoufox primary, cua-driver bridge - **2026-07-29:** Steel Browser deployed on CT201, became primary - **2026-07-30:** Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary. - **2026-08-02:** Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API) - **2026-10-12 — containment ceiling hit under a browser-heavy cron run.** Daily Marketing Job Radar drove `camofox-browser.service` to `2.5G` with swap fully consumed (`available: 0B`, 383 tasks, 9-day uptime); `/tabs` and evals hung until the service was restarted at 07:23:50 and the job finished one tab at a time. Workload problem, not a leak — limits (`MemoryHigh=2G`/`MemoryMax=3G`/`MemorySwapMax=1G`) held. - **2026-10-13 — page corrected against a live probe.** REST port `:9093` → **`:9377`**; `:9378` (claimed "Camoufox UI") does not exist; `CT450` host attribution replaced with the local `systemd --user` unit on the Hermes host (CT460). Containment drop-in documented. Same day, the 07:01 Job Radar session was audited end-to-end and its memory-pressure evidence added above. - **2026-10-29 — Steel fully removed.** Config back to `cloud_provider: camofox` / `camera_provider: camofox`, the `browser-steel` plugin deleted, `STEEL_BASE_URL` dropped from `~/.hermes/.env`, and the Steel deployments on **CT201** (containers, images, network, project dir, Serve routes) and **CT333** (container + image) torn down. Camofox is the sole backend. - **2026-11-02 — the HTTP server wedged at the start of a browser-heavy cron run; a restart cleared it.** The 07:00 Job Radar Stage 1 (`dca8482e4f76`) found the REST server accepting nothing — requests timing out with **no lines reaching the service log** — before its first tab was opened. `systemctl --user restart camofox-browser.service` cleared it and the run went on to work four boards. **Live re-probe `2026-11-03 00:16:14 AWST`:** unit `active`, `MainPID 953299`, `ActiveEnterTimestamp Mon 2026-11-02 18:11:04`, `MemoryCurrent 37.5 MB`; `/health` → **200** `{"ok":true,"engine":"camoufox","browserConnected":false,"browserRunning":false,"activeTabs":0,"activeSessions":0,"consecutiveFailures":0}`; `/tabs` → `{"running":true,"tabs":[]}`. Idle-clean and far inside the containment drop-in. ⚠ **This is a different fault from the 2026-10-12 ceiling:** that one was memory exhaustion under many open tabs (`2.5G`, swap fully consumed); this one was a **hung HTTP server at 31 MB RSS**, so the containment limits are not the control that applies. A health probe inside browser-heavy scripts should therefore test **responsiveness**, not just `/health` status code. ## Correction (2026-10-29) — Steel is out of the stack, not just disabled Verified against live state at **2026-10-29**. The earlier 00:3x probe said only "not reachable"; the truth is stronger — Steel is **fully removed**, not merely down: - **Hermes is back on Camofox, and Camofox is the sole backend.** The `browser:` block in `~/.hermes/config.yaml` reads exactly `cloud_provider: camofox`, `camera_provider: camofox`, `engine: auto`, `cdp_url: ""`. - **The `browser-steel` plugin is deleted** — `~/.hermes/plugins/browser-steel/` no longer exists, and `plugins.disabled` / `plugins.entries` carry no `browser-steel` key, so the `plugin_steel` toolset cannot be selected. Hermes core source contains **zero** Steel references, so there is no Steel fallback path in the tree. Backups: `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z`, `config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z`. - **CT201 is clean.** The containers `steel-browser-api-1` and `steel-browser-ui-1`, the `steel-network` Docker network, the images `steel-browser-api:fixed` / `ghcr.io/steel-dev/steel-browser-api:latest` / `ghcr.io/steel-dev/steel-browser-ui:latest`, and the project dir `/root/steel-browser` were all removed (~6 GB reclaimed; disk 70G used → 64G used). The Tailscale Serve routes `/steel` and `/steel-api` were deleted with `tailscale serve --set-path off`; CT201's remaining routes are only `/` → `127.0.0.1:8085` and `/mgr` → `127.0.0.1:8084`, on **both** `:443` and `:8443`. No listeners remain on 3000, 5173 or 9223. - **CT333's second, separate Steel copy is gone too.** The orphaned `steel-browser` container (image `ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image were removed; no Steel containers, images or listeners remain on CT333. - **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, plus `/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io. - **Camofox is healthy and working** — REST `http://100.118.5.51:9377` (and `http://localhost:9377`); `/health` returns **200**, `/tabs` returns `{"running":true,"tabs":[]}`. - **Stale environment variable — now cleared from the gateway.** `STEEL_BASE_URL` is no longer in `~/.hermes/.env`. It survived in the environment of already-running processes (inherited at spawn), so the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it: the new gateway tree (PID `3913841`) reads `STEEL_BASE_URL=0`. A few older, unrelated services still carry the variable in their inherited environment (the dashboard unit and `agentmail-webhook.service`) — **no core Hermes code reads `STEEL_BASE_URL`**, so it is inert either way, and it is a runtime-only leftover, **not** a config value. > ⚠ **Superseded (the 2026-10-29 00:3x probe):** an earlier version of this correction described Steel as > merely "not reachable — endpoint dark, still running on CT201". That is outdated: Steel is now **removed**, > not down. ## Related - [[comparisons/steel-vs-camoufox]] - [[concepts/browser-automation]] - [[decisions/rollback-steel-to-camofox]] - [[runbooks/restart-browser]]