Files
agent-estate-wiki/systems/browser-backend.md
T

170 lines
12 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Browser Backend
type: system
status: active
created: 2026-07-22
updated: 2026-11-03
verified_on: 2026-11-03
confidence: high
tags: [browser, camofox, automation]
sources: [live-probe, config, session-history]
---
# Browser Backend
## Purpose
Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.
## Current Configuration (re-verified 2026-10-13)
- **Primary backend:** Camofox (Firefox-based, anti-fingerprint) — `systemd --user` unit `camofox-browser.service` **on the Hermes host itself** (the unit path is `/home/hermes/.config/systemd/user/…`, and the VNC route is named after CT460). The older `CT450` attribution below was not reproducible on 2026-10-13 — see the correction note.
- **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox`
- **REST API:** **`:9377`** — live-verified 2026-10-13 (`node server.js`, `CAMOFOX_PORT=9377`; `LISTEN *:9377`). **The `:9093` figure in earlier revisions of this page is stale** — nothing was listening on 9093 on 2026-10-13.
- **VNC viewer:** NovNC `websockify` on `127.0.0.1:6080` → `127.0.0.1:5900`, published as `https://ct460.kangaroo-eel.ts.net/vnc-camofox`
- **Steel Browser (CT201):** **Fully removed on 2026-10-29** — plugin deleted, CT201 deployment torn down, Serve routes deleted. **Camofox is the sole backend.** See the 2026-10-29 correction below.
## Camofox (local `systemd --user` service)
> ⚠ Earlier revisions titled this section "Camoufox (CT450)". Live-verified 2026-10-13: the service Hermes
> talks to is a `systemd --user` unit on the Hermes host (`/home/hermes/.config/systemd/user/camofox-browser.service`),
> reached at `localhost:9377`, with its VNC route published as `…/vnc-camofox`. Treat any "runs on CT450 /
> CT201" claim as unverified until re-probed.
- **Host:** the Hermes host itself (CT460) — `systemd --user` unit resolved on `hostname = hermes`, no network hop
- **Deployment:** Systemd service `camofox-browser.service`
- **Engine:** Camoufox (custom Firefox build with anti-detection fingerprinting)
- **Display:** Headed via Xvfb `:99` (1600×1200×24) + fluxbox window manager
- **Auth:** Persistent profile `anthony` — authenticated into Uber Eats, LinkedIn, etc.
- **Connection:** REST API **and** WebSocket bridge both at `localhost:9377` (see the port correction above)
- **Last verified:** Jul 30 — `browserConnected: true`, `sessions: 1`, `tabs: 1`, active 200s on scroll/navigate
### Access
| Endpoint | URL | Used by |
|---|---|---|
| REST API | `http://localhost:9377` | Hermes browser tools + direct REST (`/tabs`, `/start`, `/sessions`) |
| VNC (NovNC) | `https://ct460.kangaroo-eel.ts.net/vnc-camofox` | Visual session viewer (`websockify` `127.0.0.1:6080` → `127.0.0.1:5900`) |
> ⚠ **Port correction (2026-10-13):** earlier revisions listed the REST API on `:9093` and a separate CDP
> WebSocket on `:9377`. Live probe: the only Camofox listener is `node server.js` on `*:9377` (with
> `CAMOFOX_URL=http://localhost:9377` in the unit). `:9093` and `:9378` were both closed. `:9377` is the
> REST API **and** the WebSocket bridge — there is no separate 9093 service.
### Config
- `cloud_provider: camofox` in `~/.hermes/config.yaml`
- `CAMOFOX_URL=http://localhost:9377` and `CAMOFOX_API_KEY` in `~/.hermes/.env`
- Unit environment (`~/.config/systemd/user/camofox-browser.service`): `CAMOFOX_PORT=9377`,
`CAMOFOX_PROFILE_DIR=/home/hermes/.camofox/profiles`, `ENABLE_VNC=1`, `HANDLER_TIMEOUT_MS=90000`,
`NAVIGATE_TIMEOUT_MS=60000`, `SESSION_TIMEOUT_MS=600000`, `TAB_INACTIVITY_MS=300000`,
`BROWSER_IDLE_TIMEOUT_MS=300000`, `Restart=always`, `RestartSec=5`. *(The unit also carries an API key
value — deliberately not reproduced here.)*
- Profile: `anthony` (502913bf) — NOT `persist-anthony`. A clean public profile `jobradar` is used for job-board scraping.
## Resource containment (live-verified 2026-10-13)
The unit carries a drop-in **`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`**:
```
[Service]
MemoryHigh=2G
MemoryMax=3G
MemorySwapMax=1G
OOMPolicy=kill
```
**Observed ceiling event — 2026-10-12 07:01–07:23 (Daily Marketing Job Radar).** A long browser run drove
the service to the top of that containment window:
- Mid-run: `Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M,
swap peak: 1G)`, **383 tasks**, uptime 9 days. Swap fully consumed, no reclaimable headroom.
- Symptom: the REST `/tabs` endpoint and page evals began hanging (they look like ordinary tool timeouts).
- Recovery: restart `camofox-browser.service`, then work **one tab at a time, closing each before opening
the next**. Post-restart the host sat at ~2.1 GB of 14 GB.
**Diagnosis:** the limits held and `OOMPolicy=kill` never had to fire — the *workload* exceeded the cgroup,
not a leak. A green cron row therefore does **not** mean the browser layer was healthy; check
`systemctl --user status camofox-browser.service` (peak/swap/tasks) whenever a browser-heavy job runs long.
Restart (`systemctl --user restart camofox-browser.service`) is measurable roughly 16 h later as a healthy
baseline: `Memory: 16.5M` cgroup, `peak: 1.4G`, `swap peak: 234.5M`, 16 tasks, `browserConnected: false`
until the next session starts.
## Steel Browser (CT201) — Removed (2026-10-29)
- **Status:** **Fully removed.** Nothing Steel-related survives on CT201 or CT333 — containers, images, network, project dir and Serve routes are all gone. Camofox is the only browser backend.
- **Plugin:** `browser-steel` — **deleted** (`~/.hermes/plugins/browser-steel/` no longer exists; no `browser-steel` key in `plugins.disabled` or `plugins.entries`)
- **CDP / REST / UI:** **retired** — the former CDP `:9223`, REST `:3000` and `/steel` UI addresses no longer serve anything
- **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, and the images remain re-pullable from ghcr.io
- **See:** [[comparisons/steel-vs-camoufox]] for the full experiment story
## History
- **2026-07-22:** Camoufox primary, cua-driver bridge
- **2026-07-29:** Steel Browser deployed on CT201, became primary
- **2026-07-30:** Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary.
- **2026-08-02:** Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API)
- **2026-10-12 — containment ceiling hit under a browser-heavy cron run.** Daily Marketing Job Radar drove
`camofox-browser.service` to `2.5G` with swap fully consumed (`available: 0B`, 383 tasks, 9-day uptime);
`/tabs` and evals hung until the service was restarted at 07:23:50 and the job finished one tab at a time.
Workload problem, not a leak — limits (`MemoryHigh=2G`/`MemoryMax=3G`/`MemorySwapMax=1G`) held.
- **2026-10-13 — page corrected against a live probe.** REST port `:9093` → **`:9377`**; `:9378` (claimed
"Camoufox UI") does not exist; `CT450` host attribution replaced with the local `systemd --user` unit on
the Hermes host (CT460). Containment drop-in documented. Same day, the 07:01 Job Radar session was audited
end-to-end and its memory-pressure evidence added above.
- **2026-10-29 — Steel fully removed.** Config back to `cloud_provider: camofox` / `camera_provider: camofox`,
the `browser-steel` plugin deleted, `STEEL_BASE_URL` dropped from `~/.hermes/.env`, and the Steel deployments
on **CT201** (containers, images, network, project dir, Serve routes) and **CT333** (container + image)
torn down. Camofox is the sole backend.
- **2026-11-02 — the HTTP server wedged at the start of a browser-heavy cron run; a restart cleared it.**
The 07:00 Job Radar Stage 1 (`dca8482e4f76`) found the REST server accepting nothing — requests timing out
with **no lines reaching the service log** — before its first tab was opened. `systemctl --user restart
camofox-browser.service` cleared it and the run went on to work four boards. **Live re-probe
`2026-11-03 00:16:14 AWST`:** unit `active`, `MainPID 953299`, `ActiveEnterTimestamp Mon 2026-11-02
18:11:04`, `MemoryCurrent 37.5 MB`; `/health` → **200**
`{"ok":true,"engine":"camoufox","browserConnected":false,"browserRunning":false,"activeTabs":0,"activeSessions":0,"consecutiveFailures":0}`;
`/tabs` → `{"running":true,"tabs":[]}`. Idle-clean and far inside the containment drop-in.
⚠ **This is a different fault from the 2026-10-12 ceiling:** that one was memory exhaustion under many
open tabs (`2.5G`, swap fully consumed); this one was a **hung HTTP server at 31 MB RSS**, so the
containment limits are not the control that applies. A health probe inside browser-heavy scripts should
therefore test **responsiveness**, not just `/health` status code.
## Correction (2026-10-29) — Steel is out of the stack, not just disabled
Verified against live state at **2026-10-29**. The earlier 00:3x probe said only "not reachable"; the truth is
stronger — Steel is **fully removed**, not merely down:
- **Hermes is back on Camofox, and Camofox is the sole backend.** The `browser:` block in
`~/.hermes/config.yaml` reads exactly `cloud_provider: camofox`, `camera_provider: camofox`, `engine: auto`,
`cdp_url: ""`.
- **The `browser-steel` plugin is deleted** — `~/.hermes/plugins/browser-steel/` no longer exists, and
`plugins.disabled` / `plugins.entries` carry no `browser-steel` key, so the `plugin_steel` toolset cannot be
selected. Hermes core source contains **zero** Steel references, so there is no Steel fallback path in the
tree. Backups: `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z`,
`config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z`.
- **CT201 is clean.** The containers `steel-browser-api-1` and `steel-browser-ui-1`, the `steel-network` Docker
network, the images `steel-browser-api:fixed` / `ghcr.io/steel-dev/steel-browser-api:latest` /
`ghcr.io/steel-dev/steel-browser-ui:latest`, and the project dir `/root/steel-browser` were all removed
(~6 GB reclaimed; disk 70G used → 64G used). The Tailscale Serve routes `/steel` and `/steel-api` were
deleted with `tailscale serve --set-path <path> off`; CT201's remaining routes are only `/` → `127.0.0.1:8085`
and `/mgr` → `127.0.0.1:8084`, on **both** `:443` and `:8443`. No listeners remain on 3000, 5173 or 9223.
- **CT333's second, separate Steel copy is gone too.** The orphaned `steel-browser` container (image
`ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image
were removed; no Steel containers, images or listeners remain on CT333.
- **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, plus
`/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io.
- **Camofox is healthy and working** — REST `http://100.118.5.51:9377` (and `http://localhost:9377`);
`/health` returns **200**, `/tabs` returns `{"running":true,"tabs":[]}`.
- **Stale environment variable — now cleared from the gateway.** `STEEL_BASE_URL` is no longer in
`~/.hermes/.env`. It survived in the environment of already-running processes (inherited at spawn), so the
**gateway was restarted on 2026-10-29 01:21 AWST** to clear it: the new gateway tree (PID `3913841`) reads
`STEEL_BASE_URL=0`. A few older, unrelated services still carry the variable in their inherited
environment (the dashboard unit and `agentmail-webhook.service`) — **no core Hermes code reads
`STEEL_BASE_URL`**, so it is inert either way, and it is a runtime-only leftover, **not** a config value.
> ⚠ **Superseded (the 2026-10-29 00:3x probe):** an earlier version of this correction described Steel as
> merely "not reachable — endpoint dark, still running on CT201". That is outdated: Steel is now **removed**,
> not down.
## Related
- [[comparisons/steel-vs-camoufox]]
- [[concepts/browser-automation]]
- [[decisions/rollback-steel-to-camofox]]
- [[runbooks/restart-browser]]