177 lines
12 KiB
Markdown
177 lines
12 KiB
Markdown
---
|
||
title: Browser Backend
|
||
type: system
|
||
status: active
|
||
created: 2026-07-22
|
||
updated: 2026-11-08
|
||
verified_on: 2026-11-08
|
||
confidence: high
|
||
tags: [browser, camofox, automation]
|
||
sources: [live-probe, config, session-history]
|
||
---
|
||
|
||
# Browser Backend
|
||
|
||
## Purpose
|
||
Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.
|
||
|
||
## Current Configuration (re-verified 2026-10-13)
|
||
|
||
> **2026-11-07/08 — the browser backend now ships as a plugin, not a core feature.** The Hermes
|
||
> `0.22.5 → 0.23.0` update installed **`camofox` v1.1.0** as an **enabled plugin** at
|
||
> `~/.hermes/plugins/camofox/` (`plugin.yaml`, `camofox_tools.py`, `camofox_policy.py`,
|
||
> `camofox_vault.py`, `camofox_cli.py`), confirmed live by `hermes plugins list` on 2026-11-07.
|
||
> No `browser.*` key was changed in this window — the sections below still describe the running
|
||
> configuration ([[systems/hermes-agent]]).
|
||
- **Primary backend:** Camofox (Firefox-based, anti-fingerprint) — `systemd --user` unit `camofox-browser.service` **on the Hermes host itself** (the unit path is `/home/hermes/.config/systemd/user/…`, and the VNC route is named after CT460). The older `CT450` attribution below was not reproducible on 2026-10-13 — see the correction note.
|
||
- **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox`
|
||
- **REST API:** **`:9377`** — live-verified 2026-10-13 (`node server.js`, `CAMOFOX_PORT=9377`; `LISTEN *:9377`). **The `:9093` figure in earlier revisions of this page is stale** — nothing was listening on 9093 on 2026-10-13.
|
||
- **VNC viewer:** NovNC `websockify` on `127.0.0.1:6080` → `127.0.0.1:5900`, published as `https://ct460.kangaroo-eel.ts.net/vnc-camofox`
|
||
- **Steel Browser (CT201):** **Fully removed on 2026-10-29** — plugin deleted, CT201 deployment torn down, Serve routes deleted. **Camofox is the sole backend.** See the 2026-10-29 correction below.
|
||
|
||
## Camofox (local `systemd --user` service)
|
||
|
||
> ⚠ Earlier revisions titled this section "Camoufox (CT450)". Live-verified 2026-10-13: the service Hermes
|
||
> talks to is a `systemd --user` unit on the Hermes host (`/home/hermes/.config/systemd/user/camofox-browser.service`),
|
||
> reached at `localhost:9377`, with its VNC route published as `…/vnc-camofox`. Treat any "runs on CT450 /
|
||
> CT201" claim as unverified until re-probed.
|
||
|
||
- **Host:** the Hermes host itself (CT460) — `systemd --user` unit resolved on `hostname = hermes`, no network hop
|
||
- **Deployment:** Systemd service `camofox-browser.service`
|
||
- **Engine:** Camoufox (custom Firefox build with anti-detection fingerprinting)
|
||
- **Display:** Headed via Xvfb `:99` (1600×1200×24) + fluxbox window manager
|
||
- **Auth:** Persistent profile `anthony` — authenticated into Uber Eats, LinkedIn, etc.
|
||
- **Connection:** REST API **and** WebSocket bridge both at `localhost:9377` (see the port correction above)
|
||
- **Last verified:** Jul 30 — `browserConnected: true`, `sessions: 1`, `tabs: 1`, active 200s on scroll/navigate
|
||
|
||
### Access
|
||
| Endpoint | URL | Used by |
|
||
|---|---|---|
|
||
| REST API | `http://localhost:9377` | Hermes browser tools + direct REST (`/tabs`, `/start`, `/sessions`) |
|
||
| VNC (NovNC) | `https://ct460.kangaroo-eel.ts.net/vnc-camofox` | Visual session viewer (`websockify` `127.0.0.1:6080` → `127.0.0.1:5900`) |
|
||
|
||
> ⚠ **Port correction (2026-10-13):** earlier revisions listed the REST API on `:9093` and a separate CDP
|
||
> WebSocket on `:9377`. Live probe: the only Camofox listener is `node server.js` on `*:9377` (with
|
||
> `CAMOFOX_URL=http://localhost:9377` in the unit). `:9093` and `:9378` were both closed. `:9377` is the
|
||
> REST API **and** the WebSocket bridge — there is no separate 9093 service.
|
||
|
||
### Config
|
||
- `cloud_provider: camofox` in `~/.hermes/config.yaml`
|
||
- `CAMOFOX_URL=http://localhost:9377` and `CAMOFOX_API_KEY` in `~/.hermes/.env`
|
||
- Unit environment (`~/.config/systemd/user/camofox-browser.service`): `CAMOFOX_PORT=9377`,
|
||
`CAMOFOX_PROFILE_DIR=/home/hermes/.camofox/profiles`, `ENABLE_VNC=1`, `HANDLER_TIMEOUT_MS=90000`,
|
||
`NAVIGATE_TIMEOUT_MS=60000`, `SESSION_TIMEOUT_MS=600000`, `TAB_INACTIVITY_MS=300000`,
|
||
`BROWSER_IDLE_TIMEOUT_MS=300000`, `Restart=always`, `RestartSec=5`. *(The unit also carries an API key
|
||
value — deliberately not reproduced here.)*
|
||
- Profile: `anthony` (502913bf) — NOT `persist-anthony`. A clean public profile `jobradar` is used for job-board scraping.
|
||
|
||
## Resource containment (live-verified 2026-10-13)
|
||
|
||
The unit carries a drop-in **`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`**:
|
||
|
||
```
|
||
[Service]
|
||
MemoryHigh=2G
|
||
MemoryMax=3G
|
||
MemorySwapMax=1G
|
||
OOMPolicy=kill
|
||
```
|
||
|
||
**Observed ceiling event — 2026-10-12 07:01–07:23 (Daily Marketing Job Radar).** A long browser run drove
|
||
the service to the top of that containment window:
|
||
|
||
- Mid-run: `Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M,
|
||
swap peak: 1G)`, **383 tasks**, uptime 9 days. Swap fully consumed, no reclaimable headroom.
|
||
- Symptom: the REST `/tabs` endpoint and page evals began hanging (they look like ordinary tool timeouts).
|
||
- Recovery: restart `camofox-browser.service`, then work **one tab at a time, closing each before opening
|
||
the next**. Post-restart the host sat at ~2.1 GB of 14 GB.
|
||
|
||
**Diagnosis:** the limits held and `OOMPolicy=kill` never had to fire — the *workload* exceeded the cgroup,
|
||
not a leak. A green cron row therefore does **not** mean the browser layer was healthy; check
|
||
`systemctl --user status camofox-browser.service` (peak/swap/tasks) whenever a browser-heavy job runs long.
|
||
|
||
Restart (`systemctl --user restart camofox-browser.service`) is measurable roughly 16 h later as a healthy
|
||
baseline: `Memory: 16.5M` cgroup, `peak: 1.4G`, `swap peak: 234.5M`, 16 tasks, `browserConnected: false`
|
||
until the next session starts.
|
||
|
||
## Steel Browser (CT201) — Removed (2026-10-29)
|
||
- **Status:** **Fully removed.** Nothing Steel-related survives on CT201 or CT333 — containers, images, network, project dir and Serve routes are all gone. Camofox is the only browser backend.
|
||
- **Plugin:** `browser-steel` — **deleted** (`~/.hermes/plugins/browser-steel/` no longer exists; no `browser-steel` key in `plugins.disabled` or `plugins.entries`)
|
||
- **CDP / REST / UI:** **retired** — the former CDP `:9223`, REST `:3000` and `/steel` UI addresses no longer serve anything
|
||
- **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, and the images remain re-pullable from ghcr.io
|
||
- **See:** [[comparisons/steel-vs-camoufox]] for the full experiment story
|
||
|
||
## History
|
||
- **2026-07-22:** Camoufox primary, cua-driver bridge
|
||
- **2026-07-29:** Steel Browser deployed on CT201, became primary
|
||
- **2026-07-30:** Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary.
|
||
- **2026-08-02:** Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API)
|
||
- **2026-10-12 — containment ceiling hit under a browser-heavy cron run.** Daily Marketing Job Radar drove
|
||
`camofox-browser.service` to `2.5G` with swap fully consumed (`available: 0B`, 383 tasks, 9-day uptime);
|
||
`/tabs` and evals hung until the service was restarted at 07:23:50 and the job finished one tab at a time.
|
||
Workload problem, not a leak — limits (`MemoryHigh=2G`/`MemoryMax=3G`/`MemorySwapMax=1G`) held.
|
||
- **2026-10-13 — page corrected against a live probe.** REST port `:9093` → **`:9377`**; `:9378` (claimed
|
||
"Camoufox UI") does not exist; `CT450` host attribution replaced with the local `systemd --user` unit on
|
||
the Hermes host (CT460). Containment drop-in documented. Same day, the 07:01 Job Radar session was audited
|
||
end-to-end and its memory-pressure evidence added above.
|
||
- **2026-10-29 — Steel fully removed.** Config back to `cloud_provider: camofox` / `camera_provider: camofox`,
|
||
the `browser-steel` plugin deleted, `STEEL_BASE_URL` dropped from `~/.hermes/.env`, and the Steel deployments
|
||
on **CT201** (containers, images, network, project dir, Serve routes) and **CT333** (container + image)
|
||
torn down. Camofox is the sole backend.
|
||
|
||
- **2026-11-02 — the HTTP server wedged at the start of a browser-heavy cron run; a restart cleared it.**
|
||
The 07:00 Job Radar Stage 1 (`dca8482e4f76`) found the REST server accepting nothing — requests timing out
|
||
with **no lines reaching the service log** — before its first tab was opened. `systemctl --user restart
|
||
camofox-browser.service` cleared it and the run went on to work four boards. **Live re-probe
|
||
`2026-11-03 00:16:14 AWST`:** unit `active`, `MainPID 953299`, `ActiveEnterTimestamp Mon 2026-11-02
|
||
18:11:04`, `MemoryCurrent 37.5 MB`; `/health` → **200**
|
||
`{"ok":true,"engine":"camoufox","browserConnected":false,"browserRunning":false,"activeTabs":0,"activeSessions":0,"consecutiveFailures":0}`;
|
||
`/tabs` → `{"running":true,"tabs":[]}`. Idle-clean and far inside the containment drop-in.
|
||
⚠ **This is a different fault from the 2026-10-12 ceiling:** that one was memory exhaustion under many
|
||
open tabs (`2.5G`, swap fully consumed); this one was a **hung HTTP server at 31 MB RSS**, so the
|
||
containment limits are not the control that applies. A health probe inside browser-heavy scripts should
|
||
therefore test **responsiveness**, not just `/health` status code.
|
||
|
||
## Correction (2026-10-29) — Steel is out of the stack, not just disabled
|
||
|
||
Verified against live state at **2026-10-29**. The earlier 00:3x probe said only "not reachable"; the truth is
|
||
stronger — Steel is **fully removed**, not merely down:
|
||
|
||
- **Hermes is back on Camofox, and Camofox is the sole backend.** The `browser:` block in
|
||
`~/.hermes/config.yaml` reads exactly `cloud_provider: camofox`, `camera_provider: camofox`, `engine: auto`,
|
||
`cdp_url: ""`.
|
||
- **The `browser-steel` plugin is deleted** — `~/.hermes/plugins/browser-steel/` no longer exists, and
|
||
`plugins.disabled` / `plugins.entries` carry no `browser-steel` key, so the `plugin_steel` toolset cannot be
|
||
selected. Hermes core source contains **zero** Steel references, so there is no Steel fallback path in the
|
||
tree. Backups: `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z`,
|
||
`config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z`.
|
||
- **CT201 is clean.** The containers `steel-browser-api-1` and `steel-browser-ui-1`, the `steel-network` Docker
|
||
network, the images `steel-browser-api:fixed` / `ghcr.io/steel-dev/steel-browser-api:latest` /
|
||
`ghcr.io/steel-dev/steel-browser-ui:latest`, and the project dir `/root/steel-browser` were all removed
|
||
(~6 GB reclaimed; disk 70G used → 64G used). The Tailscale Serve routes `/steel` and `/steel-api` were
|
||
deleted with `tailscale serve --set-path <path> off`; CT201's remaining routes are only `/` → `127.0.0.1:8085`
|
||
and `/mgr` → `127.0.0.1:8084`, on **both** `:443` and `:8443`. No listeners remain on 3000, 5173 or 9223.
|
||
- **CT333's second, separate Steel copy is gone too.** The orphaned `steel-browser` container (image
|
||
`ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image
|
||
were removed; no Steel containers, images or listeners remain on CT333.
|
||
- **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, plus
|
||
`/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io.
|
||
- **Camofox is healthy and working** — REST `http://100.118.5.51:9377` (and `http://localhost:9377`);
|
||
`/health` returns **200**, `/tabs` returns `{"running":true,"tabs":[]}`.
|
||
- **Stale environment variable — now cleared from the gateway.** `STEEL_BASE_URL` is no longer in
|
||
`~/.hermes/.env`. It survived in the environment of already-running processes (inherited at spawn), so the
|
||
**gateway was restarted on 2026-10-29 01:21 AWST** to clear it: the new gateway tree (PID `3913841`) reads
|
||
`STEEL_BASE_URL=0`. A few older, unrelated services still carry the variable in their inherited
|
||
environment (the dashboard unit and `agentmail-webhook.service`) — **no core Hermes code reads
|
||
`STEEL_BASE_URL`**, so it is inert either way, and it is a runtime-only leftover, **not** a config value.
|
||
|
||
> ⚠ **Superseded (the 2026-10-29 00:3x probe):** an earlier version of this correction described Steel as
|
||
> merely "not reachable — endpoint dark, still running on CT201". That is outdated: Steel is now **removed**,
|
||
> not down.
|
||
|
||
## Related
|
||
- [[comparisons/steel-vs-camoufox]]
|
||
- [[concepts/browser-automation]]
|
||
- [[decisions/rollback-steel-to-camofox]]
|
||
- [[runbooks/restart-browser]]
|