140 lines
8.7 KiB
Markdown
140 lines
8.7 KiB
Markdown
---
|
||
title: Browser Backend
|
||
type: system
|
||
status: active
|
||
created: 2026-07-22
|
||
updated: 2026-10-29
|
||
verified_on: 2026-10-29
|
||
confidence: high
|
||
tags: [browser, camofox, automation]
|
||
sources: [live-probe, config, session-history]
|
||
---
|
||
|
||
# Browser Backend
|
||
|
||
## Purpose
|
||
Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.
|
||
|
||
## Current Configuration (re-verified 2026-10-13)
|
||
- **Primary backend:** Camofox (Firefox-based, anti-fingerprint) — `systemd --user` unit `camofox-browser.service` **on the Hermes host itself** (the unit path is `/home/hermes/.config/systemd/user/…`, and the VNC route is named after CT460). The older `CT450` attribution below was not reproducible on 2026-10-13 — see the correction note.
|
||
- **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox`
|
||
- **REST API:** **`:9377`** — live-verified 2026-10-13 (`node server.js`, `CAMOFOX_PORT=9377`; `LISTEN *:9377`). **The `:9093` figure in earlier revisions of this page is stale** — nothing was listening on 9093 on 2026-10-13.
|
||
- **VNC viewer:** NovNC `websockify` on `127.0.0.1:6080` → `127.0.0.1:5900`, published as `https://ct460.kangaroo-eel.ts.net/vnc-camofox`
|
||
- **Steel Browser (CT201):** **Not reachable as of 2026-10-29 00:3x** — host answers ping (2.5 ms) but CDP `:9223` and REST `:3000` refuse; plugin removed. See the 2026-10-29 correction below.
|
||
|
||
## Camofox (local `systemd --user` service)
|
||
|
||
> ⚠ Earlier revisions titled this section "Camoufox (CT450)". Live-verified 2026-10-13: the service Hermes
|
||
> talks to is a `systemd --user` unit on the Hermes host (`/home/hermes/.config/systemd/user/camofox-browser.service`),
|
||
> reached at `localhost:9377`, with its VNC route published as `…/vnc-camofox`. Treat any "runs on CT450 /
|
||
> CT201" claim as unverified until re-probed.
|
||
|
||
- **Host:** the Hermes host itself (CT460) — `systemd --user` unit resolved on `hostname = hermes`, no network hop
|
||
- **Deployment:** Systemd service `camofox-browser.service`
|
||
- **Engine:** Camoufox (custom Firefox build with anti-detection fingerprinting)
|
||
- **Display:** Headed via Xvfb `:99` (1600×1200×24) + fluxbox window manager
|
||
- **Auth:** Persistent profile `anthony` — authenticated into Uber Eats, LinkedIn, etc.
|
||
- **Connection:** REST API **and** WebSocket bridge both at `localhost:9377` (see the port correction above)
|
||
- **Last verified:** Jul 30 — `browserConnected: true`, `sessions: 1`, `tabs: 1`, active 200s on scroll/navigate
|
||
|
||
### Access
|
||
| Endpoint | URL | Used by |
|
||
|---|---|---|
|
||
| REST API | `http://localhost:9377` | Hermes browser tools + direct REST (`/tabs`, `/start`, `/sessions`) |
|
||
| VNC (NovNC) | `https://ct460.kangaroo-eel.ts.net/vnc-camofox` | Visual session viewer (`websockify` `127.0.0.1:6080` → `127.0.0.1:5900`) |
|
||
|
||
> ⚠ **Port correction (2026-10-13):** earlier revisions listed the REST API on `:9093` and a separate CDP
|
||
> WebSocket on `:9377`. Live probe: the only Camofox listener is `node server.js` on `*:9377` (with
|
||
> `CAMOFOX_URL=http://localhost:9377` in the unit). `:9093` and `:9378` were both closed. `:9377` is the
|
||
> REST API **and** the WebSocket bridge — there is no separate 9093 service.
|
||
|
||
### Config
|
||
- `cloud_provider: camofox` in `~/.hermes/config.yaml`
|
||
- `CAMOFOX_URL=http://localhost:9377` and `CAMOFOX_API_KEY` in `~/.hermes/.env`
|
||
- Unit environment (`~/.config/systemd/user/camofox-browser.service`): `CAMOFOX_PORT=9377`,
|
||
`CAMOFOX_PROFILE_DIR=/home/hermes/.camofox/profiles`, `ENABLE_VNC=1`, `HANDLER_TIMEOUT_MS=90000`,
|
||
`NAVIGATE_TIMEOUT_MS=60000`, `SESSION_TIMEOUT_MS=600000`, `TAB_INACTIVITY_MS=300000`,
|
||
`BROWSER_IDLE_TIMEOUT_MS=300000`, `Restart=always`, `RestartSec=5`. *(The unit also carries an API key
|
||
value — deliberately not reproduced here.)*
|
||
- Profile: `anthony` (502913bf) — NOT `persist-anthony`. A clean public profile `jobradar` is used for job-board scraping.
|
||
|
||
## Resource containment (live-verified 2026-10-13)
|
||
|
||
The unit carries a drop-in **`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`**:
|
||
|
||
```
|
||
[Service]
|
||
MemoryHigh=2G
|
||
MemoryMax=3G
|
||
MemorySwapMax=1G
|
||
OOMPolicy=kill
|
||
```
|
||
|
||
**Observed ceiling event — 2026-10-12 07:01–07:23 (Daily Marketing Job Radar).** A long browser run drove
|
||
the service to the top of that containment window:
|
||
|
||
- Mid-run: `Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M,
|
||
swap peak: 1G)`, **383 tasks**, uptime 9 days. Swap fully consumed, no reclaimable headroom.
|
||
- Symptom: the REST `/tabs` endpoint and page evals began hanging (they look like ordinary tool timeouts).
|
||
- Recovery: restart `camofox-browser.service`, then work **one tab at a time, closing each before opening
|
||
the next**. Post-restart the host sat at ~2.1 GB of 14 GB.
|
||
|
||
**Diagnosis:** the limits held and `OOMPolicy=kill` never had to fire — the *workload* exceeded the cgroup,
|
||
not a leak. A green cron row therefore does **not** mean the browser layer was healthy; check
|
||
`systemctl --user status camofox-browser.service` (peak/swap/tasks) whenever a browser-heavy job runs long.
|
||
|
||
Restart (`systemctl --user restart camofox-browser.service`) is measurable roughly 16 h later as a healthy
|
||
baseline: `Memory: 16.5M` cgroup, `peak: 1.4G`, `swap peak: 234.5M`, 16 tasks, `browserConnected: false`
|
||
until the next session starts.
|
||
|
||
## Steel Browser (CT201) — Disabled
|
||
- **Status:** Still running on CT201, Hermes not connected since 2026-07-30
|
||
- **Plugin:** `browser-steel` at `/home/hermes/.hermes/plugins/browser-steel/` — disabled
|
||
- **CDP:** `http://100.96.244.39:9223`
|
||
- **REST API:** `http://100.96.244.39:3000`
|
||
- **UI:** `https://runtipi.kangaroo-eel.ts.net/steel`
|
||
- **See:** [[comparisons/steel-vs-camoufox]] for the full experiment story
|
||
|
||
## History
|
||
- **2026-07-22:** Camoufox primary, cua-driver bridge
|
||
- **2026-07-29:** Steel Browser deployed on CT201, became primary
|
||
- **2026-07-30:** Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary.
|
||
- **2026-08-02:** Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API)
|
||
- **2026-10-12 — containment ceiling hit under a browser-heavy cron run.** Daily Marketing Job Radar drove
|
||
`camofox-browser.service` to `2.5G` with swap fully consumed (`available: 0B`, 383 tasks, 9-day uptime);
|
||
`/tabs` and evals hung until the service was restarted at 07:23:50 and the job finished one tab at a time.
|
||
Workload problem, not a leak — limits (`MemoryHigh=2G`/`MemoryMax=3G`/`MemorySwapMax=1G`) held.
|
||
- **2026-10-13 — page corrected against a live probe.** REST port `:9093` → **`:9377`**; `:9378` (claimed
|
||
"Camoufox UI") does not exist; `CT450` host attribution replaced with the local `systemd --user` unit on
|
||
the Hermes host (CT460). Containment drop-in documented. Same day, the 07:01 Job Radar session was audited
|
||
end-to-end and its memory-pressure evidence added above.
|
||
|
||
## Correction (2026-10-29) — Steel is out of the stack, not just disabled
|
||
|
||
Verified against live state at **2026-10-29 00:3x AWST**:
|
||
|
||
- **Hermes is back on Camofox.** `config.yaml` (mtime **2026-10-29 00:15:13**) reads
|
||
`browser.cloud_provider: camofox`, `browser.camera_provider: camofox`, `engine: auto`, `cdp_url: ""`.
|
||
- **The `browser-steel` plugin is gone** — `~/.hermes/plugins/browser-steel/` no longer exists and
|
||
`plugins.enabled` carries no steel entry, so the `plugin_steel` toolset cannot be selected. Backup taken
|
||
immediately before the change: `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` (written
|
||
**2026-10-29 00:13:24**). A second backup, `config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z`, records
|
||
the 2026-10-28 22:00 attempt.
|
||
- **The Steel endpoint is not serving.** `ping 100.96.244.39` (runtipi/CT201) answers **2.5 ms**, but
|
||
`http://100.96.244.39:9223/json/version` (CDP) and `http://100.96.244.39:3000/v1/health` (REST) both return
|
||
**000** — as does the LAN address `192.168.178.77:9223`. The tunnel route
|
||
`https://runtipi.kangaroo-eel.ts.net/steel` returns **502**, and `steel.martinwa.org` no longer resolves.
|
||
Earlier pages describing Steel as "still running on CT201" are superseded by this probe.
|
||
- ⚠ **Stale environment variable still exported:** `STEEL_BASE_URL=http://100.96.244.39:3000` is present in
|
||
**10 live Hermes processes** — the gateway tree (`3371780`, `3372125`, `3372912`, `3844013`, `3880214`,
|
||
`3880663`), `agent-browser` 0.26.0 (`3843841`), two chromium crashpad handlers (`3843878`, `3843880`) and
|
||
`agentmail-webhook.py` (`402`). It now points at a dead endpoint and clears only when the gateway is
|
||
restarted. The config is explicit (`cloud_provider: camofox`), so it does not currently misroute browser
|
||
calls — but it is a trap for anything that reads the variable.
|
||
|
||
## Related
|
||
- [[comparisons/steel-vs-camoufox]]
|
||
- [[concepts/browser-automation]]
|
||
- [[decisions/rollback-steel-to-camofox]]
|
||
- [[runbooks/restart-browser]]
|