Files
agent-estate-wiki/systems/browser-backend.md
T
Tony0410andHermes 677bb0c959 docs(wiki): correct Steel Browser references across 15 pages — Steel fully removed, Camofox is sole backend
Root cause of the long-standing cloud_provider: steel residue: the 2026-07-30
rollback disabled the browser-steel plugin and set browser.engine: camofox but
left browser.cloud_provider: steel in config.yaml. The split config was
inert-but-wrong, so it survived ~3 months of audits unnoticed.

Corrects pages that still described Steel as present, running or configured:
- systems/browser-backend.md (incl. gateway-restart env-var note)
- comparisons/steel-vs-camoufox.md, comparisons/camoufox-vs-chrome.md
- decisions/rollback-steel-to-camofox.md, decisions/index.md
- infrastructure/docker-services.md, infrastructure/tailscale.md, infrastructure/hosts.md
- systems/homepage-dashboard.md, systems/scheduled-tasks.md
- concepts/browser-automation.md, runbooks/recover-docker-service.md
- current-state.md, index.md, agents/nanobot.md
- log.md (dated entry; supersedes the 00:3x audit's 'endpoint dark' wording)

Also restores the Ignis compose row in infrastructure/docker-services.md, which
the 2026-10-11 revision dropped (verified: /home/hermes/docker/ignis/docker-compose.yml
exists and the ignis container is Up on CT460).

Co-Authored-By: Hermes <hermes@local>
2026-10-29 02:02:49 +08:00

157 lines
11 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Browser Backend
type: system
status: active
created: 2026-07-22
updated: 2026-10-29
verified_on: 2026-10-29
confidence: high
tags: [browser, camofox, automation]
sources: [live-probe, config, session-history]
---
# Browser Backend
## Purpose
Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.
## Current Configuration (re-verified 2026-10-13)
- **Primary backend:** Camofox (Firefox-based, anti-fingerprint) — `systemd --user` unit `camofox-browser.service` **on the Hermes host itself** (the unit path is `/home/hermes/.config/systemd/user/…`, and the VNC route is named after CT460). The older `CT450` attribution below was not reproducible on 2026-10-13 — see the correction note.
- **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox`
- **REST API:** **`:9377`** — live-verified 2026-10-13 (`node server.js`, `CAMOFOX_PORT=9377`; `LISTEN *:9377`). **The `:9093` figure in earlier revisions of this page is stale** — nothing was listening on 9093 on 2026-10-13.
- **VNC viewer:** NovNC `websockify` on `127.0.0.1:6080` → `127.0.0.1:5900`, published as `https://ct460.kangaroo-eel.ts.net/vnc-camofox`
- **Steel Browser (CT201):** **Fully removed on 2026-10-29** — plugin deleted, CT201 deployment torn down, Serve routes deleted. **Camofox is the sole backend.** See the 2026-10-29 correction below.
## Camofox (local `systemd --user` service)
> ⚠ Earlier revisions titled this section "Camoufox (CT450)". Live-verified 2026-10-13: the service Hermes
> talks to is a `systemd --user` unit on the Hermes host (`/home/hermes/.config/systemd/user/camofox-browser.service`),
> reached at `localhost:9377`, with its VNC route published as `…/vnc-camofox`. Treat any "runs on CT450 /
> CT201" claim as unverified until re-probed.
- **Host:** the Hermes host itself (CT460) — `systemd --user` unit resolved on `hostname = hermes`, no network hop
- **Deployment:** Systemd service `camofox-browser.service`
- **Engine:** Camoufox (custom Firefox build with anti-detection fingerprinting)
- **Display:** Headed via Xvfb `:99` (1600×1200×24) + fluxbox window manager
- **Auth:** Persistent profile `anthony` — authenticated into Uber Eats, LinkedIn, etc.
- **Connection:** REST API **and** WebSocket bridge both at `localhost:9377` (see the port correction above)
- **Last verified:** Jul 30 — `browserConnected: true`, `sessions: 1`, `tabs: 1`, active 200s on scroll/navigate
### Access
| Endpoint | URL | Used by |
|---|---|---|
| REST API | `http://localhost:9377` | Hermes browser tools + direct REST (`/tabs`, `/start`, `/sessions`) |
| VNC (NovNC) | `https://ct460.kangaroo-eel.ts.net/vnc-camofox` | Visual session viewer (`websockify` `127.0.0.1:6080` → `127.0.0.1:5900`) |
> ⚠ **Port correction (2026-10-13):** earlier revisions listed the REST API on `:9093` and a separate CDP
> WebSocket on `:9377`. Live probe: the only Camofox listener is `node server.js` on `*:9377` (with
> `CAMOFOX_URL=http://localhost:9377` in the unit). `:9093` and `:9378` were both closed. `:9377` is the
> REST API **and** the WebSocket bridge — there is no separate 9093 service.
### Config
- `cloud_provider: camofox` in `~/.hermes/config.yaml`
- `CAMOFOX_URL=http://localhost:9377` and `CAMOFOX_API_KEY` in `~/.hermes/.env`
- Unit environment (`~/.config/systemd/user/camofox-browser.service`): `CAMOFOX_PORT=9377`,
`CAMOFOX_PROFILE_DIR=/home/hermes/.camofox/profiles`, `ENABLE_VNC=1`, `HANDLER_TIMEOUT_MS=90000`,
`NAVIGATE_TIMEOUT_MS=60000`, `SESSION_TIMEOUT_MS=600000`, `TAB_INACTIVITY_MS=300000`,
`BROWSER_IDLE_TIMEOUT_MS=300000`, `Restart=always`, `RestartSec=5`. *(The unit also carries an API key
value — deliberately not reproduced here.)*
- Profile: `anthony` (502913bf) — NOT `persist-anthony`. A clean public profile `jobradar` is used for job-board scraping.
## Resource containment (live-verified 2026-10-13)
The unit carries a drop-in **`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`**:
```
[Service]
MemoryHigh=2G
MemoryMax=3G
MemorySwapMax=1G
OOMPolicy=kill
```
**Observed ceiling event — 2026-10-12 07:01–07:23 (Daily Marketing Job Radar).** A long browser run drove
the service to the top of that containment window:
- Mid-run: `Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M,
swap peak: 1G)`, **383 tasks**, uptime 9 days. Swap fully consumed, no reclaimable headroom.
- Symptom: the REST `/tabs` endpoint and page evals began hanging (they look like ordinary tool timeouts).
- Recovery: restart `camofox-browser.service`, then work **one tab at a time, closing each before opening
the next**. Post-restart the host sat at ~2.1 GB of 14 GB.
**Diagnosis:** the limits held and `OOMPolicy=kill` never had to fire — the *workload* exceeded the cgroup,
not a leak. A green cron row therefore does **not** mean the browser layer was healthy; check
`systemctl --user status camofox-browser.service` (peak/swap/tasks) whenever a browser-heavy job runs long.
Restart (`systemctl --user restart camofox-browser.service`) is measurable roughly 16 h later as a healthy
baseline: `Memory: 16.5M` cgroup, `peak: 1.4G`, `swap peak: 234.5M`, 16 tasks, `browserConnected: false`
until the next session starts.
## Steel Browser (CT201) — Removed (2026-10-29)
- **Status:** **Fully removed.** Nothing Steel-related survives on CT201 or CT333 — containers, images, network, project dir and Serve routes are all gone. Camofox is the only browser backend.
- **Plugin:** `browser-steel` — **deleted** (`~/.hermes/plugins/browser-steel/` no longer exists; no `browser-steel` key in `plugins.disabled` or `plugins.entries`)
- **CDP / REST / UI:** **retired** — the former CDP `:9223`, REST `:3000` and `/steel` UI addresses no longer serve anything
- **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, and the images remain re-pullable from ghcr.io
- **See:** [[comparisons/steel-vs-camoufox]] for the full experiment story
## History
- **2026-07-22:** Camoufox primary, cua-driver bridge
- **2026-07-29:** Steel Browser deployed on CT201, became primary
- **2026-07-30:** Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary.
- **2026-08-02:** Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API)
- **2026-10-12 — containment ceiling hit under a browser-heavy cron run.** Daily Marketing Job Radar drove
`camofox-browser.service` to `2.5G` with swap fully consumed (`available: 0B`, 383 tasks, 9-day uptime);
`/tabs` and evals hung until the service was restarted at 07:23:50 and the job finished one tab at a time.
Workload problem, not a leak — limits (`MemoryHigh=2G`/`MemoryMax=3G`/`MemorySwapMax=1G`) held.
- **2026-10-13 — page corrected against a live probe.** REST port `:9093` → **`:9377`**; `:9378` (claimed
"Camoufox UI") does not exist; `CT450` host attribution replaced with the local `systemd --user` unit on
the Hermes host (CT460). Containment drop-in documented. Same day, the 07:01 Job Radar session was audited
end-to-end and its memory-pressure evidence added above.
- **2026-10-29 — Steel fully removed.** Config back to `cloud_provider: camofox` / `camera_provider: camofox`,
the `browser-steel` plugin deleted, `STEEL_BASE_URL` dropped from `~/.hermes/.env`, and the Steel deployments
on **CT201** (containers, images, network, project dir, Serve routes) and **CT333** (container + image)
torn down. Camofox is the sole backend.
## Correction (2026-10-29) — Steel is out of the stack, not just disabled
Verified against live state at **2026-10-29**. The earlier 00:3x probe said only "not reachable"; the truth is
stronger — Steel is **fully removed**, not merely down:
- **Hermes is back on Camofox, and Camofox is the sole backend.** The `browser:` block in
`~/.hermes/config.yaml` reads exactly `cloud_provider: camofox`, `camera_provider: camofox`, `engine: auto`,
`cdp_url: ""`.
- **The `browser-steel` plugin is deleted** — `~/.hermes/plugins/browser-steel/` no longer exists, and
`plugins.disabled` / `plugins.entries` carry no `browser-steel` key, so the `plugin_steel` toolset cannot be
selected. Hermes core source contains **zero** Steel references, so there is no Steel fallback path in the
tree. Backups: `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z`,
`config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z`.
- **CT201 is clean.** The containers `steel-browser-api-1` and `steel-browser-ui-1`, the `steel-network` Docker
network, the images `steel-browser-api:fixed` / `ghcr.io/steel-dev/steel-browser-api:latest` /
`ghcr.io/steel-dev/steel-browser-ui:latest`, and the project dir `/root/steel-browser` were all removed
(~6 GB reclaimed; disk 70G used → 64G used). The Tailscale Serve routes `/steel` and `/steel-api` were
deleted with `tailscale serve --set-path <path> off`; CT201's remaining routes are only `/` → `127.0.0.1:8085`
and `/mgr` → `127.0.0.1:8084`, on **both** `:443` and `:8443`. No listeners remain on 3000, 5173 or 9223.
- **CT333's second, separate Steel copy is gone too.** The orphaned `steel-browser` container (image
`ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image
were removed; no Steel containers, images or listeners remain on CT333.
- **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, plus
`/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io.
- **Camofox is healthy and working** — REST `http://100.118.5.51:9377` (and `http://localhost:9377`);
`/health` returns **200**, `/tabs` returns `{"running":true,"tabs":[]}`.
- **Stale environment variable — now cleared from the gateway.** `STEEL_BASE_URL` is no longer in
`~/.hermes/.env`. It survived in the environment of already-running processes (inherited at spawn), so the
**gateway was restarted on 2026-10-29 01:21 AWST** to clear it: the new gateway tree (PID `3913841`) reads
`STEEL_BASE_URL=0`. A few older, unrelated services still carry the variable in their inherited
environment (the dashboard unit and `agentmail-webhook.service`) — **no core Hermes code reads
`STEEL_BASE_URL`**, so it is inert either way, and it is a runtime-only leftover, **not** a config value.
> ⚠ **Superseded (the 2026-10-29 00:3x probe):** an earlier version of this correction described Steel as
> merely "not reachable — endpoint dark, still running on CT201". That is outdated: Steel is now **removed**,
> not down.
## Related
- [[comparisons/steel-vs-camoufox]]
- [[concepts/browser-automation]]
- [[decisions/rollback-steel-to-camofox]]
- [[runbooks/restart-browser]]