Files
agent-estate-wiki/systems/browser-backend.md
T
Tony0410andHermes 677bb0c959 docs(wiki): correct Steel Browser references across 15 pages — Steel fully removed, Camofox is sole backend
Root cause of the long-standing cloud_provider: steel residue: the 2026-07-30
rollback disabled the browser-steel plugin and set browser.engine: camofox but
left browser.cloud_provider: steel in config.yaml. The split config was
inert-but-wrong, so it survived ~3 months of audits unnoticed.

Corrects pages that still described Steel as present, running or configured:
- systems/browser-backend.md (incl. gateway-restart env-var note)
- comparisons/steel-vs-camoufox.md, comparisons/camoufox-vs-chrome.md
- decisions/rollback-steel-to-camofox.md, decisions/index.md
- infrastructure/docker-services.md, infrastructure/tailscale.md, infrastructure/hosts.md
- systems/homepage-dashboard.md, systems/scheduled-tasks.md
- concepts/browser-automation.md, runbooks/recover-docker-service.md
- current-state.md, index.md, agents/nanobot.md
- log.md (dated entry; supersedes the 00:3x audit's 'endpoint dark' wording)

Also restores the Ignis compose row in infrastructure/docker-services.md, which
the 2026-10-11 revision dropped (verified: /home/hermes/docker/ignis/docker-compose.yml
exists and the ignis container is Up on CT460).

Co-Authored-By: Hermes <hermes@local>
2026-10-29 02:02:49 +08:00

11 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Browser Backend system active 2026-07-22 2026-10-29 2026-10-29 high
browser
camofox
automation
live-probe
config
session-history

Browser Backend

Purpose

Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.

Current Configuration (re-verified 2026-10-13)

  • Primary backend: Camofox (Firefox-based, anti-fingerprint) — systemd --user unit camofox-browser.service on the Hermes host itself (the unit path is /home/hermes/.config/systemd/user/…, and the VNC route is named after CT460). The older CT450 attribution below was not reproducible on 2026-10-13 — see the correction note.
  • Provider config: browser.cloud_provider: camofox, camera_provider: camofox
  • REST API: :9377 — live-verified 2026-10-13 (node server.js, CAMOFOX_PORT=9377; LISTEN *:9377). The :9093 figure in earlier revisions of this page is stale — nothing was listening on 9093 on 2026-10-13.
  • VNC viewer: NovNC websockify on 127.0.0.1:6080 → 127.0.0.1:5900, published as https://ct460.kangaroo-eel.ts.net/vnc-camofox
  • Steel Browser (CT201): Fully removed on 2026-10-29 — plugin deleted, CT201 deployment torn down, Serve routes deleted. Camofox is the sole backend. See the 2026-10-29 correction below.

Camofox (local systemd --user service)

⚠ Earlier revisions titled this section "Camoufox (CT450)". Live-verified 2026-10-13: the service Hermes talks to is a systemd --user unit on the Hermes host (/home/hermes/.config/systemd/user/camofox-browser.service), reached at localhost:9377, with its VNC route published as …/vnc-camofox. Treat any "runs on CT450 / CT201" claim as unverified until re-probed.

  • Host: the Hermes host itself (CT460) — systemd --user unit resolved on hostname = hermes, no network hop
  • Deployment: Systemd service camofox-browser.service
  • Engine: Camoufox (custom Firefox build with anti-detection fingerprinting)
  • Display: Headed via Xvfb :99 (1600×1200×24) + fluxbox window manager
  • Auth: Persistent profile anthony — authenticated into Uber Eats, LinkedIn, etc.
  • Connection: REST API and WebSocket bridge both at localhost:9377 (see the port correction above)
  • Last verified: Jul 30 — browserConnected: true, sessions: 1, tabs: 1, active 200s on scroll/navigate

Access

Endpoint URL Used by
REST API http://localhost:9377 Hermes browser tools + direct REST (/tabs, /start, /sessions)
VNC (NovNC) https://ct460.kangaroo-eel.ts.net/vnc-camofox Visual session viewer (websockify 127.0.0.1:6080 → 127.0.0.1:5900)

⚠ Port correction (2026-10-13): earlier revisions listed the REST API on :9093 and a separate CDP WebSocket on :9377. Live probe: the only Camofox listener is node server.js on *:9377 (with CAMOFOX_URL=http://localhost:9377 in the unit). :9093 and :9378 were both closed. :9377 is the REST API and the WebSocket bridge — there is no separate 9093 service.

Config

  • cloud_provider: camofox in ~/.hermes/config.yaml
  • CAMOFOX_URL=http://localhost:9377 and CAMOFOX_API_KEY in ~/.hermes/.env
  • Unit environment (~/.config/systemd/user/camofox-browser.service): CAMOFOX_PORT=9377, CAMOFOX_PROFILE_DIR=/home/hermes/.camofox/profiles, ENABLE_VNC=1, HANDLER_TIMEOUT_MS=90000, NAVIGATE_TIMEOUT_MS=60000, SESSION_TIMEOUT_MS=600000, TAB_INACTIVITY_MS=300000, BROWSER_IDLE_TIMEOUT_MS=300000, Restart=always, RestartSec=5. (The unit also carries an API key value — deliberately not reproduced here.)
  • Profile: anthony (502913bf) — NOT persist-anthony. A clean public profile jobradar is used for job-board scraping.

Resource containment (live-verified 2026-10-13)

The unit carries a drop-in ~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf:

[Service]
MemoryHigh=2G
MemoryMax=3G
MemorySwapMax=1G
OOMPolicy=kill

Observed ceiling event — 2026-10-12 07:01–07:23 (Daily Marketing Job Radar). A long browser run drove the service to the top of that containment window:

  • Mid-run: Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M, swap peak: 1G), 383 tasks, uptime 9 days. Swap fully consumed, no reclaimable headroom.
  • Symptom: the REST /tabs endpoint and page evals began hanging (they look like ordinary tool timeouts).
  • Recovery: restart camofox-browser.service, then work one tab at a time, closing each before opening the next. Post-restart the host sat at ~2.1 GB of 14 GB.

Diagnosis: the limits held and OOMPolicy=kill never had to fire — the workload exceeded the cgroup, not a leak. A green cron row therefore does not mean the browser layer was healthy; check systemctl --user status camofox-browser.service (peak/swap/tasks) whenever a browser-heavy job runs long.

Restart (systemctl --user restart camofox-browser.service) is measurable roughly 16 h later as a healthy baseline: Memory: 16.5M cgroup, peak: 1.4G, swap peak: 234.5M, 16 tasks, browserConnected: false until the next session starts.

Steel Browser (CT201) — Removed (2026-10-29)

  • Status: Fully removed. Nothing Steel-related survives on CT201 or CT333 — containers, images, network, project dir and Serve routes are all gone. Camofox is the only browser backend.
  • Plugin: browser-steel — deleted (~/.hermes/plugins/browser-steel/ no longer exists; no browser-steel key in plugins.disabled or plugins.entries)
  • CDP / REST / UI: retired — the former CDP :9223, REST :3000 and /steel UI addresses no longer serve anything
  • Reversibility: the compose file and Dockerfile are archived on CT201 under /root/_archived-steel/, and the images remain re-pullable from ghcr.io
  • See: comparisons/steel-vs-camoufox for the full experiment story

History

  • 2026-07-22: Camoufox primary, cua-driver bridge
  • 2026-07-29: Steel Browser deployed on CT201, became primary
  • 2026-07-30: Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary.
  • 2026-08-02: Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API)
  • 2026-10-12 — containment ceiling hit under a browser-heavy cron run. Daily Marketing Job Radar drove camofox-browser.service to 2.5G with swap fully consumed (available: 0B, 383 tasks, 9-day uptime); /tabs and evals hung until the service was restarted at 07:23:50 and the job finished one tab at a time. Workload problem, not a leak — limits (MemoryHigh=2G/MemoryMax=3G/MemorySwapMax=1G) held.
  • 2026-10-13 — page corrected against a live probe. REST port :9093 → :9377; :9378 (claimed "Camoufox UI") does not exist; CT450 host attribution replaced with the local systemd --user unit on the Hermes host (CT460). Containment drop-in documented. Same day, the 07:01 Job Radar session was audited end-to-end and its memory-pressure evidence added above.
  • 2026-10-29 — Steel fully removed. Config back to cloud_provider: camofox / camera_provider: camofox, the browser-steel plugin deleted, STEEL_BASE_URL dropped from ~/.hermes/.env, and the Steel deployments on CT201 (containers, images, network, project dir, Serve routes) and CT333 (container + image) torn down. Camofox is the sole backend.

Correction (2026-10-29) — Steel is out of the stack, not just disabled

Verified against live state at 2026-10-29. The earlier 00:3x probe said only "not reachable"; the truth is stronger — Steel is fully removed, not merely down:

  • Hermes is back on Camofox, and Camofox is the sole backend. The browser: block in ~/.hermes/config.yaml reads exactly cloud_provider: camofox, camera_provider: camofox, engine: auto, cdp_url: "".
  • The browser-steel plugin is deleted — ~/.hermes/plugins/browser-steel/ no longer exists, and plugins.disabled / plugins.entries carry no browser-steel key, so the plugin_steel toolset cannot be selected. Hermes core source contains zero Steel references, so there is no Steel fallback path in the tree. Backups: config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z, config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z.
  • CT201 is clean. The containers steel-browser-api-1 and steel-browser-ui-1, the steel-network Docker network, the images steel-browser-api:fixed / ghcr.io/steel-dev/steel-browser-api:latest / ghcr.io/steel-dev/steel-browser-ui:latest, and the project dir /root/steel-browser were all removed (~6 GB reclaimed; disk 70G used → 64G used). The Tailscale Serve routes /steel and /steel-api were deleted with tailscale serve --set-path <path> off; CT201's remaining routes are only / → 127.0.0.1:8085 and /mgr → 127.0.0.1:8084, on both :443 and :8443. No listeners remain on 3000, 5173 or 9223.
  • CT333's second, separate Steel copy is gone too. The orphaned steel-browser container (image ghcr.io/steel-dev/steel-browser:latest, 5.53 GB, created 2026-05-30, publishing port 3000) and its image were removed; no Steel containers, images or listeners remain on CT333.
  • Reversibility: the compose file and Dockerfile are archived on CT201 under /root/_archived-steel/, plus /root/_archived-steel/steel-cache-archive-20261029.tgz; the images remain re-pullable from ghcr.io.
  • Camofox is healthy and working — REST http://100.118.5.51:9377 (and http://localhost:9377); /health returns 200, /tabs returns {"running":true,"tabs":[]}.
  • Stale environment variable — now cleared from the gateway. STEEL_BASE_URL is no longer in ~/.hermes/.env. It survived in the environment of already-running processes (inherited at spawn), so the gateway was restarted on 2026-10-29 01:21 AWST to clear it: the new gateway tree (PID 3913841) reads STEEL_BASE_URL=0. A few older, unrelated services still carry the variable in their inherited environment (the dashboard unit and agentmail-webhook.service) — no core Hermes code reads STEEL_BASE_URL, so it is inert either way, and it is a runtime-only leftover, not a config value.

⚠ Superseded (the 2026-10-29 00:3x probe): an earlier version of this correction described Steel as merely "not reachable — endpoint dark, still running on CT201". That is outdated: Steel is now removed, not down.