Root cause of the long-standing cloud_provider: steel residue: the 2026-07-30 rollback disabled the browser-steel plugin and set browser.engine: camofox but left browser.cloud_provider: steel in config.yaml. The split config was inert-but-wrong, so it survived ~3 months of audits unnoticed. Corrects pages that still described Steel as present, running or configured: - systems/browser-backend.md (incl. gateway-restart env-var note) - comparisons/steel-vs-camoufox.md, comparisons/camoufox-vs-chrome.md - decisions/rollback-steel-to-camofox.md, decisions/index.md - infrastructure/docker-services.md, infrastructure/tailscale.md, infrastructure/hosts.md - systems/homepage-dashboard.md, systems/scheduled-tasks.md - concepts/browser-automation.md, runbooks/recover-docker-service.md - current-state.md, index.md, agents/nanobot.md - log.md (dated entry; supersedes the 00:3x audit's 'endpoint dark' wording) Also restores the Ignis compose row in infrastructure/docker-services.md, which the 2026-10-11 revision dropped (verified: /home/hermes/docker/ignis/docker-compose.yml exists and the ignis container is Up on CT460). Co-Authored-By: Hermes <hermes@local>
11 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Browser Backend | system | active | 2026-07-22 | 2026-10-29 | 2026-10-29 | high |
|
|
Browser Backend
Purpose
Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.
Current Configuration (re-verified 2026-10-13)
- Primary backend: Camofox (Firefox-based, anti-fingerprint) —
systemd --userunitcamofox-browser.serviceon the Hermes host itself (the unit path is/home/hermes/.config/systemd/user/…, and the VNC route is named after CT460). The olderCT450attribution below was not reproducible on 2026-10-13 — see the correction note. - Provider config:
browser.cloud_provider: camofox,camera_provider: camofox - REST API:
:9377— live-verified 2026-10-13 (node server.js,CAMOFOX_PORT=9377;LISTEN *:9377). The:9093figure in earlier revisions of this page is stale — nothing was listening on 9093 on 2026-10-13. - VNC viewer: NovNC
websockifyon127.0.0.1:6080→127.0.0.1:5900, published ashttps://ct460.kangaroo-eel.ts.net/vnc-camofox - Steel Browser (CT201): Fully removed on 2026-10-29 — plugin deleted, CT201 deployment torn down, Serve routes deleted. Camofox is the sole backend. See the 2026-10-29 correction below.
Camofox (local systemd --user service)
⚠ Earlier revisions titled this section "Camoufox (CT450)". Live-verified 2026-10-13: the service Hermes talks to is a
systemd --userunit on the Hermes host (/home/hermes/.config/systemd/user/camofox-browser.service), reached atlocalhost:9377, with its VNC route published as…/vnc-camofox. Treat any "runs on CT450 / CT201" claim as unverified until re-probed.
- Host: the Hermes host itself (CT460) —
systemd --userunit resolved onhostname = hermes, no network hop - Deployment: Systemd service
camofox-browser.service - Engine: Camoufox (custom Firefox build with anti-detection fingerprinting)
- Display: Headed via Xvfb
:99(1600×1200×24) + fluxbox window manager - Auth: Persistent profile
anthony— authenticated into Uber Eats, LinkedIn, etc. - Connection: REST API and WebSocket bridge both at
localhost:9377(see the port correction above) - Last verified: Jul 30 —
browserConnected: true,sessions: 1,tabs: 1, active 200s on scroll/navigate
Access
| Endpoint | URL | Used by |
|---|---|---|
| REST API | http://localhost:9377 |
Hermes browser tools + direct REST (/tabs, /start, /sessions) |
| VNC (NovNC) | https://ct460.kangaroo-eel.ts.net/vnc-camofox |
Visual session viewer (websockify 127.0.0.1:6080 → 127.0.0.1:5900) |
⚠ Port correction (2026-10-13): earlier revisions listed the REST API on
:9093and a separate CDP WebSocket on:9377. Live probe: the only Camofox listener isnode server.json*:9377(withCAMOFOX_URL=http://localhost:9377in the unit).:9093and:9378were both closed.:9377is the REST API and the WebSocket bridge — there is no separate 9093 service.
Config
cloud_provider: camofoxin~/.hermes/config.yamlCAMOFOX_URL=http://localhost:9377andCAMOFOX_API_KEYin~/.hermes/.env- Unit environment (
~/.config/systemd/user/camofox-browser.service):CAMOFOX_PORT=9377,CAMOFOX_PROFILE_DIR=/home/hermes/.camofox/profiles,ENABLE_VNC=1,HANDLER_TIMEOUT_MS=90000,NAVIGATE_TIMEOUT_MS=60000,SESSION_TIMEOUT_MS=600000,TAB_INACTIVITY_MS=300000,BROWSER_IDLE_TIMEOUT_MS=300000,Restart=always,RestartSec=5. (The unit also carries an API key value — deliberately not reproduced here.) - Profile:
anthony(502913bf) — NOTpersist-anthony. A clean public profilejobradaris used for job-board scraping.
Resource containment (live-verified 2026-10-13)
The unit carries a drop-in ~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf:
[Service]
MemoryHigh=2G
MemoryMax=3G
MemorySwapMax=1G
OOMPolicy=kill
Observed ceiling event — 2026-10-12 07:01–07:23 (Daily Marketing Job Radar). A long browser run drove the service to the top of that containment window:
- Mid-run:
Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M, swap peak: 1G), 383 tasks, uptime 9 days. Swap fully consumed, no reclaimable headroom. - Symptom: the REST
/tabsendpoint and page evals began hanging (they look like ordinary tool timeouts). - Recovery: restart
camofox-browser.service, then work one tab at a time, closing each before opening the next. Post-restart the host sat at ~2.1 GB of 14 GB.
Diagnosis: the limits held and OOMPolicy=kill never had to fire — the workload exceeded the cgroup,
not a leak. A green cron row therefore does not mean the browser layer was healthy; check
systemctl --user status camofox-browser.service (peak/swap/tasks) whenever a browser-heavy job runs long.
Restart (systemctl --user restart camofox-browser.service) is measurable roughly 16 h later as a healthy
baseline: Memory: 16.5M cgroup, peak: 1.4G, swap peak: 234.5M, 16 tasks, browserConnected: false
until the next session starts.
Steel Browser (CT201) — Removed (2026-10-29)
- Status: Fully removed. Nothing Steel-related survives on CT201 or CT333 — containers, images, network, project dir and Serve routes are all gone. Camofox is the only browser backend.
- Plugin:
browser-steel— deleted (~/.hermes/plugins/browser-steel/no longer exists; nobrowser-steelkey inplugins.disabledorplugins.entries) - CDP / REST / UI: retired — the former CDP
:9223, REST:3000and/steelUI addresses no longer serve anything - Reversibility: the compose file and Dockerfile are archived on CT201 under
/root/_archived-steel/, and the images remain re-pullable from ghcr.io - See: comparisons/steel-vs-camoufox for the full experiment story
History
- 2026-07-22: Camoufox primary, cua-driver bridge
- 2026-07-29: Steel Browser deployed on CT201, became primary
- 2026-07-30: Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary.
- 2026-08-02: Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API)
- 2026-10-12 — containment ceiling hit under a browser-heavy cron run. Daily Marketing Job Radar drove
camofox-browser.serviceto2.5Gwith swap fully consumed (available: 0B, 383 tasks, 9-day uptime);/tabsand evals hung until the service was restarted at 07:23:50 and the job finished one tab at a time. Workload problem, not a leak — limits (MemoryHigh=2G/MemoryMax=3G/MemorySwapMax=1G) held. - 2026-10-13 — page corrected against a live probe. REST port
:9093→:9377;:9378(claimed "Camoufox UI") does not exist;CT450host attribution replaced with the localsystemd --userunit on the Hermes host (CT460). Containment drop-in documented. Same day, the 07:01 Job Radar session was audited end-to-end and its memory-pressure evidence added above. - 2026-10-29 — Steel fully removed. Config back to
cloud_provider: camofox/camera_provider: camofox, thebrowser-steelplugin deleted,STEEL_BASE_URLdropped from~/.hermes/.env, and the Steel deployments on CT201 (containers, images, network, project dir, Serve routes) and CT333 (container + image) torn down. Camofox is the sole backend.
Correction (2026-10-29) — Steel is out of the stack, not just disabled
Verified against live state at 2026-10-29. The earlier 00:3x probe said only "not reachable"; the truth is stronger — Steel is fully removed, not merely down:
- Hermes is back on Camofox, and Camofox is the sole backend. The
browser:block in~/.hermes/config.yamlreads exactlycloud_provider: camofox,camera_provider: camofox,engine: auto,cdp_url: "". - The
browser-steelplugin is deleted —~/.hermes/plugins/browser-steel/no longer exists, andplugins.disabled/plugins.entriescarry nobrowser-steelkey, so theplugin_steeltoolset cannot be selected. Hermes core source contains zero Steel references, so there is no Steel fallback path in the tree. Backups:config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z,config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z. - CT201 is clean. The containers
steel-browser-api-1andsteel-browser-ui-1, thesteel-networkDocker network, the imagessteel-browser-api:fixed/ghcr.io/steel-dev/steel-browser-api:latest/ghcr.io/steel-dev/steel-browser-ui:latest, and the project dir/root/steel-browserwere all removed (~6 GB reclaimed; disk 70G used → 64G used). The Tailscale Serve routes/steeland/steel-apiwere deleted withtailscale serve --set-path <path> off; CT201's remaining routes are only/→127.0.0.1:8085and/mgr→127.0.0.1:8084, on both:443and:8443. No listeners remain on 3000, 5173 or 9223. - CT333's second, separate Steel copy is gone too. The orphaned
steel-browsercontainer (imageghcr.io/steel-dev/steel-browser:latest, 5.53 GB, created 2026-05-30, publishing port 3000) and its image were removed; no Steel containers, images or listeners remain on CT333. - Reversibility: the compose file and Dockerfile are archived on CT201 under
/root/_archived-steel/, plus/root/_archived-steel/steel-cache-archive-20261029.tgz; the images remain re-pullable from ghcr.io. - Camofox is healthy and working — REST
http://100.118.5.51:9377(andhttp://localhost:9377);/healthreturns 200,/tabsreturns{"running":true,"tabs":[]}. - Stale environment variable — now cleared from the gateway.
STEEL_BASE_URLis no longer in~/.hermes/.env. It survived in the environment of already-running processes (inherited at spawn), so the gateway was restarted on 2026-10-29 01:21 AWST to clear it: the new gateway tree (PID3913841) readsSTEEL_BASE_URL=0. A few older, unrelated services still carry the variable in their inherited environment (the dashboard unit andagentmail-webhook.service) — no core Hermes code readsSTEEL_BASE_URL, so it is inert either way, and it is a runtime-only leftover, not a config value.
⚠ Superseded (the 2026-10-29 00:3x probe): an earlier version of this correction described Steel as merely "not reachable — endpoint dark, still running on CT201". That is outdated: Steel is now removed, not down.