Files
agent-estate-wiki/systems/browser-backend.md
T
hermes-audit ebf57146f1 audit(2026-10-13): 10-12 clean (174/0, 3rd day) but GitHub Trending Radar token DEAD (getMe 401) - supersedes the wrong 2026-10-06 'refuted' entry
Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline.

NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450).

Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
2026-10-13 00:19:15 +08:00

6.8 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Browser Backend system active 2026-07-22 2026-10-13 2026-10-13 high
browser
camofox
automation
live-probe
config
session-history

Browser Backend

Purpose

Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.

Current Configuration (re-verified 2026-10-13)

  • Primary backend: Camofox (Firefox-based, anti-fingerprint) — systemd --user unit camofox-browser.service on the Hermes host itself (the unit path is /home/hermes/.config/systemd/user/…, and the VNC route is named after CT460). The older CT450 attribution below was not reproducible on 2026-10-13 — see the correction note.
  • Provider config: browser.cloud_provider: camofox, camera_provider: camofox
  • REST API: :9377 — live-verified 2026-10-13 (node server.js, CAMOFOX_PORT=9377; LISTEN *:9377). The :9093 figure in earlier revisions of this page is stale — nothing was listening on 9093 on 2026-10-13.
  • VNC viewer: NovNC websockify on 127.0.0.1:6080 → 127.0.0.1:5900, published as https://ct460.kangaroo-eel.ts.net/vnc-camofox
  • Steel Browser (CT201): Still running but Hermes not connected — plugin disabled

Camofox (local systemd --user service)

⚠ Earlier revisions titled this section "Camoufox (CT450)". Live-verified 2026-10-13: the service Hermes talks to is a systemd --user unit on the Hermes host (/home/hermes/.config/systemd/user/camofox-browser.service), reached at localhost:9377, with its VNC route published as …/vnc-camofox. Treat any "runs on CT450 / CT201" claim as unverified until re-probed.

  • Host: the Hermes host itself (CT460) — systemd --user unit resolved on hostname = hermes, no network hop
  • Deployment: Systemd service camofox-browser.service
  • Engine: Camoufox (custom Firefox build with anti-detection fingerprinting)
  • Display: Headed via Xvfb :99 (1600×1200×24) + fluxbox window manager
  • Auth: Persistent profile anthony — authenticated into Uber Eats, LinkedIn, etc.
  • Connection: REST API and WebSocket bridge both at localhost:9377 (see the port correction above)
  • Last verified: Jul 30 — browserConnected: true, sessions: 1, tabs: 1, active 200s on scroll/navigate

Access

Endpoint URL Used by
REST API http://localhost:9377 Hermes browser tools + direct REST (/tabs, /start, /sessions)
VNC (NovNC) https://ct460.kangaroo-eel.ts.net/vnc-camofox Visual session viewer (websockify 127.0.0.1:6080 → 127.0.0.1:5900)

⚠ Port correction (2026-10-13): earlier revisions listed the REST API on :9093 and a separate CDP WebSocket on :9377. Live probe: the only Camofox listener is node server.js on *:9377 (with CAMOFOX_URL=http://localhost:9377 in the unit). :9093 and :9378 were both closed. :9377 is the REST API and the WebSocket bridge — there is no separate 9093 service.

Config

  • cloud_provider: camofox in ~/.hermes/config.yaml
  • CAMOFOX_URL=http://localhost:9377 and CAMOFOX_API_KEY in ~/.hermes/.env
  • Unit environment (~/.config/systemd/user/camofox-browser.service): CAMOFOX_PORT=9377, CAMOFOX_PROFILE_DIR=/home/hermes/.camofox/profiles, ENABLE_VNC=1, HANDLER_TIMEOUT_MS=90000, NAVIGATE_TIMEOUT_MS=60000, SESSION_TIMEOUT_MS=600000, TAB_INACTIVITY_MS=300000, BROWSER_IDLE_TIMEOUT_MS=300000, Restart=always, RestartSec=5. (The unit also carries an API key value — deliberately not reproduced here.)
  • Profile: anthony (502913bf) — NOT persist-anthony. A clean public profile jobradar is used for job-board scraping.

Resource containment (live-verified 2026-10-13)

The unit carries a drop-in ~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf:

[Service]
MemoryHigh=2G
MemoryMax=3G
MemorySwapMax=1G
OOMPolicy=kill

Observed ceiling event — 2026-10-12 07:01–07:23 (Daily Marketing Job Radar). A long browser run drove the service to the top of that containment window:

  • Mid-run: Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M, swap peak: 1G), 383 tasks, uptime 9 days. Swap fully consumed, no reclaimable headroom.
  • Symptom: the REST /tabs endpoint and page evals began hanging (they look like ordinary tool timeouts).
  • Recovery: restart camofox-browser.service, then work one tab at a time, closing each before opening the next. Post-restart the host sat at ~2.1 GB of 14 GB.

Diagnosis: the limits held and OOMPolicy=kill never had to fire — the workload exceeded the cgroup, not a leak. A green cron row therefore does not mean the browser layer was healthy; check systemctl --user status camofox-browser.service (peak/swap/tasks) whenever a browser-heavy job runs long.

Restart (systemctl --user restart camofox-browser.service) is measurable roughly 16 h later as a healthy baseline: Memory: 16.5M cgroup, peak: 1.4G, swap peak: 234.5M, 16 tasks, browserConnected: false until the next session starts.

Steel Browser (CT201) — Disabled

  • Status: Still running on CT201, Hermes not connected since 2026-07-30
  • Plugin: browser-steel at /home/hermes/.hermes/plugins/browser-steel/ — disabled
  • CDP: http://100.96.244.39:9223
  • REST API: http://100.96.244.39:3000
  • UI: https://runtipi.kangaroo-eel.ts.net/steel
  • See: comparisons/steel-vs-camoufox for the full experiment story

History

  • 2026-07-22: Camoufox primary, cua-driver bridge
  • 2026-07-29: Steel Browser deployed on CT201, became primary
  • 2026-07-30: Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary.
  • 2026-08-02: Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API)
  • 2026-10-12 — containment ceiling hit under a browser-heavy cron run. Daily Marketing Job Radar drove camofox-browser.service to 2.5G with swap fully consumed (available: 0B, 383 tasks, 9-day uptime); /tabs and evals hung until the service was restarted at 07:23:50 and the job finished one tab at a time. Workload problem, not a leak — limits (MemoryHigh=2G/MemoryMax=3G/MemorySwapMax=1G) held.
  • 2026-10-13 — page corrected against a live probe. REST port :9093 → :9377; :9378 (claimed "Camoufox UI") does not exist; CT450 host attribution replaced with the local systemd --user unit on the Hermes host (CT460). Containment drop-in documented. Same day, the 07:01 Job Radar session was audited end-to-end and its memory-pressure evidence added above.