Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline. NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450). Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
6.8 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Browser Backend | system | active | 2026-07-22 | 2026-10-13 | 2026-10-13 | high |
|
|
Browser Backend
Purpose
Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.
Current Configuration (re-verified 2026-10-13)
- Primary backend: Camofox (Firefox-based, anti-fingerprint) —
systemd --userunitcamofox-browser.serviceon the Hermes host itself (the unit path is/home/hermes/.config/systemd/user/…, and the VNC route is named after CT460). The olderCT450attribution below was not reproducible on 2026-10-13 — see the correction note. - Provider config:
browser.cloud_provider: camofox,camera_provider: camofox - REST API:
:9377— live-verified 2026-10-13 (node server.js,CAMOFOX_PORT=9377;LISTEN *:9377). The:9093figure in earlier revisions of this page is stale — nothing was listening on 9093 on 2026-10-13. - VNC viewer: NovNC
websockifyon127.0.0.1:6080→127.0.0.1:5900, published ashttps://ct460.kangaroo-eel.ts.net/vnc-camofox - Steel Browser (CT201): Still running but Hermes not connected — plugin disabled
Camofox (local systemd --user service)
⚠ Earlier revisions titled this section "Camoufox (CT450)". Live-verified 2026-10-13: the service Hermes talks to is a
systemd --userunit on the Hermes host (/home/hermes/.config/systemd/user/camofox-browser.service), reached atlocalhost:9377, with its VNC route published as…/vnc-camofox. Treat any "runs on CT450 / CT201" claim as unverified until re-probed.
- Host: the Hermes host itself (CT460) —
systemd --userunit resolved onhostname = hermes, no network hop - Deployment: Systemd service
camofox-browser.service - Engine: Camoufox (custom Firefox build with anti-detection fingerprinting)
- Display: Headed via Xvfb
:99(1600×1200×24) + fluxbox window manager - Auth: Persistent profile
anthony— authenticated into Uber Eats, LinkedIn, etc. - Connection: REST API and WebSocket bridge both at
localhost:9377(see the port correction above) - Last verified: Jul 30 —
browserConnected: true,sessions: 1,tabs: 1, active 200s on scroll/navigate
Access
| Endpoint | URL | Used by |
|---|---|---|
| REST API | http://localhost:9377 |
Hermes browser tools + direct REST (/tabs, /start, /sessions) |
| VNC (NovNC) | https://ct460.kangaroo-eel.ts.net/vnc-camofox |
Visual session viewer (websockify 127.0.0.1:6080 → 127.0.0.1:5900) |
⚠ Port correction (2026-10-13): earlier revisions listed the REST API on
:9093and a separate CDP WebSocket on:9377. Live probe: the only Camofox listener isnode server.json*:9377(withCAMOFOX_URL=http://localhost:9377in the unit).:9093and:9378were both closed.:9377is the REST API and the WebSocket bridge — there is no separate 9093 service.
Config
cloud_provider: camofoxin~/.hermes/config.yamlCAMOFOX_URL=http://localhost:9377andCAMOFOX_API_KEYin~/.hermes/.env- Unit environment (
~/.config/systemd/user/camofox-browser.service):CAMOFOX_PORT=9377,CAMOFOX_PROFILE_DIR=/home/hermes/.camofox/profiles,ENABLE_VNC=1,HANDLER_TIMEOUT_MS=90000,NAVIGATE_TIMEOUT_MS=60000,SESSION_TIMEOUT_MS=600000,TAB_INACTIVITY_MS=300000,BROWSER_IDLE_TIMEOUT_MS=300000,Restart=always,RestartSec=5. (The unit also carries an API key value — deliberately not reproduced here.) - Profile:
anthony(502913bf) — NOTpersist-anthony. A clean public profilejobradaris used for job-board scraping.
Resource containment (live-verified 2026-10-13)
The unit carries a drop-in ~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf:
[Service]
MemoryHigh=2G
MemoryMax=3G
MemorySwapMax=1G
OOMPolicy=kill
Observed ceiling event — 2026-10-12 07:01–07:23 (Daily Marketing Job Radar). A long browser run drove the service to the top of that containment window:
- Mid-run:
Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M, swap peak: 1G), 383 tasks, uptime 9 days. Swap fully consumed, no reclaimable headroom. - Symptom: the REST
/tabsendpoint and page evals began hanging (they look like ordinary tool timeouts). - Recovery: restart
camofox-browser.service, then work one tab at a time, closing each before opening the next. Post-restart the host sat at ~2.1 GB of 14 GB.
Diagnosis: the limits held and OOMPolicy=kill never had to fire — the workload exceeded the cgroup,
not a leak. A green cron row therefore does not mean the browser layer was healthy; check
systemctl --user status camofox-browser.service (peak/swap/tasks) whenever a browser-heavy job runs long.
Restart (systemctl --user restart camofox-browser.service) is measurable roughly 16 h later as a healthy
baseline: Memory: 16.5M cgroup, peak: 1.4G, swap peak: 234.5M, 16 tasks, browserConnected: false
until the next session starts.
Steel Browser (CT201) — Disabled
- Status: Still running on CT201, Hermes not connected since 2026-07-30
- Plugin:
browser-steelat/home/hermes/.hermes/plugins/browser-steel/— disabled - CDP:
http://100.96.244.39:9223 - REST API:
http://100.96.244.39:3000 - UI:
https://runtipi.kangaroo-eel.ts.net/steel - See: comparisons/steel-vs-camoufox for the full experiment story
History
- 2026-07-22: Camoufox primary, cua-driver bridge
- 2026-07-29: Steel Browser deployed on CT201, became primary
- 2026-07-30: Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary.
- 2026-08-02: Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API)
- 2026-10-12 — containment ceiling hit under a browser-heavy cron run. Daily Marketing Job Radar drove
camofox-browser.serviceto2.5Gwith swap fully consumed (available: 0B, 383 tasks, 9-day uptime);/tabsand evals hung until the service was restarted at 07:23:50 and the job finished one tab at a time. Workload problem, not a leak — limits (MemoryHigh=2G/MemoryMax=3G/MemorySwapMax=1G) held. - 2026-10-13 — page corrected against a live probe. REST port
:9093→:9377;:9378(claimed "Camoufox UI") does not exist;CT450host attribution replaced with the localsystemd --userunit on the Hermes host (CT460). Containment drop-in documented. Same day, the 07:01 Job Radar session was audited end-to-end and its memory-pressure evidence added above.