--- title: Hermes Gateway Resource Resilience type: runbook status: active created: 2026-07-25 updated: 2026-10-20 review_after: 2026-11-19 verified_on: 2026-10-20 version_applies_to: Proxmox CT460 confidence: high tags: [hermes, infrastructure, runbook] sources: [live systemd and cgroup v2 state on CT460] --- # Hermes Gateway Resource Resilience ## Purpose Keep Telegram and the Hermes gateway available when the browser or Web UI causes severe memory pressure inside CT460. The observed failure was real: the CT460 memory cgroup OOM-killed a large Web UI process and a Chrome child. Because the gateway previously used `OOMPolicy=stop`, the Chrome child death caused systemd to stop the entire gateway until `Restart=always` brought it back. ## Active controls ### Gateway File: `/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf` ```ini [Service] OOMPolicy=continue MemoryLow=512M CPUWeight=1000 ``` The gateway has no `MemoryHigh` or `MemoryMax`. It receives reclaim protection and maximum relative CPU weight, while `OOMPolicy=continue` prevents an OOM-killed browser child from stopping the whole unit. If the main gateway process exits, its existing `Restart=always` policy still restarts it. ### Camofox File: `/home/hermes/.config/systemd/user/camofox-browser.service.d/resource-containment.conf` ```ini [Service] MemoryHigh=2G MemoryMax=3G MemorySwapMax=1G OOMPolicy=kill ``` > **⚠ Limits have been RAISED since 2026-07-25 (verified 2026-10-20).** The Camofox drop-in now > reads `MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G` — the page previously recorded > `1G` / `1536M` / `768M`. Effective values confirmed via > `systemctl --user show camofox-browser.service`: > `MemoryHigh=2147483648` (2 GiB), `MemoryMax=3221225472` (3 GiB), > `MemorySwapMax=1073741824` (1 GiB), `OOMPolicy=kill`. > The `OOMPolicy=kill` intent is unchanged; only the ceilings moved. The Camofox service **is** > running (`camofox-browser.service`, active/running) — it is the browser workload the containment > is meant to protect, and it is the unit that grew. ### Hermes Web UI File: `/etc/systemd/system/hermes-webui.service.d/resource-containment.conf` ```ini [Service] MemoryHigh=1G MemoryMax=1536M MemorySwapMax=1G OOMPolicy=kill ``` Camofox and Web UI are allowed to restart under extreme growth instead of consuming most of the container and taking Telegram down with them. > **Re-verified 2026-10-20 — the gateway and Web UI controls are still exactly as documented.** > - Gateway drop-in `/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf` > still reads `OOMPolicy=continue`, `MemoryLow=512M`, `CPUWeight=1000` (no `MemoryHigh`/`MemoryMax`). > Effective: `OOMPolicy=continue`, `CPUWeight=1000`, `MemoryLow=536870912`, > `MemoryHigh=infinity`, `MemoryMax=infinity` — **matches the "Expected gateway values" note below.** > - Web UI: `OOMPolicy=kill`, `MemoryHigh=1073741824` (1 GiB), `MemoryMax=1610612736` (1.5 GiB), > `MemorySwapMax=1073741824` (1 GiB) — **matches the drop-in shown above, unchanged.** > - Camofox: see the raised limits noted above. > - `hermes-gateway.service` and `camofox-browser.service` are both **active (running)**. > > **Note:** the Web UI drop-in lives at `/etc/systemd/system/hermes-webui.service.d/` which the > `hermes` user cannot read (permission denied) — its *effective* values are still verifiable via > `systemctl show`, which is what the health-check section below uses. ## I/O priority limitation Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation is changed and verified. CT460 exposes the I/O controller at its top cgroup, but it is not delegated into the container's systemd service tree. Systemd accepts the property, but no live `io.weight` control is created. ## Health check ```bash systemctl is-active hermes-webui.service runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \ systemctl --user is-active camofox-browser.service hermes-gateway.service ``` Check effective limits: ```bash systemctl show hermes-webui.service \ -p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \ systemctl --user show hermes-gateway.service \ -p OOMPolicy -p MemoryLow -p MemoryHigh -p MemoryMax -p CPUWeight runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \ systemctl --user show camofox-browser.service \ -p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax ``` Expected gateway values include `OOMPolicy=continue`, `MemoryLow=536870912`, `MemoryHigh=infinity`, `MemoryMax=infinity`, and `CPUWeight=1000`. **Confirmed live on 2026-10-20** — see the note under "Active controls". ## Backups and rollback Original source units are backed up at: `/home/hermes/.hermes/backups/resource-hardening-20260725/` To roll back, remove only these three drop-ins: - `resource-resilience.conf` for `hermes-gateway.service` - `resource-containment.conf` for `camofox-browser.service` - `resource-containment.conf` for `hermes-webui.service` Then reload the system and user systemd managers and restart only those three services. The original source units were never modified. ## Related - [[systems/hermes-agent]] - [[systems/messaging-integrations]] - [[systems/browser-backend]] - [[runbooks/restart-hermes]] - [[systems/tool-search]]