Agent Estate Wiki

Hermes Gateway Resource Resilience

Properties1
tagshermes, infrastructure, runbook

2 min read

Hermes Gateway Resource Resilience

Purpose

Keep Telegram and the Hermes gateway available when the browser or Web UI causes severe memory pressure inside CT460.

The observed failure was real: the CT460 memory cgroup OOM-killed a large Web UI process and a Chrome child. Because the gateway previously used OOMPolicy=stop, the Chrome child death caused systemd to stop the entire gateway until Restart=always brought it back.

Active controls

Gateway

File: /home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf

[Service]
OOMPolicy=continue
MemoryLow=512M
CPUWeight=1000

The gateway has no MemoryHigh or MemoryMax. It receives reclaim protection and maximum relative CPU weight, while OOMPolicy=continue prevents an OOM-killed browser child from stopping the whole unit. If the main gateway process exits, its existing Restart=always policy still restarts it.

Camofox

File: /home/hermes/.config/systemd/user/camofox-browser.service.d/resource-containment.conf

[Service]
MemoryHigh=1G
MemoryMax=1536M
MemorySwapMax=768M
OOMPolicy=kill

Hermes Web UI

File: /etc/systemd/system/hermes-webui.service.d/resource-containment.conf

[Service]
MemoryHigh=1G
MemoryMax=1536M
MemorySwapMax=1G
OOMPolicy=kill

Camofox and Web UI are allowed to restart under extreme growth instead of consuming most of the container and taking Telegram down with them.

I/O priority limitation

Do not add IOWeight to the gateway drop-in unless Proxmox cgroup delegation is changed and verified. CT460 exposes the I/O controller at its top cgroup, but it is not delegated into the container’s systemd service tree. Systemd accepts the property, but no live io.weight control is created.

Health check

systemctl is-active hermes-webui.service
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
  systemctl --user is-active camofox-browser.service hermes-gateway.service

Check effective limits:

systemctl show hermes-webui.service \
  -p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
  systemctl --user show hermes-gateway.service \
  -p OOMPolicy -p MemoryLow -p MemoryHigh -p MemoryMax -p CPUWeight
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
  systemctl --user show camofox-browser.service \
  -p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax

Expected gateway values include OOMPolicy=continue, MemoryLow=536870912, MemoryHigh=infinity, MemoryMax=infinity, and CPUWeight=1000.

Backups and rollback

Original source units are backed up at:

/home/hermes/.hermes/backups/resource-hardening-20260725/

To roll back, remove only these three drop-ins:

  • resource-resilience.conf for hermes-gateway.service
  • resource-containment.conf for camofox-browser.service
  • resource-containment.conf for hermes-webui.service

Then reload the system and user systemd managers and restart only those three services. The original source units were never modified.

Graph View

Backlinks