---
title: Hermes Gateway Resource Resilience
type: runbook
status: active
created: 2026-07-25
updated: 2026-07-25
review_after: 2026-10-23
verified_on: 2026-07-25
version_applies_to: Proxmox CT460
confidence: high
tags: [hermes, infrastructure, runbook]
sources: [live systemd and cgroup v2 state on CT460]
---

# Hermes Gateway Resource Resilience

## Purpose

Keep Telegram and the Hermes gateway available when the browser or Web UI
causes severe memory pressure inside CT460.

The observed failure was real: the CT460 memory cgroup OOM-killed a large Web
UI process and a Chrome child. Because the gateway previously used
`OOMPolicy=stop`, the Chrome child death caused systemd to stop the entire
gateway until `Restart=always` brought it back.

## Active controls

### Gateway

File:
`/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf`

```ini
[Service]
OOMPolicy=continue
MemoryLow=512M
CPUWeight=1000
```

The gateway has no `MemoryHigh` or `MemoryMax`. It receives reclaim protection
and maximum relative CPU weight, while `OOMPolicy=continue` prevents an
OOM-killed browser child from stopping the whole unit. If the main gateway
process exits, its existing `Restart=always` policy still restarts it.

### Camofox

File:
`/home/hermes/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`

```ini
[Service]
MemoryHigh=1G
MemoryMax=1536M
MemorySwapMax=768M
OOMPolicy=kill
```

### Hermes Web UI

File:
`/etc/systemd/system/hermes-webui.service.d/resource-containment.conf`

```ini
[Service]
MemoryHigh=1G
MemoryMax=1536M
MemorySwapMax=1G
OOMPolicy=kill
```

Camofox and Web UI are allowed to restart under extreme growth instead of
consuming most of the container and taking Telegram down with them.

## I/O priority limitation

Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation
is changed and verified. CT460 exposes the I/O controller at its top cgroup,
but it is not delegated into the container's systemd service tree. Systemd
accepts the property, but no live `io.weight` control is created.

## Health check

```bash
systemctl is-active hermes-webui.service
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
  systemctl --user is-active camofox-browser.service hermes-gateway.service
```

Check effective limits:

```bash
systemctl show hermes-webui.service \
  -p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
  systemctl --user show hermes-gateway.service \
  -p OOMPolicy -p MemoryLow -p MemoryHigh -p MemoryMax -p CPUWeight
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
  systemctl --user show camofox-browser.service \
  -p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
```

Expected gateway values include `OOMPolicy=continue`,
`MemoryLow=536870912`, `MemoryHigh=infinity`, `MemoryMax=infinity`, and
`CPUWeight=1000`.

## Backups and rollback

Original source units are backed up at:

`/home/hermes/.hermes/backups/resource-hardening-20260725/`

To roll back, remove only these three drop-ins:

- `resource-resilience.conf` for `hermes-gateway.service`
- `resource-containment.conf` for `camofox-browser.service`
- `resource-containment.conf` for `hermes-webui.service`

Then reload the system and user systemd managers and restart only those three
services. The original source units were never modified.

## Related

- [[systems/hermes-agent]]
- [[systems/messaging-integrations]]
- [[systems/browser-backend]]
- [[runbooks/restart-hermes]]
- [[systems/tool-search]]