153 lines
5.3 KiB
Markdown
153 lines
5.3 KiB
Markdown
---
|
|
title: Hermes Gateway Resource Resilience
|
|
type: runbook
|
|
status: active
|
|
created: 2026-07-25
|
|
updated: 2026-10-20
|
|
review_after: 2026-11-19
|
|
verified_on: 2026-10-20
|
|
version_applies_to: Proxmox CT460
|
|
confidence: high
|
|
tags: [hermes, infrastructure, runbook]
|
|
sources: [live systemd and cgroup v2 state on CT460]
|
|
---
|
|
|
|
# Hermes Gateway Resource Resilience
|
|
|
|
## Purpose
|
|
|
|
Keep Telegram and the Hermes gateway available when the browser or Web UI
|
|
causes severe memory pressure inside CT460.
|
|
|
|
The observed failure was real: the CT460 memory cgroup OOM-killed a large Web
|
|
UI process and a Chrome child. Because the gateway previously used
|
|
`OOMPolicy=stop`, the Chrome child death caused systemd to stop the entire
|
|
gateway until `Restart=always` brought it back.
|
|
|
|
## Active controls
|
|
|
|
### Gateway
|
|
|
|
File:
|
|
`/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf`
|
|
|
|
```ini
|
|
[Service]
|
|
OOMPolicy=continue
|
|
MemoryLow=512M
|
|
CPUWeight=1000
|
|
```
|
|
|
|
The gateway has no `MemoryHigh` or `MemoryMax`. It receives reclaim protection
|
|
and maximum relative CPU weight, while `OOMPolicy=continue` prevents an
|
|
OOM-killed browser child from stopping the whole unit. If the main gateway
|
|
process exits, its existing `Restart=always` policy still restarts it.
|
|
|
|
### Camofox
|
|
|
|
File:
|
|
`/home/hermes/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`
|
|
|
|
```ini
|
|
[Service]
|
|
MemoryHigh=2G
|
|
MemoryMax=3G
|
|
MemorySwapMax=1G
|
|
OOMPolicy=kill
|
|
```
|
|
|
|
> **⚠ Limits have been RAISED since 2026-07-25 (verified 2026-10-20).** The Camofox drop-in now
|
|
> reads `MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G` — the page previously recorded
|
|
> `1G` / `1536M` / `768M`. Effective values confirmed via
|
|
> `systemctl --user show camofox-browser.service`:
|
|
> `MemoryHigh=2147483648` (2 GiB), `MemoryMax=3221225472` (3 GiB),
|
|
> `MemorySwapMax=1073741824` (1 GiB), `OOMPolicy=kill`.
|
|
> The `OOMPolicy=kill` intent is unchanged; only the ceilings moved. The Camofox service **is**
|
|
> running (`camofox-browser.service`, active/running) — it is the browser workload the containment
|
|
> is meant to protect, and it is the unit that grew.
|
|
|
|
### Hermes Web UI
|
|
|
|
File:
|
|
`/etc/systemd/system/hermes-webui.service.d/resource-containment.conf`
|
|
|
|
```ini
|
|
[Service]
|
|
MemoryHigh=1G
|
|
MemoryMax=1536M
|
|
MemorySwapMax=1G
|
|
OOMPolicy=kill
|
|
```
|
|
|
|
Camofox and Web UI are allowed to restart under extreme growth instead of
|
|
consuming most of the container and taking Telegram down with them.
|
|
|
|
> **Re-verified 2026-10-20 — the gateway and Web UI controls are still exactly as documented.**
|
|
> - Gateway drop-in `/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf`
|
|
> still reads `OOMPolicy=continue`, `MemoryLow=512M`, `CPUWeight=1000` (no `MemoryHigh`/`MemoryMax`).
|
|
> Effective: `OOMPolicy=continue`, `CPUWeight=1000`, `MemoryLow=536870912`,
|
|
> `MemoryHigh=infinity`, `MemoryMax=infinity` — **matches the "Expected gateway values" note below.**
|
|
> - Web UI: `OOMPolicy=kill`, `MemoryHigh=1073741824` (1 GiB), `MemoryMax=1610612736` (1.5 GiB),
|
|
> `MemorySwapMax=1073741824` (1 GiB) — **matches the drop-in shown above, unchanged.**
|
|
> - Camofox: see the raised limits noted above.
|
|
> - `hermes-gateway.service` and `camofox-browser.service` are both **active (running)**.
|
|
>
|
|
> **Note:** the Web UI drop-in lives at `/etc/systemd/system/hermes-webui.service.d/` which the
|
|
> `hermes` user cannot read (permission denied) — its *effective* values are still verifiable via
|
|
> `systemctl show`, which is what the health-check section below uses.
|
|
|
|
## I/O priority limitation
|
|
|
|
Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation
|
|
is changed and verified. CT460 exposes the I/O controller at its top cgroup,
|
|
but it is not delegated into the container's systemd service tree. Systemd
|
|
accepts the property, but no live `io.weight` control is created.
|
|
|
|
## Health check
|
|
|
|
```bash
|
|
systemctl is-active hermes-webui.service
|
|
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
|
|
systemctl --user is-active camofox-browser.service hermes-gateway.service
|
|
```
|
|
|
|
Check effective limits:
|
|
|
|
```bash
|
|
systemctl show hermes-webui.service \
|
|
-p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
|
|
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
|
|
systemctl --user show hermes-gateway.service \
|
|
-p OOMPolicy -p MemoryLow -p MemoryHigh -p MemoryMax -p CPUWeight
|
|
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
|
|
systemctl --user show camofox-browser.service \
|
|
-p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
|
|
```
|
|
|
|
Expected gateway values include `OOMPolicy=continue`,
|
|
`MemoryLow=536870912`, `MemoryHigh=infinity`, `MemoryMax=infinity`, and
|
|
`CPUWeight=1000`. **Confirmed live on 2026-10-20** — see the note under "Active controls".
|
|
|
|
## Backups and rollback
|
|
|
|
Original source units are backed up at:
|
|
|
|
`/home/hermes/.hermes/backups/resource-hardening-20260725/`
|
|
|
|
To roll back, remove only these three drop-ins:
|
|
|
|
- `resource-resilience.conf` for `hermes-gateway.service`
|
|
- `resource-containment.conf` for `camofox-browser.service`
|
|
- `resource-containment.conf` for `hermes-webui.service`
|
|
|
|
Then reload the system and user systemd managers and restart only those three
|
|
services. The original source units were never modified.
|
|
|
|
## Related
|
|
|
|
- [[systems/hermes-agent]]
|
|
- [[systems/messaging-integrations]]
|
|
- [[systems/browser-backend]]
|
|
- [[runbooks/restart-hermes]]
|
|
- [[systems/tool-search]]
|