snapshot: preserve central wiki state 2026-08-15 (7 modified + 1 new file)

2026-08-15 22:11:30 +08:00
commit 92e8028bb8
71 changed files with 9897 additions and 0 deletions
@@ -0,0 +1,128 @@
+---
+title: Hermes Gateway Resource Resilience
+type: runbook
+status: active
+created: 2026-07-25
+updated: 2026-07-25
+review_after: 2026-10-23
+verified_on: 2026-07-25
+version_applies_to: Proxmox CT460
+confidence: high
+tags: [hermes, infrastructure, runbook]
+sources: [live systemd and cgroup v2 state on CT460]
+---
+
+# Hermes Gateway Resource Resilience
+
+## Purpose
+
+Keep Telegram and the Hermes gateway available when the browser or Web UI
+causes severe memory pressure inside CT460.
+
+The observed failure was real: the CT460 memory cgroup OOM-killed a large Web
+UI process and a Chrome child. Because the gateway previously used
+`OOMPolicy=stop`, the Chrome child death caused systemd to stop the entire
+gateway until `Restart=always` brought it back.
+
+## Active controls
+
+### Gateway
+
+File:
+`/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf`
+
+```ini
+[Service]
+OOMPolicy=continue
+MemoryLow=512M
+CPUWeight=1000
+```
+
+The gateway has no `MemoryHigh` or `MemoryMax`. It receives reclaim protection
+and maximum relative CPU weight, while `OOMPolicy=continue` prevents an
+OOM-killed browser child from stopping the whole unit. If the main gateway
+process exits, its existing `Restart=always` policy still restarts it.
+
+### Camofox
+
+File:
+`/home/hermes/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`
+
+```ini
+[Service]
+MemoryHigh=1G
+MemoryMax=1536M
+MemorySwapMax=768M
+OOMPolicy=kill
+```
+
+### Hermes Web UI
+
+File:
+`/etc/systemd/system/hermes-webui.service.d/resource-containment.conf`
+
+```ini
+[Service]
+MemoryHigh=1G
+MemoryMax=1536M
+MemorySwapMax=1G
+OOMPolicy=kill
+```
+
+Camofox and Web UI are allowed to restart under extreme growth instead of
+consuming most of the container and taking Telegram down with them.
+
+## I/O priority limitation
+
+Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation
+is changed and verified. CT460 exposes the I/O controller at its top cgroup,
+but it is not delegated into the container's systemd service tree. Systemd
+accepts the property, but no live `io.weight` control is created.
+
+## Health check
+
+```bash
+systemctl is-active hermes-webui.service
+runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
+  systemctl --user is-active camofox-browser.service hermes-gateway.service
+```
+
+Check effective limits:
+
+```bash
+systemctl show hermes-webui.service \
+  -p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
+runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
+  systemctl --user show hermes-gateway.service \
+  -p OOMPolicy -p MemoryLow -p MemoryHigh -p MemoryMax -p CPUWeight
+runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
+  systemctl --user show camofox-browser.service \
+  -p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
+```
+
+Expected gateway values include `OOMPolicy=continue`,
+`MemoryLow=536870912`, `MemoryHigh=infinity`, `MemoryMax=infinity`, and
+`CPUWeight=1000`.
+
+## Backups and rollback
+
+Original source units are backed up at:
+
+`/home/hermes/.hermes/backups/resource-hardening-20260725/`
+
+To roll back, remove only these three drop-ins:
+
+- `resource-resilience.conf` for `hermes-gateway.service`
+- `resource-containment.conf` for `camofox-browser.service`
+- `resource-containment.conf` for `hermes-webui.service`
+
+Then reload the system and user systemd managers and restart only those three
+services. The original source units were never modified.
+
+## Related
+
+- [[systems/hermes-agent]]
+- [[systems/messaging-integrations]]
+- [[systems/browser-backend]]
+- [[runbooks/restart-hermes]]
+- [[systems/tool-search]]