Files
agent-estate-wiki/runbooks/gateway-resource-resilience.md
T
Hermes 341dda95dd stocktake 2026-11-11: 36 stale pages re-verified (0 stale); fallback chain rewritten 4/6 slots; CT333 'clawtest' naming claim corrected; nous 404 credits signature replaces refresh-token 401; estate shrank (VM 310 + CT601 gone)
- freshness: 36 stale / 93 fresh / 3 no-date -> 0 stale / 129 fresh / 3 no-date of 132 (same-day report regenerated after the corrections, per pattern 13)
- skills: 299 skills / 67 categories (count unchanged; generator arithmetic re-verified by hand against the live tree: 300 SKILL.md files - 1 nested duplicate = 299)
- runbooks/test-openrouter: live fallback chain rewritten (primary back to smart-route/custom:omni-101; mimo-v2.5, stepfun, qwen3.5/ollama-cloud out; guacano + kilo-reflection-beam-free in); ollama.com dropped from the URL test list; guacano/litellm noted as internal + key-gated
- comparisons/hermes-vs-nanobot: the 2026-10-20 'CT333 is zeroclaw in Proxmox naming' claim was itself wrong - pct list and hostname both return clawtest
- incidents/2026-10-22-nous: refresh-token failure gone from logs; new signature 404 insufficient_credits_for_paid_model from auxiliary.pulse_memory (ninth slot, back on nous, paid model) - open item revised
- incidents/2026-08-29-gmail: email crons now ok but root cause unfixed 74 days (himalaya message read still in fetch_newsletters.py; goimapnotify back up PID 441; monitor still paused)
- 30 further pages re-verified live and de-flagged with dated probe notes (ct207 ingress, CT205 guanaco v0.8.15, CT270 nextcloud 33.0.2.2, headless stack, hermes-ui both hosts, systemd units + cgroup limits, terminal/tool_search/mnemosyne/custom_providers config blocks, ops profile still deleted, hosts/pve guest lists)
- log.md addendum + index.md date bump; no page deleted, no secrets written
2026-11-11 03:52:48 +08:00

162 lines
5.9 KiB
Markdown

---
title: Hermes Gateway Resource Resilience
type: runbook
status: active
created: 2026-07-25
updated: 2026-11-11
review_after: 2026-11-19
verified_on: 2026-11-11
version_applies_to: Proxmox CT460
confidence: high
tags: [hermes, infrastructure, runbook]
sources: [live systemd and cgroup v2 state on CT460]
---
# Hermes Gateway Resource Resilience
## Purpose
Keep Telegram and the Hermes gateway available when the browser or Web UI
causes severe memory pressure inside CT460.
The observed failure was real: the CT460 memory cgroup OOM-killed a large Web
UI process and a Chrome child. Because the gateway previously used
`OOMPolicy=stop`, the Chrome child death caused systemd to stop the entire
gateway until `Restart=always` brought it back.
## Active controls
### Gateway
File:
`/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf`
```ini
[Service]
OOMPolicy=continue
MemoryLow=512M
CPUWeight=1000
```
The gateway has no `MemoryHigh` or `MemoryMax`. It receives reclaim protection
and maximum relative CPU weight, while `OOMPolicy=continue` prevents an
OOM-killed browser child from stopping the whole unit. If the main gateway
process exits, its existing `Restart=always` policy still restarts it.
### Camofox
File:
`/home/hermes/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`
```ini
[Service]
MemoryHigh=2G
MemoryMax=3G
MemorySwapMax=1G
OOMPolicy=kill
```
> **⚠ Limits have been RAISED since 2026-07-25 (verified 2026-10-20).** The Camofox drop-in now
> reads `MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G` — the page previously recorded
> `1G` / `1536M` / `768M`. Effective values confirmed via
> `systemctl --user show camofox-browser.service`:
> `MemoryHigh=2147483648` (2 GiB), `MemoryMax=3221225472` (3 GiB),
> `MemorySwapMax=1073741824` (1 GiB), `OOMPolicy=kill`.
> The `OOMPolicy=kill` intent is unchanged; only the ceilings moved. The Camofox service **is**
> running (`camofox-browser.service`, active/running) — it is the browser workload the containment
> is meant to protect, and it is the unit that grew.
### Hermes Web UI
File:
`/etc/systemd/system/hermes-webui.service.d/resource-containment.conf`
```ini
[Service]
MemoryHigh=1G
MemoryMax=1536M
MemorySwapMax=1G
OOMPolicy=kill
```
Camofox and Web UI are allowed to restart under extreme growth instead of
consuming most of the container and taking Telegram down with them.
> **Re-verified 2026-10-20 — the gateway and Web UI controls are still exactly as documented.**
> - Gateway drop-in `/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf`
> still reads `OOMPolicy=continue`, `MemoryLow=512M`, `CPUWeight=1000` (no `MemoryHigh`/`MemoryMax`).
> Effective: `OOMPolicy=continue`, `CPUWeight=1000`, `MemoryLow=536870912`,
> `MemoryHigh=infinity`, `MemoryMax=infinity` — **matches the "Expected gateway values" note below.**
> - Web UI: `OOMPolicy=kill`, `MemoryHigh=1073741824` (1 GiB), `MemoryMax=1610612736` (1.5 GiB),
> `MemorySwapMax=1073741824` (1 GiB) — **matches the drop-in shown above, unchanged.**
> - Camofox: see the raised limits noted above.
> - `hermes-gateway.service` and `camofox-browser.service` are both **active (running)**.
>
> **Note:** the Web UI drop-in lives at `/etc/systemd/system/hermes-webui.service.d/` which the
> `hermes` user cannot read (permission denied) — its *effective* values are still verifiable via
> `systemctl show`, which is what the health-check section below uses.
> **All effective values re-confirmed live 2026-11-11 via `systemctl show`.**
> Gateway (`hermes-gateway.service`, active): `OOMPolicy=continue`, `MemoryLow=536870912`,
> `MemoryHigh=infinity`, `MemoryMax=infinity`, `CPUWeight=1000`, `Restart=always`; the drop-in file
> still reads `OOMPolicy=continue` / `MemoryLow=512M` / `CPUWeight=1000`. Camofox
> (`camofox-browser.service`, active): `OOMPolicy=kill`, `MemoryHigh=2147483648`,
> `MemoryMax=3221225472`, `MemorySwapMax=1073741824` — the raised 2G/3G/1G ceilings hold. Web UI
> (`hermes-webui.service`, active): `OOMPolicy=kill`, `MemoryHigh=1073741824`,
> `MemoryMax=1610612736`, `MemorySwapMax=1073741824`. Unchanged since 2026-10-20.
## I/O priority limitation
Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation
is changed and verified. CT460 exposes the I/O controller at its top cgroup,
but it is not delegated into the container's systemd service tree. Systemd
accepts the property, but no live `io.weight` control is created.
## Health check
```bash
systemctl is-active hermes-webui.service
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
systemctl --user is-active camofox-browser.service hermes-gateway.service
```
Check effective limits:
```bash
systemctl show hermes-webui.service \
-p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
systemctl --user show hermes-gateway.service \
-p OOMPolicy -p MemoryLow -p MemoryHigh -p MemoryMax -p CPUWeight
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
systemctl --user show camofox-browser.service \
-p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
```
Expected gateway values include `OOMPolicy=continue`,
`MemoryLow=536870912`, `MemoryHigh=infinity`, `MemoryMax=infinity`, and
`CPUWeight=1000`. **Confirmed live on 2026-10-20** — see the note under "Active controls".
## Backups and rollback
Original source units are backed up at:
`/home/hermes/.hermes/backups/resource-hardening-20260725/`
To roll back, remove only these three drop-ins:
- `resource-resilience.conf` for `hermes-gateway.service`
- `resource-containment.conf` for `camofox-browser.service`
- `resource-containment.conf` for `hermes-webui.service`
Then reload the system and user systemd managers and restart only those three
services. The original source units were never modified.
## Related
- [[systems/hermes-agent]]
- [[systems/messaging-integrations]]
- [[systems/browser-backend]]
- [[runbooks/restart-hermes]]
- [[systems/tool-search]]