Files
agent-estate-wiki/runbooks/gateway-resource-resilience.md
T
Hermes 341dda95dd stocktake 2026-11-11: 36 stale pages re-verified (0 stale); fallback chain rewritten 4/6 slots; CT333 'clawtest' naming claim corrected; nous 404 credits signature replaces refresh-token 401; estate shrank (VM 310 + CT601 gone)
- freshness: 36 stale / 93 fresh / 3 no-date -> 0 stale / 129 fresh / 3 no-date of 132 (same-day report regenerated after the corrections, per pattern 13)
- skills: 299 skills / 67 categories (count unchanged; generator arithmetic re-verified by hand against the live tree: 300 SKILL.md files - 1 nested duplicate = 299)
- runbooks/test-openrouter: live fallback chain rewritten (primary back to smart-route/custom:omni-101; mimo-v2.5, stepfun, qwen3.5/ollama-cloud out; guacano + kilo-reflection-beam-free in); ollama.com dropped from the URL test list; guacano/litellm noted as internal + key-gated
- comparisons/hermes-vs-nanobot: the 2026-10-20 'CT333 is zeroclaw in Proxmox naming' claim was itself wrong - pct list and hostname both return clawtest
- incidents/2026-10-22-nous: refresh-token failure gone from logs; new signature 404 insufficient_credits_for_paid_model from auxiliary.pulse_memory (ninth slot, back on nous, paid model) - open item revised
- incidents/2026-08-29-gmail: email crons now ok but root cause unfixed 74 days (himalaya message read still in fetch_newsletters.py; goimapnotify back up PID 441; monitor still paused)
- 30 further pages re-verified live and de-flagged with dated probe notes (ct207 ingress, CT205 guanaco v0.8.15, CT270 nextcloud 33.0.2.2, headless stack, hermes-ui both hosts, systemd units + cgroup limits, terminal/tool_search/mnemosyne/custom_providers config blocks, ops profile still deleted, hosts/pve guest lists)
- log.md addendum + index.md date bump; no page deleted, no secrets written
2026-11-11 03:52:48 +08:00

5.9 KiB

title, type, status, created, updated, review_after, verified_on, version_applies_to, confidence, tags, sources
title type status created updated review_after verified_on version_applies_to confidence tags sources
Hermes Gateway Resource Resilience runbook active 2026-07-25 2026-11-11 2026-11-19 2026-11-11 Proxmox CT460 high
hermes
infrastructure
runbook
live systemd and cgroup v2 state on CT460

Hermes Gateway Resource Resilience

Purpose

Keep Telegram and the Hermes gateway available when the browser or Web UI causes severe memory pressure inside CT460.

The observed failure was real: the CT460 memory cgroup OOM-killed a large Web UI process and a Chrome child. Because the gateway previously used OOMPolicy=stop, the Chrome child death caused systemd to stop the entire gateway until Restart=always brought it back.

Active controls

Gateway

File: /home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf

[Service]
OOMPolicy=continue
MemoryLow=512M
CPUWeight=1000

The gateway has no MemoryHigh or MemoryMax. It receives reclaim protection and maximum relative CPU weight, while OOMPolicy=continue prevents an OOM-killed browser child from stopping the whole unit. If the main gateway process exits, its existing Restart=always policy still restarts it.

Camofox

File: /home/hermes/.config/systemd/user/camofox-browser.service.d/resource-containment.conf

[Service]
MemoryHigh=2G
MemoryMax=3G
MemorySwapMax=1G
OOMPolicy=kill

⚠ Limits have been RAISED since 2026-07-25 (verified 2026-10-20). The Camofox drop-in now reads MemoryHigh=2G, MemoryMax=3G, MemorySwapMax=1G — the page previously recorded 1G / 1536M / 768M. Effective values confirmed via systemctl --user show camofox-browser.service: MemoryHigh=2147483648 (2 GiB), MemoryMax=3221225472 (3 GiB), MemorySwapMax=1073741824 (1 GiB), OOMPolicy=kill. The OOMPolicy=kill intent is unchanged; only the ceilings moved. The Camofox service is running (camofox-browser.service, active/running) — it is the browser workload the containment is meant to protect, and it is the unit that grew.

Hermes Web UI

File: /etc/systemd/system/hermes-webui.service.d/resource-containment.conf

[Service]
MemoryHigh=1G
MemoryMax=1536M
MemorySwapMax=1G
OOMPolicy=kill

Camofox and Web UI are allowed to restart under extreme growth instead of consuming most of the container and taking Telegram down with them.

Re-verified 2026-10-20 — the gateway and Web UI controls are still exactly as documented.

  • Gateway drop-in /home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf still reads OOMPolicy=continue, MemoryLow=512M, CPUWeight=1000 (no MemoryHigh/MemoryMax). Effective: OOMPolicy=continue, CPUWeight=1000, MemoryLow=536870912, MemoryHigh=infinity, MemoryMax=infinity — matches the "Expected gateway values" note below.
  • Web UI: OOMPolicy=kill, MemoryHigh=1073741824 (1 GiB), MemoryMax=1610612736 (1.5 GiB), MemorySwapMax=1073741824 (1 GiB) — matches the drop-in shown above, unchanged.
  • Camofox: see the raised limits noted above.
  • hermes-gateway.service and camofox-browser.service are both active (running).

Note: the Web UI drop-in lives at /etc/systemd/system/hermes-webui.service.d/ which the hermes user cannot read (permission denied) — its effective values are still verifiable via systemctl show, which is what the health-check section below uses.

All effective values re-confirmed live 2026-11-11 via systemctl show. Gateway (hermes-gateway.service, active): OOMPolicy=continue, MemoryLow=536870912, MemoryHigh=infinity, MemoryMax=infinity, CPUWeight=1000, Restart=always; the drop-in file still reads OOMPolicy=continue / MemoryLow=512M / CPUWeight=1000. Camofox (camofox-browser.service, active): OOMPolicy=kill, MemoryHigh=2147483648, MemoryMax=3221225472, MemorySwapMax=1073741824 — the raised 2G/3G/1G ceilings hold. Web UI (hermes-webui.service, active): OOMPolicy=kill, MemoryHigh=1073741824, MemoryMax=1610612736, MemorySwapMax=1073741824. Unchanged since 2026-10-20.

I/O priority limitation

Do not add IOWeight to the gateway drop-in unless Proxmox cgroup delegation is changed and verified. CT460 exposes the I/O controller at its top cgroup, but it is not delegated into the container's systemd service tree. Systemd accepts the property, but no live io.weight control is created.

Health check

systemctl is-active hermes-webui.service
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
  systemctl --user is-active camofox-browser.service hermes-gateway.service

Check effective limits:

systemctl show hermes-webui.service \
  -p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
  systemctl --user show hermes-gateway.service \
  -p OOMPolicy -p MemoryLow -p MemoryHigh -p MemoryMax -p CPUWeight
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
  systemctl --user show camofox-browser.service \
  -p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax

Expected gateway values include OOMPolicy=continue, MemoryLow=536870912, MemoryHigh=infinity, MemoryMax=infinity, and CPUWeight=1000. Confirmed live on 2026-10-20 — see the note under "Active controls".

Backups and rollback

Original source units are backed up at:

/home/hermes/.hermes/backups/resource-hardening-20260725/

To roll back, remove only these three drop-ins:

  • resource-resilience.conf for hermes-gateway.service
  • resource-containment.conf for camofox-browser.service
  • resource-containment.conf for hermes-webui.service

Then reload the system and user systemd managers and restart only those three services. The original source units were never modified.