- freshness: 36 stale / 93 fresh / 3 no-date -> 0 stale / 129 fresh / 3 no-date of 132 (same-day report regenerated after the corrections, per pattern 13) - skills: 299 skills / 67 categories (count unchanged; generator arithmetic re-verified by hand against the live tree: 300 SKILL.md files - 1 nested duplicate = 299) - runbooks/test-openrouter: live fallback chain rewritten (primary back to smart-route/custom:omni-101; mimo-v2.5, stepfun, qwen3.5/ollama-cloud out; guacano + kilo-reflection-beam-free in); ollama.com dropped from the URL test list; guacano/litellm noted as internal + key-gated - comparisons/hermes-vs-nanobot: the 2026-10-20 'CT333 is zeroclaw in Proxmox naming' claim was itself wrong - pct list and hostname both return clawtest - incidents/2026-10-22-nous: refresh-token failure gone from logs; new signature 404 insufficient_credits_for_paid_model from auxiliary.pulse_memory (ninth slot, back on nous, paid model) - open item revised - incidents/2026-08-29-gmail: email crons now ok but root cause unfixed 74 days (himalaya message read still in fetch_newsletters.py; goimapnotify back up PID 441; monitor still paused) - 30 further pages re-verified live and de-flagged with dated probe notes (ct207 ingress, CT205 guanaco v0.8.15, CT270 nextcloud 33.0.2.2, headless stack, hermes-ui both hosts, systemd units + cgroup limits, terminal/tool_search/mnemosyne/custom_providers config blocks, ops profile still deleted, hosts/pve guest lists) - log.md addendum + index.md date bump; no page deleted, no secrets written
5.9 KiB
title, type, status, created, updated, review_after, verified_on, version_applies_to, confidence, tags, sources
| title | type | status | created | updated | review_after | verified_on | version_applies_to | confidence | tags | sources | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Hermes Gateway Resource Resilience | runbook | active | 2026-07-25 | 2026-11-11 | 2026-11-19 | 2026-11-11 | Proxmox CT460 | high |
|
|
Hermes Gateway Resource Resilience
Purpose
Keep Telegram and the Hermes gateway available when the browser or Web UI causes severe memory pressure inside CT460.
The observed failure was real: the CT460 memory cgroup OOM-killed a large Web
UI process and a Chrome child. Because the gateway previously used
OOMPolicy=stop, the Chrome child death caused systemd to stop the entire
gateway until Restart=always brought it back.
Active controls
Gateway
File:
/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf
[Service]
OOMPolicy=continue
MemoryLow=512M
CPUWeight=1000
The gateway has no MemoryHigh or MemoryMax. It receives reclaim protection
and maximum relative CPU weight, while OOMPolicy=continue prevents an
OOM-killed browser child from stopping the whole unit. If the main gateway
process exits, its existing Restart=always policy still restarts it.
Camofox
File:
/home/hermes/.config/systemd/user/camofox-browser.service.d/resource-containment.conf
[Service]
MemoryHigh=2G
MemoryMax=3G
MemorySwapMax=1G
OOMPolicy=kill
⚠ Limits have been RAISED since 2026-07-25 (verified 2026-10-20). The Camofox drop-in now reads
MemoryHigh=2G,MemoryMax=3G,MemorySwapMax=1G— the page previously recorded1G/1536M/768M. Effective values confirmed viasystemctl --user show camofox-browser.service:MemoryHigh=2147483648(2 GiB),MemoryMax=3221225472(3 GiB),MemorySwapMax=1073741824(1 GiB),OOMPolicy=kill. TheOOMPolicy=killintent is unchanged; only the ceilings moved. The Camofox service is running (camofox-browser.service, active/running) — it is the browser workload the containment is meant to protect, and it is the unit that grew.
Hermes Web UI
File:
/etc/systemd/system/hermes-webui.service.d/resource-containment.conf
[Service]
MemoryHigh=1G
MemoryMax=1536M
MemorySwapMax=1G
OOMPolicy=kill
Camofox and Web UI are allowed to restart under extreme growth instead of consuming most of the container and taking Telegram down with them.
Re-verified 2026-10-20 — the gateway and Web UI controls are still exactly as documented.
- Gateway drop-in
/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.confstill readsOOMPolicy=continue,MemoryLow=512M,CPUWeight=1000(noMemoryHigh/MemoryMax). Effective:OOMPolicy=continue,CPUWeight=1000,MemoryLow=536870912,MemoryHigh=infinity,MemoryMax=infinity— matches the "Expected gateway values" note below.- Web UI:
OOMPolicy=kill,MemoryHigh=1073741824(1 GiB),MemoryMax=1610612736(1.5 GiB),MemorySwapMax=1073741824(1 GiB) — matches the drop-in shown above, unchanged.- Camofox: see the raised limits noted above.
hermes-gateway.serviceandcamofox-browser.serviceare both active (running).Note: the Web UI drop-in lives at
/etc/systemd/system/hermes-webui.service.d/which thehermesuser cannot read (permission denied) — its effective values are still verifiable viasystemctl show, which is what the health-check section below uses.
All effective values re-confirmed live 2026-11-11 via
systemctl show. Gateway (hermes-gateway.service, active):OOMPolicy=continue,MemoryLow=536870912,MemoryHigh=infinity,MemoryMax=infinity,CPUWeight=1000,Restart=always; the drop-in file still readsOOMPolicy=continue/MemoryLow=512M/CPUWeight=1000. Camofox (camofox-browser.service, active):OOMPolicy=kill,MemoryHigh=2147483648,MemoryMax=3221225472,MemorySwapMax=1073741824— the raised 2G/3G/1G ceilings hold. Web UI (hermes-webui.service, active):OOMPolicy=kill,MemoryHigh=1073741824,MemoryMax=1610612736,MemorySwapMax=1073741824. Unchanged since 2026-10-20.
I/O priority limitation
Do not add IOWeight to the gateway drop-in unless Proxmox cgroup delegation
is changed and verified. CT460 exposes the I/O controller at its top cgroup,
but it is not delegated into the container's systemd service tree. Systemd
accepts the property, but no live io.weight control is created.
Health check
systemctl is-active hermes-webui.service
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
systemctl --user is-active camofox-browser.service hermes-gateway.service
Check effective limits:
systemctl show hermes-webui.service \
-p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
systemctl --user show hermes-gateway.service \
-p OOMPolicy -p MemoryLow -p MemoryHigh -p MemoryMax -p CPUWeight
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
systemctl --user show camofox-browser.service \
-p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
Expected gateway values include OOMPolicy=continue,
MemoryLow=536870912, MemoryHigh=infinity, MemoryMax=infinity, and
CPUWeight=1000. Confirmed live on 2026-10-20 — see the note under "Active controls".
Backups and rollback
Original source units are backed up at:
/home/hermes/.hermes/backups/resource-hardening-20260725/
To roll back, remove only these three drop-ins:
resource-resilience.confforhermes-gateway.serviceresource-containment.confforcamofox-browser.serviceresource-containment.confforhermes-webui.service
Then reload the system and user systemd managers and restart only those three services. The original source units were never modified.