wiki: publish pending edits (2026-10-20)

This commit is contained in:
Tony0410
2026-10-20 18:38:11 +08:00
parent 667ab7f41d
commit fbd4f297eb
46 changed files with 1012 additions and 197 deletions
+12 -3
View File
@@ -3,9 +3,9 @@ title: Backup Wiki
type: runbook
status: active
created: 2026-07-22
updated: 2026-07-22
verified_on: 2026-07-22
last_tested: 2026-07-22
updated: 2026-10-20
verified_on: 2026-10-20
last_tested: 2026-10-20
confidence: high
tags: [runbook, wiki, backup]
sources: []
@@ -26,6 +26,15 @@ Create a durable backup of the wiki so recoverable copies exist before mutating
- Git is available
- Backup destination is writable
> **Two trees, not one (verified 2026-10-20).** This runbook still backs up `/home/hermes/wiki`
> only — the **Obsidian vault mirror**, which is a **local-only** git repo with **no remote**
> (`git remote -v` is empty). The **authoring tree** that is actually pushed to Gitea
> (`Anthony/agent-estate-wiki`) lives at `/home/hermes/workspace/agent-estate-wiki` and is a
> **separate repo**. A tarball of `/home/hermes/wiki` does **not** protect the authoring tree.
> To protect both, run step 3 against each path, or back up the authoring tree and rely on Gitea.
>
> Also note `/home/hermes/wiki-backups/` does not yet exist — this runbook has never been run.
## Procedure
### 1. Sanity-check repo state
+12 -3
View File
@@ -3,9 +3,9 @@ title: Diagnose Failed Docker Service
type: runbook
status: active
created: 2026-07-22
updated: 2026-07-22
verified_on: 2026-07-22
last_tested: 2026-07-22
updated: 2026-10-20
verified_on: 2026-10-20
last_tested: 2026-10-20
confidence: medium
tags: [runbook, docker, homelab]
sources: []
@@ -26,6 +26,15 @@ Diagnose container or compose issues on hosts where Docker is present.
- Docker daemon running
- Compose file path known for the affected project
> **Important scoping note, verified 2026-10-20: the `hermes` user is NOT in the `docker` group on
> CT460.** `systemctl is-active docker` → `active`, but `docker ps` as `hermes` returns
> **`permission denied while trying to connect to the docker API at unix:///var/run/docker.sock`**.
> Most Docker services in this estate run on **other hosts** (runtipi CT201, cloudreve CT209,
> LiteLLM CT500 on m93p, etc.) where the invoking account does have socket access.
> So: on CT460, either use `sudo docker …` or SSH to the host that actually owns the container.
> Note also that `systems/homepage-dashboard.md` now records the CT201 stack as **degraded** —
> the main dashboard container is gone — which is exactly the symptom this runbook would be used for.
## Procedure
### 1. Check Docker daemon state
+31 -7
View File
@@ -3,9 +3,9 @@ title: Hermes Gateway Resource Resilience
type: runbook
status: active
created: 2026-07-25
updated: 2026-07-25
review_after: 2026-10-23
verified_on: 2026-07-25
updated: 2026-10-20
review_after: 2026-11-19
verified_on: 2026-10-20
version_applies_to: Proxmox CT460
confidence: high
tags: [hermes, infrastructure, runbook]
@@ -50,12 +50,22 @@ File:
```ini
[Service]
MemoryHigh=1G
MemoryMax=1536M
MemorySwapMax=768M
MemoryHigh=2G
MemoryMax=3G
MemorySwapMax=1G
OOMPolicy=kill
```
> **⚠ Limits have been RAISED since 2026-07-25 (verified 2026-10-20).** The Camofox drop-in now
> reads `MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G` — the page previously recorded
> `1G` / `1536M` / `768M`. Effective values confirmed via
> `systemctl --user show camofox-browser.service`:
> `MemoryHigh=2147483648` (2 GiB), `MemoryMax=3221225472` (3 GiB),
> `MemorySwapMax=1073741824` (1 GiB), `OOMPolicy=kill`.
> The `OOMPolicy=kill` intent is unchanged; only the ceilings moved. The Camofox service **is**
> running (`camofox-browser.service`, active/running) — it is the browser workload the containment
> is meant to protect, and it is the unit that grew.
### Hermes Web UI
File:
@@ -72,6 +82,20 @@ OOMPolicy=kill
Camofox and Web UI are allowed to restart under extreme growth instead of
consuming most of the container and taking Telegram down with them.
> **Re-verified 2026-10-20 — the gateway and Web UI controls are still exactly as documented.**
> - Gateway drop-in `/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf`
> still reads `OOMPolicy=continue`, `MemoryLow=512M`, `CPUWeight=1000` (no `MemoryHigh`/`MemoryMax`).
> Effective: `OOMPolicy=continue`, `CPUWeight=1000`, `MemoryLow=536870912`,
> `MemoryHigh=infinity`, `MemoryMax=infinity` — **matches the "Expected gateway values" note below.**
> - Web UI: `OOMPolicy=kill`, `MemoryHigh=1073741824` (1 GiB), `MemoryMax=1610612736` (1.5 GiB),
> `MemorySwapMax=1073741824` (1 GiB) — **matches the drop-in shown above, unchanged.**
> - Camofox: see the raised limits noted above.
> - `hermes-gateway.service` and `camofox-browser.service` are both **active (running)**.
>
> **Note:** the Web UI drop-in lives at `/etc/systemd/system/hermes-webui.service.d/` which the
> `hermes` user cannot read (permission denied) — its *effective* values are still verifiable via
> `systemctl show`, which is what the health-check section below uses.
## I/O priority limitation
Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation
@@ -102,7 +126,7 @@ runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
Expected gateway values include `OOMPolicy=continue`,
`MemoryLow=536870912`, `MemoryHigh=infinity`, `MemoryMax=infinity`, and
`CPUWeight=1000`.
`CPUWeight=1000`. **Confirmed live on 2026-10-20** — see the note under "Active controls".
## Backups and rollback
+24 -2
View File
@@ -3,8 +3,8 @@ title: Multiplexer Setup (Multi-Profile Gateway)
type: runbook
status: active
created: 2026-07-28
updated: 2026-07-28
verified_on: 2026-07-28
updated: 2026-10-20
verified_on: 2026-10-20
confidence: high
tags: [hermes, multiplexer, telegram, profiles, gateway]
---
@@ -67,6 +67,28 @@ platforms:
enabled: false # ← DISABLED — multiplexer handles this
```
## ⚠ The `ops` profile was DELETED on 2026-10-11 (verified 2026-10-20)
`/home/hermes/.hermes/profiles/` now contains **only** `.deleted/ops` (an 8-byte marker file whose
entire content is the word `deleted`, dated 2026-10-11). **`~/.hermes/profiles/ops/` no longer
exists** — there is no `config.yaml`, no `SOUL.md`, no profile directory.
**But the default profile still routes to it.** `~/.hermes/config.yaml` still carries both
`profile_routes` entries (`ops-group` → `-1003914987043`, `ops-group-2` → `-1003932503629`,
both `profile: ops`), and `multiplex_profiles: true` is still set. The ops SOUL (Tusk, the African
elephant) and its `allowed_chats` list are gone with the directory.
**Consequence:** messages arriving in the two ops chats have a route pointing at a profile that no
longer exists. Treat the "Ops profile" blocks below as **historical** — the second profile's
config can no longer be read at `~/.hermes/profiles/ops/config.yaml` because that path is gone.
**Two things need a decision (not actioned here):**
1. Should `profile_routes` be removed, or should the `ops` profile be recreated?
2. Is the `multiplex_profiles: true` + two-route configuration still needed at all?
Until that is settled, treat the ops chat IDs as **possibly unroutable** and verify the gateway
actually resolves them before relying on this runbook.
## The Silent-Drop Pitfall
The `allowed_chats` gate runs on the **default profile's adapter** BEFORE `profile_routes` stamps `source.profile`. If a chat is only authorized in the secondary profile's config but not the default's, messages from that chat are silently dropped.
+15 -1
View File
@@ -3,7 +3,8 @@ title: Recover a Docker service
type: runbook
status: active
created: 2026-07-22
updated: 2026-07-22
updated: 2026-10-20
verified_on: 2026-10-20
tags: [runbook, docker, recovery]
sources: []
confidence: high
@@ -18,3 +19,16 @@ confidence: high
Container shows healthy after recovery probe reboots.
## Last successfully used
2026-07-22
> **Verified 2026-10-20 (procedure, not a live recovery).** Before running any `docker` command
> from CT460, note that the **`hermes` user has no access to `/var/run/docker.sock`** —
> `docker ps` returns `permission denied while trying to connect to the docker API`. Use `sudo`,
> or SSH to the host that owns the container. The containers in this estate mostly live on
> **CT201 (runtipi)**, **CT209 (cloudreve)** and **m93p CT500 (LiteLLM)** — not on CT460.
>
> **Two live situations this runbook would have been used for in the last week:**
> - The **CT201 homepage dashboard** is degraded — main container removed, `asset_proxy.py` on 8085
> proxying to a dead 8082, and the Tailscale `/` route returning 502. See
> [[systems/homepage-dashboard]].
> - The **Steel browser** service is unreachable (`steel.martinwa.org` → connection failure).
> See [[decisions/rollback-steel-to-camofox]].
+15 -6
View File
@@ -3,9 +3,9 @@ title: Restart Browser Helper
type: runbook
status: active
created: 2026-07-22
updated: 2026-07-22
verified_on: 2026-07-22
last_tested: 2026-07-22
updated: 2026-10-20
verified_on: 2026-10-20
last_tested: 2026-10-20
confidence: high
tags: [runbook, browser, cua-driver, camofox]
sources: []
@@ -22,8 +22,12 @@ Restart the computer-use browser stack that Hermes uses for web automation, scre
- VNC/noVNC is unreachable on `5901` / `6081`
## Prerequisites
- Display helper already running: `Xvfb :99`, x11vnc on `127.0.0.1:5901`, noVNC at `127.0.0.1:6081`
- `cua-driver` binary installed at `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver`
- Display helper already running: `Xvfb :99`, x11vnc on `0.0.0.0:5901`, noVNC at `127.0.0.1:6081`
- `cua-driver` binary installed — **the release directory has moved on.** The path this runbook
used to hardcode (`.../releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver`) no longer exists;
`/home/hermes/.cua-driver/packages/releases/` now holds `0.19.3`, `0.22.2`, `0.23.2`, `0.24.0`
and **`0.28.2`**, and `/home/hermes/.local/bin/cua-driver` reports **cua-driver 0.28.2**.
Prefer the stable `/home/hermes/.local/bin/cua-driver` path over a versioned release path.
- Browser VM/desktop helper is expected but not mandatory
## Procedure
@@ -72,7 +76,12 @@ pkill -f "fluxbox"
- The browser backend path can work without browser automation; this stack is optional depending on task
## Last tested
2026-07-22
2026-07-22 — procedure re-checked against live processes 2026-10-20. `Xvfb :99`, `x11vnc -display :99
-forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw`, `fluxbox` and both
websockify instances (`6080→5900`, `6081→5901`) are running, and the `vnc-watcher.sh` helper
(`/home/hermes/services/camofox-browser/plugins/vnc/vnc-watcher.sh`) is up. Note **5900 has no
listener** — the 6080 pair points at a dead port (Camofox's front door); the live display chain is
`:99 → 5901 → 6081`. See [[systems/headless-server]].
## Related
- [[browser-backend]]
+25 -7
View File
@@ -3,9 +3,9 @@ title: Restart Hermes
type: runbook
status: active
created: 2026-07-22
updated: 2026-07-22
verified_on: 2026-07-22
last_tested: 2026-07-22
updated: 2026-10-20
verified_on: 2026-10-20
last_tested: 2026-10-20
confidence: high
tags: [runbook, hermes, restart]
sources: [raw/configs/hermes-config-sanitized.txt]
@@ -32,17 +32,35 @@ Restart the Hermes agent, gateway, dashboard, and browser helper processes clean
```bash
ps aux | grep -E "hermes_cli.main (gateway|dashboard) run|hermes-webui/server.py|cua-driver serve|x11vnc.*5901" | grep -v grep
```
Expected active pieces on this install:
- gateway: `/home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run`
- dashboard: `/home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main dashboard ...`
- webui: `/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py`
Expected active pieces on this install (live `ps` 2026-10-20):
- gateway: `hermes gateway run` → `/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 … gateway run`
(unit `hermes-gateway.service`)
- dashboard: `hermes dashboard --host 0.0.0.0 --port 9119 --no-open` (unit `hermes-dashboard.service`)
- webui: `/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py` (port 8787)
- camofox/cua-driver: `/home/hermes/.local/bin/cua-driver serve`
- x11vnc: `-display :99 ... 5901`
> **The `venv/` path in this runbook is wrong for the gateway and dashboard (verified 2026-10-20).**
> They no longer run from `~/.hermes/hermes-agent/venv/`; they run from the **uv-managed runtime**
> `~/.hermes/tools/python-3.14.7+20260901-linux-x64/`, launched through
> `hermes_cli/main.py` with a `runpy` bootstrap. The `hermes` launcher at
> `/home/hermes/.local/bin/hermes` is a 73-byte shim. **Prefer the `hermes` CLI and the systemd
> user units over hand-written `python -m hermes_cli.main …` commands** — the units already carry
> the correct interpreter path and the `_hermes_closure_repair()` bootstrap that protects against a
> half-finished `hermes update`.
>
> **Live user units (2026-10-20):** `hermes-gateway.service`, `hermes-dashboard.service`,
> `hermes-ui.service`, `hermes-ui-prod.service`, `hermes-wiki-serve.service` — all
> `active (running)`.
### 2. Restart the gateway
```bash
hermes gateway restart
```
Equivalent and preferred (survives a broken CLI shim):
```bash
systemctl --user restart hermes-gateway.service
```
### 3. Restart the dashboard if needed
```bash
+21 -2
View File
@@ -3,7 +3,8 @@ title: Restore OpenRouter connectivity
type: runbook
status: active
created: 2026-07-22
updated: 2026-07-22
updated: 2026-10-20
verified_on: 2026-10-20
tags: [runbook, openrouter, recovery]
sources: []
confidence: high
@@ -19,9 +20,27 @@ confidence: high
3. Restart the Hermes process with the GoOp tool.
4. Send a small test completion request.
5. Confirm tool calling works.
> **⚠ This runbook is out of step with the current install (flagged 2026-10-20).**
> Steps 2 and 3 reference **WARP** and a **"GoOp tool"** — neither exists on this box. There is no
> WARP client configured and no `GoOp` binary; `hermes` has no such subcommand. The steps are
> carried over from a different install and **were not executed during the 2026-10-20 review**.
>
> **The procedure that actually applies here:**
> 1. Confirm OpenRouter is not a standalone provider in the first place — `custom_providers` in
> `~/.hermes/config.yaml` holds only `litellm`, `guacano`, `Omniroute:20128`, `command-code` and
> `omni-101`. See [[systems/openrouter]].
> 2. If the failure is a *fallback* failure, use [[runbooks/test-openrouter]] and check the six
> entries of the live chain.
> 3. Restart with `systemctl --user restart hermes-gateway.service` (not "the GoOp tool").
> 4. Send a small test completion and confirm tool calling works.
>
> **Last successfully used: 2026-06-22** — which predates every configuration referenced on this
> page. Treat the page as unverified until someone runs it end to end.
## Verification
A test prompt returns successfully through OpenRouter.
## Escalation
If the failure persists, check DNS, certificate expiry and provider status.
## Last successfully used
2026-06-22
2026-06-22 — **120 days ago.** This is the single oldest verification date still in the wiki
alongside the original 2026-07-22 page set.
+27 -11
View File
@@ -3,9 +3,9 @@ title: Test OpenRouter
type: runbook
status: active
created: 2026-07-22
updated: 2026-07-22
verified_on: 2026-07-22
last_tested: 2026-07-22
updated: 2026-10-20
verified_on: 2026-10-20
last_tested: 2026-10-20
confidence: high
tags: [runbook, openrouter, providers, fallback]
sources: [raw/configs/hermes-config-sanitized.txt]
@@ -22,7 +22,8 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar
- User wants to confirm OpenRouter is usable without relying on catalog names
## Prerequisites
- Hermes CLI available: `/home/hermes/.local/bin/hermes`
- Hermes CLI available: `hermes` on PATH (`/home/hermes/.local/bin/hermes` is a shim to the
uv-managed runtime — see the note under **Last tested**)
- Internet egress for opencode.ai, inference-api.nousresearch.com, ollama.com, kilo.ai, litellm:4000
## Procedure
@@ -31,12 +32,22 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar
```bash
hermes fallback list
```
Expected active fallbacks on this install:
1. `deepseek-v4-flash-free` via opencode-zen
2. `stepfun/step-3.7-flash:free` via nous
3. `qwen3.5:397b` via ollama-cloud
4. `poolside/laguna-m.1:free` via kilocode
5. `or-inclusionai_ling-2_6-1t` via custom/litellm
Expected active fallbacks on this install (live `hermes fallback list`, 2026-10-20 — **6** entries):
1. `mimo-v2.5` via opencode-go
2. `deepseek-v4-flash-free` via opencode-zen — `https://opencode.ai/zen/v1`
3. `stepfun/step-3.7-flash:free` via nous — `https://inference-api.nousresearch.com/v1`
4. `qwen3.5:397b` via ollama-cloud — `https://ollama.com/v1`
5. `poolside/laguna-m.1:free` via kilocode — `https://api.kilo.ai/api/gateway`
6. `or-inclusionai_ling-2_6-1t` via custom — `http://litellm:4000/v1`
**Primary:** `meituan/longcat-2.5-preview:free` (via nous).
> **Changes since 2026-07-22:** `mimo-v2.5` via **opencode-go** has been **added at position 1** of the
> chain, and the primary changed to `meituan/longcat-2.5-preview:free` (was `freellmapi-auto`).
> The other five entries and their order are unchanged.
> `hermes fallback list` also prints a standing notice on every invocation:
> *"Claude Subscription DirectSDK (Experimental): Claude Code is not installed"* — expected on this
> box, since `claude` is not on PATH. It is a notice, not a failure of the fallback chain.
### 2. Test OpenRouter-style direct access if present
```bash
@@ -77,7 +88,12 @@ Verify provider names and URLs in config match what the user wants to use.
- This runbook tests confirmed fallback paths, not assumed ones
## Last tested
2026-07-22
2026-07-22 — re-run 2026-10-20 against live `hermes fallback list` output (6 entries, primary
`meituan/longcat-2.5-preview:free`). Note the CLI reports Hermes Agent **v0.22.2 (2026.9.24),
local `d923a5eb487a`**, and the `/home/hermes/.local/bin/hermes` launcher is a 73-byte shim that
delegates to the uv-managed runtime at
`/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/` — *not* the
`~/.hermes/hermes-agent/venv/` path named in some older runbooks. Use the `hermes` on PATH.
## Related
- [[provider-health]]
+88 -13
View File
@@ -3,24 +3,99 @@ title: Troubleshoot Tool Loop
type: runbook
status: unresolved
created: 2026-07-22
updated: 2026-07-22
review_after: 2026-08-22
confidence: low
updated: 2026-10-20
review_after: 2026-11-19
verified_on: 2026-10-20
confidence: medium
tags: [runbook, hermes]
sources: []
---
> **Stub — structure only.** Headings exist but content has not been verified/written yet. Flagged during 2026-07-22 wiki audit; fill in with verified facts or mark deprecated if no longer relevant.
> **Filled in 2026-10-20** from live configuration, live process state and two recorded incidents.
> The "stub" note below is retained as history.
# Troubleshoot Tool Loop
## Symptoms
Agent loops on one tool or endpoint.
## Procedure
1. Inspect recent Hermes agent logs.
2. Identify the looping tool path.
3. Restart worker or flush stuck request.
4. Re-run limited test scenario.
## Verification
Agent handles the same scenario without repeating.
- The agent repeats the **same tool call** with identical arguments and makes no progress.
- The agent repeats **itself in the response stream** (no tool involved) — the model loops on its
own output and Hermes cuts the stream.
**These are two different faults with two different detectors.** Do not assume one covers the other.
## 1. Tool-call loop — detected by `tool_loop_guardrails`
### Symptoms
Agent loops on one tool or endpoint. Hermes emits a warning naming the tool.
### Configuration (verified live 2026-10-20, `~/.hermes/config.yaml`)
```yaml
tool_loop_guardrails:
warnings_enabled: true
hard_stop_enabled: false
warn_after:
exact_failure: 2
same_tool_failure: 3
idempotent_no_progress: 2
hard_stop_after:
exact_failure: 5
same_tool_failure: 8
idempotent_no_progress: 5
```
> **`hard_stop_enabled: false` is the key fact.** Hermes will *warn* about a stuck tool loop but
> will **not** force-stop itself. A genuinely stuck loop only gets a warning — a human (or the
> messaging integration) is the real backstop. See [[concepts/tool-calling]].
### Procedure
1. Inspect recent Hermes agent logs: `~/.hermes/logs/agent.log*` (grep for the tool name).
2. Identify the looping tool path. Common culprits in this estate: a browser tool against a dead
Camofox service, a terminal tool against a stale persistent shell, or a webhook that returns
200 with an empty body.
3. **Check the dependency, not just the agent.** On this estate the most frequent tool-loop cause
was an upstream service being down while the tool still returned a response. Verify with the
relevant runbook: [[runbooks/restart-browser]], [[runbooks/restart-hermes]],
[[runbooks/provider-health]].
4. If the loop is a stuck worker, restart it: `systemctl --user restart hermes-gateway.service`.
5. Re-run a limited test scenario and confirm the guard no longer fires.
### Verification
Agent handles the same scenario without repeating, and no `tool_loop_guardrails` warning appears
in `agent.log`.
## 2. Model-output (degenerate stream) loop — detected by the stream guard
### Symptoms
The run **delivers a canned non-answer** such as *"Your request was not processed. Send it again
if you still want me to carry it out."*, and then fails with
`RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding
the degenerate response.`
### This is the 2026-10-18 case — see
[[incidents/2026-10-18-degenerate-stream-canned-substitute]]
### Procedure
1. Read the run's `error` in `~/.hermes/cron/executions.db` — the phrase `repetition loop`
identifies this fault.
2. **Check `delivery_outcome` on the same row.** In the 2026-10-18 incident the row was
`delivery_outcome: delivered` — the canned non-answer reached Telegram *before* the run
failed. **A delivered message is not evidence of delivered content.**
3. Confirm the provider is healthy: the 2026-10-18 case rode the same `custom:omni-101` alias
that produced the 2026-10-14/15 connection-refused storm and the 2026-10-16/17 and 2026-10-19
resource-pressure 503 bursts. See [[runbooks/provider-health]] and
[[incidents/2026-10-16-omniroute-503-resource-pressure]].
4. Do **not** lower `tool_loop_guardrails` thresholds for this fault — they watch tool calls, and
the degenerate-stream guard is a separate mechanism.
### Verification
The next scheduled run of the job completes with `status: completed` and `error: null`, and the
delivered message contains real content.
## Last successfully used
2026-07-22
2026-07-22 — re-verified against live config and live incidents on 2026-10-20; the *procedure* has
not itself been executed end-to-end in a live loop, so `confidence` was raised only to `medium`.
## Related
- [[concepts/tool-calling]]
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
- [[runbooks/provider-health]]
- [[runbooks/restart-hermes]]