wiki: publish pending edits (2026-10-20)
This commit is contained in:
+12
-3
@@ -3,9 +3,9 @@ title: Backup Wiki
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-07-22
|
||||
verified_on: 2026-07-22
|
||||
last_tested: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
last_tested: 2026-10-20
|
||||
confidence: high
|
||||
tags: [runbook, wiki, backup]
|
||||
sources: []
|
||||
@@ -26,6 +26,15 @@ Create a durable backup of the wiki so recoverable copies exist before mutating
|
||||
- Git is available
|
||||
- Backup destination is writable
|
||||
|
||||
> **Two trees, not one (verified 2026-10-20).** This runbook still backs up `/home/hermes/wiki`
|
||||
> only — the **Obsidian vault mirror**, which is a **local-only** git repo with **no remote**
|
||||
> (`git remote -v` is empty). The **authoring tree** that is actually pushed to Gitea
|
||||
> (`Anthony/agent-estate-wiki`) lives at `/home/hermes/workspace/agent-estate-wiki` and is a
|
||||
> **separate repo**. A tarball of `/home/hermes/wiki` does **not** protect the authoring tree.
|
||||
> To protect both, run step 3 against each path, or back up the authoring tree and rely on Gitea.
|
||||
>
|
||||
> Also note `/home/hermes/wiki-backups/` does not yet exist — this runbook has never been run.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Sanity-check repo state
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Diagnose Failed Docker Service
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-07-22
|
||||
verified_on: 2026-07-22
|
||||
last_tested: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
last_tested: 2026-10-20
|
||||
confidence: medium
|
||||
tags: [runbook, docker, homelab]
|
||||
sources: []
|
||||
@@ -26,6 +26,15 @@ Diagnose container or compose issues on hosts where Docker is present.
|
||||
- Docker daemon running
|
||||
- Compose file path known for the affected project
|
||||
|
||||
> **Important scoping note, verified 2026-10-20: the `hermes` user is NOT in the `docker` group on
|
||||
> CT460.** `systemctl is-active docker` → `active`, but `docker ps` as `hermes` returns
|
||||
> **`permission denied while trying to connect to the docker API at unix:///var/run/docker.sock`**.
|
||||
> Most Docker services in this estate run on **other hosts** (runtipi CT201, cloudreve CT209,
|
||||
> LiteLLM CT500 on m93p, etc.) where the invoking account does have socket access.
|
||||
> So: on CT460, either use `sudo docker …` or SSH to the host that actually owns the container.
|
||||
> Note also that `systems/homepage-dashboard.md` now records the CT201 stack as **degraded** —
|
||||
> the main dashboard container is gone — which is exactly the symptom this runbook would be used for.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Check Docker daemon state
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Hermes Gateway Resource Resilience
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-25
|
||||
updated: 2026-07-25
|
||||
review_after: 2026-10-23
|
||||
verified_on: 2026-07-25
|
||||
updated: 2026-10-20
|
||||
review_after: 2026-11-19
|
||||
verified_on: 2026-10-20
|
||||
version_applies_to: Proxmox CT460
|
||||
confidence: high
|
||||
tags: [hermes, infrastructure, runbook]
|
||||
@@ -50,12 +50,22 @@ File:
|
||||
|
||||
```ini
|
||||
[Service]
|
||||
MemoryHigh=1G
|
||||
MemoryMax=1536M
|
||||
MemorySwapMax=768M
|
||||
MemoryHigh=2G
|
||||
MemoryMax=3G
|
||||
MemorySwapMax=1G
|
||||
OOMPolicy=kill
|
||||
```
|
||||
|
||||
> **⚠ Limits have been RAISED since 2026-07-25 (verified 2026-10-20).** The Camofox drop-in now
|
||||
> reads `MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G` — the page previously recorded
|
||||
> `1G` / `1536M` / `768M`. Effective values confirmed via
|
||||
> `systemctl --user show camofox-browser.service`:
|
||||
> `MemoryHigh=2147483648` (2 GiB), `MemoryMax=3221225472` (3 GiB),
|
||||
> `MemorySwapMax=1073741824` (1 GiB), `OOMPolicy=kill`.
|
||||
> The `OOMPolicy=kill` intent is unchanged; only the ceilings moved. The Camofox service **is**
|
||||
> running (`camofox-browser.service`, active/running) — it is the browser workload the containment
|
||||
> is meant to protect, and it is the unit that grew.
|
||||
|
||||
### Hermes Web UI
|
||||
|
||||
File:
|
||||
@@ -72,6 +82,20 @@ OOMPolicy=kill
|
||||
Camofox and Web UI are allowed to restart under extreme growth instead of
|
||||
consuming most of the container and taking Telegram down with them.
|
||||
|
||||
> **Re-verified 2026-10-20 — the gateway and Web UI controls are still exactly as documented.**
|
||||
> - Gateway drop-in `/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf`
|
||||
> still reads `OOMPolicy=continue`, `MemoryLow=512M`, `CPUWeight=1000` (no `MemoryHigh`/`MemoryMax`).
|
||||
> Effective: `OOMPolicy=continue`, `CPUWeight=1000`, `MemoryLow=536870912`,
|
||||
> `MemoryHigh=infinity`, `MemoryMax=infinity` — **matches the "Expected gateway values" note below.**
|
||||
> - Web UI: `OOMPolicy=kill`, `MemoryHigh=1073741824` (1 GiB), `MemoryMax=1610612736` (1.5 GiB),
|
||||
> `MemorySwapMax=1073741824` (1 GiB) — **matches the drop-in shown above, unchanged.**
|
||||
> - Camofox: see the raised limits noted above.
|
||||
> - `hermes-gateway.service` and `camofox-browser.service` are both **active (running)**.
|
||||
>
|
||||
> **Note:** the Web UI drop-in lives at `/etc/systemd/system/hermes-webui.service.d/` which the
|
||||
> `hermes` user cannot read (permission denied) — its *effective* values are still verifiable via
|
||||
> `systemctl show`, which is what the health-check section below uses.
|
||||
|
||||
## I/O priority limitation
|
||||
|
||||
Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation
|
||||
@@ -102,7 +126,7 @@ runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
|
||||
|
||||
Expected gateway values include `OOMPolicy=continue`,
|
||||
`MemoryLow=536870912`, `MemoryHigh=infinity`, `MemoryMax=infinity`, and
|
||||
`CPUWeight=1000`.
|
||||
`CPUWeight=1000`. **Confirmed live on 2026-10-20** — see the note under "Active controls".
|
||||
|
||||
## Backups and rollback
|
||||
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Multiplexer Setup (Multi-Profile Gateway)
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-28
|
||||
updated: 2026-07-28
|
||||
verified_on: 2026-07-28
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
confidence: high
|
||||
tags: [hermes, multiplexer, telegram, profiles, gateway]
|
||||
---
|
||||
@@ -67,6 +67,28 @@ platforms:
|
||||
enabled: false # ← DISABLED — multiplexer handles this
|
||||
```
|
||||
|
||||
## ⚠ The `ops` profile was DELETED on 2026-10-11 (verified 2026-10-20)
|
||||
|
||||
`/home/hermes/.hermes/profiles/` now contains **only** `.deleted/ops` (an 8-byte marker file whose
|
||||
entire content is the word `deleted`, dated 2026-10-11). **`~/.hermes/profiles/ops/` no longer
|
||||
exists** — there is no `config.yaml`, no `SOUL.md`, no profile directory.
|
||||
|
||||
**But the default profile still routes to it.** `~/.hermes/config.yaml` still carries both
|
||||
`profile_routes` entries (`ops-group` → `-1003914987043`, `ops-group-2` → `-1003932503629`,
|
||||
both `profile: ops`), and `multiplex_profiles: true` is still set. The ops SOUL (Tusk, the African
|
||||
elephant) and its `allowed_chats` list are gone with the directory.
|
||||
|
||||
**Consequence:** messages arriving in the two ops chats have a route pointing at a profile that no
|
||||
longer exists. Treat the "Ops profile" blocks below as **historical** — the second profile's
|
||||
config can no longer be read at `~/.hermes/profiles/ops/config.yaml` because that path is gone.
|
||||
|
||||
**Two things need a decision (not actioned here):**
|
||||
1. Should `profile_routes` be removed, or should the `ops` profile be recreated?
|
||||
2. Is the `multiplex_profiles: true` + two-route configuration still needed at all?
|
||||
|
||||
Until that is settled, treat the ops chat IDs as **possibly unroutable** and verify the gateway
|
||||
actually resolves them before relying on this runbook.
|
||||
|
||||
## The Silent-Drop Pitfall
|
||||
|
||||
The `allowed_chats` gate runs on the **default profile's adapter** BEFORE `profile_routes` stamps `source.profile`. If a chat is only authorized in the secondary profile's config but not the default's, messages from that chat are silently dropped.
|
||||
|
||||
@@ -3,7 +3,8 @@ title: Recover a Docker service
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
tags: [runbook, docker, recovery]
|
||||
sources: []
|
||||
confidence: high
|
||||
@@ -18,3 +19,16 @@ confidence: high
|
||||
Container shows healthy after recovery probe reboots.
|
||||
## Last successfully used
|
||||
2026-07-22
|
||||
|
||||
> **Verified 2026-10-20 (procedure, not a live recovery).** Before running any `docker` command
|
||||
> from CT460, note that the **`hermes` user has no access to `/var/run/docker.sock`** —
|
||||
> `docker ps` returns `permission denied while trying to connect to the docker API`. Use `sudo`,
|
||||
> or SSH to the host that owns the container. The containers in this estate mostly live on
|
||||
> **CT201 (runtipi)**, **CT209 (cloudreve)** and **m93p CT500 (LiteLLM)** — not on CT460.
|
||||
>
|
||||
> **Two live situations this runbook would have been used for in the last week:**
|
||||
> - The **CT201 homepage dashboard** is degraded — main container removed, `asset_proxy.py` on 8085
|
||||
> proxying to a dead 8082, and the Tailscale `/` route returning 502. See
|
||||
> [[systems/homepage-dashboard]].
|
||||
> - The **Steel browser** service is unreachable (`steel.martinwa.org` → connection failure).
|
||||
> See [[decisions/rollback-steel-to-camofox]].
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Restart Browser Helper
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-07-22
|
||||
verified_on: 2026-07-22
|
||||
last_tested: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
last_tested: 2026-10-20
|
||||
confidence: high
|
||||
tags: [runbook, browser, cua-driver, camofox]
|
||||
sources: []
|
||||
@@ -22,8 +22,12 @@ Restart the computer-use browser stack that Hermes uses for web automation, scre
|
||||
- VNC/noVNC is unreachable on `5901` / `6081`
|
||||
|
||||
## Prerequisites
|
||||
- Display helper already running: `Xvfb :99`, x11vnc on `127.0.0.1:5901`, noVNC at `127.0.0.1:6081`
|
||||
- `cua-driver` binary installed at `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver`
|
||||
- Display helper already running: `Xvfb :99`, x11vnc on `0.0.0.0:5901`, noVNC at `127.0.0.1:6081`
|
||||
- `cua-driver` binary installed — **the release directory has moved on.** The path this runbook
|
||||
used to hardcode (`.../releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver`) no longer exists;
|
||||
`/home/hermes/.cua-driver/packages/releases/` now holds `0.19.3`, `0.22.2`, `0.23.2`, `0.24.0`
|
||||
and **`0.28.2`**, and `/home/hermes/.local/bin/cua-driver` reports **cua-driver 0.28.2**.
|
||||
Prefer the stable `/home/hermes/.local/bin/cua-driver` path over a versioned release path.
|
||||
- Browser VM/desktop helper is expected but not mandatory
|
||||
|
||||
## Procedure
|
||||
@@ -72,7 +76,12 @@ pkill -f "fluxbox"
|
||||
- The browser backend path can work without browser automation; this stack is optional depending on task
|
||||
|
||||
## Last tested
|
||||
2026-07-22
|
||||
2026-07-22 — procedure re-checked against live processes 2026-10-20. `Xvfb :99`, `x11vnc -display :99
|
||||
-forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw`, `fluxbox` and both
|
||||
websockify instances (`6080→5900`, `6081→5901`) are running, and the `vnc-watcher.sh` helper
|
||||
(`/home/hermes/services/camofox-browser/plugins/vnc/vnc-watcher.sh`) is up. Note **5900 has no
|
||||
listener** — the 6080 pair points at a dead port (Camofox's front door); the live display chain is
|
||||
`:99 → 5901 → 6081`. See [[systems/headless-server]].
|
||||
|
||||
## Related
|
||||
- [[browser-backend]]
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Restart Hermes
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-07-22
|
||||
verified_on: 2026-07-22
|
||||
last_tested: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
last_tested: 2026-10-20
|
||||
confidence: high
|
||||
tags: [runbook, hermes, restart]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -32,17 +32,35 @@ Restart the Hermes agent, gateway, dashboard, and browser helper processes clean
|
||||
```bash
|
||||
ps aux | grep -E "hermes_cli.main (gateway|dashboard) run|hermes-webui/server.py|cua-driver serve|x11vnc.*5901" | grep -v grep
|
||||
```
|
||||
Expected active pieces on this install:
|
||||
- gateway: `/home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run`
|
||||
- dashboard: `/home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main dashboard ...`
|
||||
- webui: `/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py`
|
||||
Expected active pieces on this install (live `ps` 2026-10-20):
|
||||
- gateway: `hermes gateway run` → `/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 … gateway run`
|
||||
(unit `hermes-gateway.service`)
|
||||
- dashboard: `hermes dashboard --host 0.0.0.0 --port 9119 --no-open` (unit `hermes-dashboard.service`)
|
||||
- webui: `/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py` (port 8787)
|
||||
- camofox/cua-driver: `/home/hermes/.local/bin/cua-driver serve`
|
||||
- x11vnc: `-display :99 ... 5901`
|
||||
|
||||
> **The `venv/` path in this runbook is wrong for the gateway and dashboard (verified 2026-10-20).**
|
||||
> They no longer run from `~/.hermes/hermes-agent/venv/`; they run from the **uv-managed runtime**
|
||||
> `~/.hermes/tools/python-3.14.7+20260901-linux-x64/`, launched through
|
||||
> `hermes_cli/main.py` with a `runpy` bootstrap. The `hermes` launcher at
|
||||
> `/home/hermes/.local/bin/hermes` is a 73-byte shim. **Prefer the `hermes` CLI and the systemd
|
||||
> user units over hand-written `python -m hermes_cli.main …` commands** — the units already carry
|
||||
> the correct interpreter path and the `_hermes_closure_repair()` bootstrap that protects against a
|
||||
> half-finished `hermes update`.
|
||||
>
|
||||
> **Live user units (2026-10-20):** `hermes-gateway.service`, `hermes-dashboard.service`,
|
||||
> `hermes-ui.service`, `hermes-ui-prod.service`, `hermes-wiki-serve.service` — all
|
||||
> `active (running)`.
|
||||
|
||||
### 2. Restart the gateway
|
||||
```bash
|
||||
hermes gateway restart
|
||||
```
|
||||
Equivalent and preferred (survives a broken CLI shim):
|
||||
```bash
|
||||
systemctl --user restart hermes-gateway.service
|
||||
```
|
||||
|
||||
### 3. Restart the dashboard if needed
|
||||
```bash
|
||||
|
||||
@@ -3,7 +3,8 @@ title: Restore OpenRouter connectivity
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
tags: [runbook, openrouter, recovery]
|
||||
sources: []
|
||||
confidence: high
|
||||
@@ -19,9 +20,27 @@ confidence: high
|
||||
3. Restart the Hermes process with the GoOp tool.
|
||||
4. Send a small test completion request.
|
||||
5. Confirm tool calling works.
|
||||
|
||||
> **⚠ This runbook is out of step with the current install (flagged 2026-10-20).**
|
||||
> Steps 2 and 3 reference **WARP** and a **"GoOp tool"** — neither exists on this box. There is no
|
||||
> WARP client configured and no `GoOp` binary; `hermes` has no such subcommand. The steps are
|
||||
> carried over from a different install and **were not executed during the 2026-10-20 review**.
|
||||
>
|
||||
> **The procedure that actually applies here:**
|
||||
> 1. Confirm OpenRouter is not a standalone provider in the first place — `custom_providers` in
|
||||
> `~/.hermes/config.yaml` holds only `litellm`, `guacano`, `Omniroute:20128`, `command-code` and
|
||||
> `omni-101`. See [[systems/openrouter]].
|
||||
> 2. If the failure is a *fallback* failure, use [[runbooks/test-openrouter]] and check the six
|
||||
> entries of the live chain.
|
||||
> 3. Restart with `systemctl --user restart hermes-gateway.service` (not "the GoOp tool").
|
||||
> 4. Send a small test completion and confirm tool calling works.
|
||||
>
|
||||
> **Last successfully used: 2026-06-22** — which predates every configuration referenced on this
|
||||
> page. Treat the page as unverified until someone runs it end to end.
|
||||
## Verification
|
||||
A test prompt returns successfully through OpenRouter.
|
||||
## Escalation
|
||||
If the failure persists, check DNS, certificate expiry and provider status.
|
||||
## Last successfully used
|
||||
2026-06-22
|
||||
2026-06-22 — **120 days ago.** This is the single oldest verification date still in the wiki
|
||||
alongside the original 2026-07-22 page set.
|
||||
|
||||
+27
-11
@@ -3,9 +3,9 @@ title: Test OpenRouter
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-07-22
|
||||
verified_on: 2026-07-22
|
||||
last_tested: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
last_tested: 2026-10-20
|
||||
confidence: high
|
||||
tags: [runbook, openrouter, providers, fallback]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -22,7 +22,8 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar
|
||||
- User wants to confirm OpenRouter is usable without relying on catalog names
|
||||
|
||||
## Prerequisites
|
||||
- Hermes CLI available: `/home/hermes/.local/bin/hermes`
|
||||
- Hermes CLI available: `hermes` on PATH (`/home/hermes/.local/bin/hermes` is a shim to the
|
||||
uv-managed runtime — see the note under **Last tested**)
|
||||
- Internet egress for opencode.ai, inference-api.nousresearch.com, ollama.com, kilo.ai, litellm:4000
|
||||
|
||||
## Procedure
|
||||
@@ -31,12 +32,22 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar
|
||||
```bash
|
||||
hermes fallback list
|
||||
```
|
||||
Expected active fallbacks on this install:
|
||||
1. `deepseek-v4-flash-free` via opencode-zen
|
||||
2. `stepfun/step-3.7-flash:free` via nous
|
||||
3. `qwen3.5:397b` via ollama-cloud
|
||||
4. `poolside/laguna-m.1:free` via kilocode
|
||||
5. `or-inclusionai_ling-2_6-1t` via custom/litellm
|
||||
Expected active fallbacks on this install (live `hermes fallback list`, 2026-10-20 — **6** entries):
|
||||
1. `mimo-v2.5` via opencode-go
|
||||
2. `deepseek-v4-flash-free` via opencode-zen — `https://opencode.ai/zen/v1`
|
||||
3. `stepfun/step-3.7-flash:free` via nous — `https://inference-api.nousresearch.com/v1`
|
||||
4. `qwen3.5:397b` via ollama-cloud — `https://ollama.com/v1`
|
||||
5. `poolside/laguna-m.1:free` via kilocode — `https://api.kilo.ai/api/gateway`
|
||||
6. `or-inclusionai_ling-2_6-1t` via custom — `http://litellm:4000/v1`
|
||||
|
||||
**Primary:** `meituan/longcat-2.5-preview:free` (via nous).
|
||||
|
||||
> **Changes since 2026-07-22:** `mimo-v2.5` via **opencode-go** has been **added at position 1** of the
|
||||
> chain, and the primary changed to `meituan/longcat-2.5-preview:free` (was `freellmapi-auto`).
|
||||
> The other five entries and their order are unchanged.
|
||||
> `hermes fallback list` also prints a standing notice on every invocation:
|
||||
> *"Claude Subscription DirectSDK (Experimental): Claude Code is not installed"* — expected on this
|
||||
> box, since `claude` is not on PATH. It is a notice, not a failure of the fallback chain.
|
||||
|
||||
### 2. Test OpenRouter-style direct access if present
|
||||
```bash
|
||||
@@ -77,7 +88,12 @@ Verify provider names and URLs in config match what the user wants to use.
|
||||
- This runbook tests confirmed fallback paths, not assumed ones
|
||||
|
||||
## Last tested
|
||||
2026-07-22
|
||||
2026-07-22 — re-run 2026-10-20 against live `hermes fallback list` output (6 entries, primary
|
||||
`meituan/longcat-2.5-preview:free`). Note the CLI reports Hermes Agent **v0.22.2 (2026.9.24),
|
||||
local `d923a5eb487a`**, and the `/home/hermes/.local/bin/hermes` launcher is a 73-byte shim that
|
||||
delegates to the uv-managed runtime at
|
||||
`/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/` — *not* the
|
||||
`~/.hermes/hermes-agent/venv/` path named in some older runbooks. Use the `hermes` on PATH.
|
||||
|
||||
## Related
|
||||
- [[provider-health]]
|
||||
|
||||
@@ -3,24 +3,99 @@ title: Troubleshoot Tool Loop
|
||||
type: runbook
|
||||
status: unresolved
|
||||
created: 2026-07-22
|
||||
updated: 2026-07-22
|
||||
review_after: 2026-08-22
|
||||
confidence: low
|
||||
updated: 2026-10-20
|
||||
review_after: 2026-11-19
|
||||
verified_on: 2026-10-20
|
||||
confidence: medium
|
||||
tags: [runbook, hermes]
|
||||
sources: []
|
||||
---
|
||||
|
||||
> **Stub — structure only.** Headings exist but content has not been verified/written yet. Flagged during 2026-07-22 wiki audit; fill in with verified facts or mark deprecated if no longer relevant.
|
||||
> **Filled in 2026-10-20** from live configuration, live process state and two recorded incidents.
|
||||
> The "stub" note below is retained as history.
|
||||
|
||||
# Troubleshoot Tool Loop
|
||||
|
||||
## Symptoms
|
||||
Agent loops on one tool or endpoint.
|
||||
## Procedure
|
||||
1. Inspect recent Hermes agent logs.
|
||||
2. Identify the looping tool path.
|
||||
3. Restart worker or flush stuck request.
|
||||
4. Re-run limited test scenario.
|
||||
## Verification
|
||||
Agent handles the same scenario without repeating.
|
||||
- The agent repeats the **same tool call** with identical arguments and makes no progress.
|
||||
- The agent repeats **itself in the response stream** (no tool involved) — the model loops on its
|
||||
own output and Hermes cuts the stream.
|
||||
|
||||
**These are two different faults with two different detectors.** Do not assume one covers the other.
|
||||
|
||||
## 1. Tool-call loop — detected by `tool_loop_guardrails`
|
||||
|
||||
### Symptoms
|
||||
Agent loops on one tool or endpoint. Hermes emits a warning naming the tool.
|
||||
|
||||
### Configuration (verified live 2026-10-20, `~/.hermes/config.yaml`)
|
||||
```yaml
|
||||
tool_loop_guardrails:
|
||||
warnings_enabled: true
|
||||
hard_stop_enabled: false
|
||||
warn_after:
|
||||
exact_failure: 2
|
||||
same_tool_failure: 3
|
||||
idempotent_no_progress: 2
|
||||
hard_stop_after:
|
||||
exact_failure: 5
|
||||
same_tool_failure: 8
|
||||
idempotent_no_progress: 5
|
||||
```
|
||||
> **`hard_stop_enabled: false` is the key fact.** Hermes will *warn* about a stuck tool loop but
|
||||
> will **not** force-stop itself. A genuinely stuck loop only gets a warning — a human (or the
|
||||
> messaging integration) is the real backstop. See [[concepts/tool-calling]].
|
||||
|
||||
### Procedure
|
||||
1. Inspect recent Hermes agent logs: `~/.hermes/logs/agent.log*` (grep for the tool name).
|
||||
2. Identify the looping tool path. Common culprits in this estate: a browser tool against a dead
|
||||
Camofox service, a terminal tool against a stale persistent shell, or a webhook that returns
|
||||
200 with an empty body.
|
||||
3. **Check the dependency, not just the agent.** On this estate the most frequent tool-loop cause
|
||||
was an upstream service being down while the tool still returned a response. Verify with the
|
||||
relevant runbook: [[runbooks/restart-browser]], [[runbooks/restart-hermes]],
|
||||
[[runbooks/provider-health]].
|
||||
4. If the loop is a stuck worker, restart it: `systemctl --user restart hermes-gateway.service`.
|
||||
5. Re-run a limited test scenario and confirm the guard no longer fires.
|
||||
|
||||
### Verification
|
||||
Agent handles the same scenario without repeating, and no `tool_loop_guardrails` warning appears
|
||||
in `agent.log`.
|
||||
|
||||
## 2. Model-output (degenerate stream) loop — detected by the stream guard
|
||||
|
||||
### Symptoms
|
||||
The run **delivers a canned non-answer** such as *"Your request was not processed. Send it again
|
||||
if you still want me to carry it out."*, and then fails with
|
||||
`RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding
|
||||
the degenerate response.`
|
||||
|
||||
### This is the 2026-10-18 case — see
|
||||
[[incidents/2026-10-18-degenerate-stream-canned-substitute]]
|
||||
|
||||
### Procedure
|
||||
1. Read the run's `error` in `~/.hermes/cron/executions.db` — the phrase `repetition loop`
|
||||
identifies this fault.
|
||||
2. **Check `delivery_outcome` on the same row.** In the 2026-10-18 incident the row was
|
||||
`delivery_outcome: delivered` — the canned non-answer reached Telegram *before* the run
|
||||
failed. **A delivered message is not evidence of delivered content.**
|
||||
3. Confirm the provider is healthy: the 2026-10-18 case rode the same `custom:omni-101` alias
|
||||
that produced the 2026-10-14/15 connection-refused storm and the 2026-10-16/17 and 2026-10-19
|
||||
resource-pressure 503 bursts. See [[runbooks/provider-health]] and
|
||||
[[incidents/2026-10-16-omniroute-503-resource-pressure]].
|
||||
4. Do **not** lower `tool_loop_guardrails` thresholds for this fault — they watch tool calls, and
|
||||
the degenerate-stream guard is a separate mechanism.
|
||||
|
||||
### Verification
|
||||
The next scheduled run of the job completes with `status: completed` and `error: null`, and the
|
||||
delivered message contains real content.
|
||||
|
||||
## Last successfully used
|
||||
2026-07-22
|
||||
2026-07-22 — re-verified against live config and live incidents on 2026-10-20; the *procedure* has
|
||||
not itself been executed end-to-end in a live loop, so `confidence` was raised only to `medium`.
|
||||
|
||||
## Related
|
||||
- [[concepts/tool-calling]]
|
||||
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
|
||||
- [[runbooks/provider-health]]
|
||||
- [[runbooks/restart-hermes]]
|
||||
|
||||
Reference in New Issue
Block a user