stocktake 2026-11-11: 36 stale pages re-verified (0 stale); fallback chain rewritten 4/6 slots; CT333 'clawtest' naming claim corrected; nous 404 credits signature replaces refresh-token 401; estate shrank (VM 310 + CT601 gone)
- freshness: 36 stale / 93 fresh / 3 no-date -> 0 stale / 129 fresh / 3 no-date of 132 (same-day report regenerated after the corrections, per pattern 13) - skills: 299 skills / 67 categories (count unchanged; generator arithmetic re-verified by hand against the live tree: 300 SKILL.md files - 1 nested duplicate = 299) - runbooks/test-openrouter: live fallback chain rewritten (primary back to smart-route/custom:omni-101; mimo-v2.5, stepfun, qwen3.5/ollama-cloud out; guacano + kilo-reflection-beam-free in); ollama.com dropped from the URL test list; guacano/litellm noted as internal + key-gated - comparisons/hermes-vs-nanobot: the 2026-10-20 'CT333 is zeroclaw in Proxmox naming' claim was itself wrong - pct list and hostname both return clawtest - incidents/2026-10-22-nous: refresh-token failure gone from logs; new signature 404 insufficient_credits_for_paid_model from auxiliary.pulse_memory (ninth slot, back on nous, paid model) - open item revised - incidents/2026-08-29-gmail: email crons now ok but root cause unfixed 74 days (himalaya message read still in fetch_newsletters.py; goimapnotify back up PID 441; monitor still paused) - 30 further pages re-verified live and de-flagged with dated probe notes (ct207 ingress, CT205 guanaco v0.8.15, CT270 nextcloud 33.0.2.2, headless stack, hermes-ui both hosts, systemd units + cgroup limits, terminal/tool_search/mnemosyne/custom_providers config blocks, ops profile still deleted, hosts/pve guest lists) - log.md addendum + index.md date bump; no page deleted, no secrets written
This commit is contained in:
@@ -3,8 +3,8 @@ title: Backup Wiki
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
last_tested: 2026-10-20
|
||||
confidence: high
|
||||
tags: [runbook, wiki, backup]
|
||||
@@ -35,6 +35,12 @@ Create a durable backup of the wiki so recoverable copies exist before mutating
|
||||
>
|
||||
> Also note `/home/hermes/wiki-backups/` does not yet exist — this runbook has never been run.
|
||||
|
||||
> **Re-verified 2026-11-11 — the page's own caveats still stand.** `/home/hermes/wiki-backups/`
|
||||
> **still does not exist**, so this runbook still has never been run end to end. `/home/hermes/wiki`
|
||||
> is still a git repo with **no remote** (`git remote -v` empty), and
|
||||
> `/home/hermes/workspace/agent-estate-wiki` remains the separate authoring repo pushed to Gitea
|
||||
> `Anthony/agent-estate-wiki`. Both trees are real and still need two separate backups.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Sanity-check repo state
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Diagnose Failed Docker Service
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
last_tested: 2026-10-20
|
||||
confidence: medium
|
||||
tags: [runbook, docker, homelab]
|
||||
@@ -35,6 +35,12 @@ Diagnose container or compose issues on hosts where Docker is present.
|
||||
> Note also that `systems/homepage-dashboard.md` now records the CT201 stack as **degraded** —
|
||||
> the main dashboard container is gone — which is exactly the symptom this runbook would be used for.
|
||||
|
||||
> **Scoping note re-confirmed 2026-11-11.** `id -nG hermes` still contains **no `docker` group**
|
||||
> membership, so on CT460 `docker ps` as `hermes` still fails with
|
||||
> `permission denied while trying to connect to the docker API at unix:///var/run/docker.sock`.
|
||||
> The rule stands: on this box use `sudo docker …`, or SSH/`pct exec` to the host that owns the
|
||||
> container. (No container claims on this page were re-verified in this pass.)
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Check Docker daemon state
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Hermes Gateway Resource Resilience
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-25
|
||||
updated: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
review_after: 2026-11-19
|
||||
verified_on: 2026-10-20
|
||||
verified_on: 2026-11-11
|
||||
version_applies_to: Proxmox CT460
|
||||
confidence: high
|
||||
tags: [hermes, infrastructure, runbook]
|
||||
@@ -96,6 +96,15 @@ consuming most of the container and taking Telegram down with them.
|
||||
> `hermes` user cannot read (permission denied) — its *effective* values are still verifiable via
|
||||
> `systemctl show`, which is what the health-check section below uses.
|
||||
|
||||
> **All effective values re-confirmed live 2026-11-11 via `systemctl show`.**
|
||||
> Gateway (`hermes-gateway.service`, active): `OOMPolicy=continue`, `MemoryLow=536870912`,
|
||||
> `MemoryHigh=infinity`, `MemoryMax=infinity`, `CPUWeight=1000`, `Restart=always`; the drop-in file
|
||||
> still reads `OOMPolicy=continue` / `MemoryLow=512M` / `CPUWeight=1000`. Camofox
|
||||
> (`camofox-browser.service`, active): `OOMPolicy=kill`, `MemoryHigh=2147483648`,
|
||||
> `MemoryMax=3221225472`, `MemorySwapMax=1073741824` — the raised 2G/3G/1G ceilings hold. Web UI
|
||||
> (`hermes-webui.service`, active): `OOMPolicy=kill`, `MemoryHigh=1073741824`,
|
||||
> `MemoryMax=1610612736`, `MemorySwapMax=1073741824`. Unchanged since 2026-10-20.
|
||||
|
||||
## I/O priority limitation
|
||||
|
||||
Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Multiplexer Setup (Multi-Profile Gateway)
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-28
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [hermes, multiplexer, telegram, profiles, gateway]
|
||||
---
|
||||
@@ -89,6 +89,14 @@ config can no longer be read at `~/.hermes/profiles/ops/config.yaml` because tha
|
||||
Until that is settled, treat the ops chat IDs as **possibly unroutable** and verify the gateway
|
||||
actually resolves them before relying on this runbook.
|
||||
|
||||
> **Re-probed 2026-11-11 — the situation is unchanged and the two open decisions are still open.**
|
||||
> /home/hermes/.hermes/profiles/` contains **only** `.deleted/ops` (8-byte marker reading
|
||||
> `deleted`, dated 2026-10-11); `~/.hermes/profiles/ops/` still does not exist. The default
|
||||
> profile's config **still** carries `multiplex_profiles: true` and both `profile_routes` entries
|
||||
> (`ops-group` → `-1003914987043`, `ops-group-2` → `-1003932503629`, both `profile: ops`), and
|
||||
> `telegram.allowed_chats` / `group_allowed_chats` still list all three chat IDs. So the two ops
|
||||
> chats still route to a profile that no longer exists — **31 days unresolved**.
|
||||
|
||||
## The Silent-Drop Pitfall
|
||||
|
||||
The `allowed_chats` gate runs on the **default profile's adapter** BEFORE `profile_routes` stamps `source.profile`. If a chat is only authorized in the secondary profile's config but not the default's, messages from that chat are silently dropped.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Restart Browser Helper
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
last_tested: 2026-10-20
|
||||
confidence: high
|
||||
tags: [runbook, browser, cua-driver, camofox]
|
||||
@@ -62,6 +62,13 @@ pkill -f "fluxbox"
|
||||
/home/hermes/.local/bin/cua-driver serve
|
||||
```
|
||||
|
||||
> **Procedure re-checked against live state 2026-11-11.** `Xvfb :99`, `x11vnc -display :99
|
||||
> -forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw`, `fluxbox` and **both**
|
||||
> websockify instances are running; `cua-driver serve` is up (PID 4176) and
|
||||
> `/home/hermes/.local/bin/cua-driver --version` reports **0.28.2**, matching the newest entry in
|
||||
> `/home/hermes/.cua-driver/packages/releases/` (`0.19.3`, `0.22.2`, `0.23.2`, `0.24.0`, `0.28.2`).
|
||||
> **5900 still has no listener** — the 6080 websockify still points at a dead port. All as recorded.
|
||||
|
||||
## Verification
|
||||
- `cua-driver serve` process is present in `ps`
|
||||
- `5901` and `6081` are reachable
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Restart Hermes
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
last_tested: 2026-10-20
|
||||
confidence: high
|
||||
tags: [runbook, hermes, restart]
|
||||
@@ -85,6 +85,17 @@ pkill -f "hermes-webui/server.py"
|
||||
/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py
|
||||
```
|
||||
|
||||
> **Steps re-checked live 2026-11-11.** All five user units are **active (running)**:
|
||||
> `hermes-gateway.service`, `hermes-dashboard.service`, `hermes-ui.service`,
|
||||
> `hermes-ui-prod.service`, `hermes-wiki-serve.service`. The gateway runs from the **uv-managed
|
||||
> runtime** (`/home/hermes/.hermes/tools/python-3.14.9+202****1021-linux-x64/bin/python3 … gateway
|
||||
> run`) through the `runpy` bootstrap that carries `_hermes_closure_repair()`; the dashboard is
|
||||
> `hermes dashboard --host 0.0.0.0 --port 9119 --no-open`; the WebUI is still
|
||||
> `/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py` (port
|
||||
> 8787); `cua-driver serve` is running (PID 4176, cua-driver **0.28.2**); x11vnc is on `:99`/5901.
|
||||
> Note the runtime date-stamp in the interpreter path has advanced (`…14.9+20261021…` vs the
|
||||
> `…14.7+20260901…` recorded on 2026-10-20) — always resolve the path live, never hardcode it.
|
||||
|
||||
## Verification
|
||||
```bash
|
||||
hermes gateway status
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Restore OpenRouter connectivity
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
tags: [runbook, openrouter, recovery]
|
||||
sources: []
|
||||
confidence: high
|
||||
@@ -37,6 +37,11 @@ confidence: high
|
||||
>
|
||||
> **Last successfully used: 2026-06-22** — which predates every configuration referenced on this
|
||||
> page. Treat the page as unverified until someone runs it end to end.
|
||||
> **The "out of step" flag re-confirmed 2026-11-11.** There is still **no `GoOp` binary** (`which
|
||||
> GoOp` and `/usr/local/bin/GoOp`, `/opt/GoOp` all absent), no `GoOp` subcommand in `hermes
|
||||
> --help`, and **no WARP process** (`pgrep -af warp` empty) and no WARP client configured. The
|
||||
> substitute procedure recorded on 2026-10-20 is therefore still the correct one.
|
||||
|
||||
## Verification
|
||||
A test prompt returns successfully through OpenRouter.
|
||||
## Escalation
|
||||
|
||||
+41
-17
@@ -3,8 +3,8 @@ title: Test OpenRouter
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
last_tested: 2026-10-20
|
||||
confidence: high
|
||||
tags: [runbook, openrouter, providers, fallback]
|
||||
@@ -24,7 +24,8 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar
|
||||
## Prerequisites
|
||||
- Hermes CLI available: `hermes` on PATH (`/home/hermes/.local/bin/hermes` is a shim to the
|
||||
uv-managed runtime — see the note under **Last tested**)
|
||||
- Internet egress for opencode.ai, inference-api.nousresearch.com, ollama.com, kilo.ai, litellm:4000
|
||||
- Internet egress for opencode.ai, inference-api.nousresearch.com, kilo.ai (ollama.com is no
|
||||
longer in the live chain — entry 3 uses the internal `guacano` endpoint instead)
|
||||
|
||||
## Procedure
|
||||
|
||||
@@ -32,22 +33,38 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar
|
||||
```bash
|
||||
hermes fallback list
|
||||
```
|
||||
Expected active fallbacks on this install (live `hermes fallback list`, 2026-10-20 — **6** entries):
|
||||
1. `mimo-v2.5` via opencode-go
|
||||
2. `deepseek-v4-flash-free` via opencode-zen — `https://opencode.ai/zen/v1`
|
||||
3. `stepfun/step-3.7-flash:free` via nous — `https://inference-api.nousresearch.com/v1`
|
||||
4. `qwen3.5:397b` via ollama-cloud — `https://ollama.com/v1`
|
||||
Expected active fallbacks on this install (live `hermes fallback list`, 2026-11-11 — **6** entries):
|
||||
1. `meituan/longcat-2.5-preview:free` via nous
|
||||
2. `deepseek-v4.1-flash` via opencode-zen — `https://opencode.ai/zen/v1`
|
||||
3. `deepseek-v4.1-flash` via custom:guacano — `http://ollama:11000/v1`
|
||||
4. `kilo-reflection-beam-free` via custom:litellm — `http://litellm:4000/v1`
|
||||
5. `poolside/laguna-m.1:free` via kilocode — `https://api.kilo.ai/api/gateway`
|
||||
6. `or-inclusionai_ling-2_6-1t` via custom — `http://litellm:4000/v1`
|
||||
|
||||
**Primary:** `meituan/longcat-2.5-preview:free` (via nous).
|
||||
**Primary:** `smart-route` (via `custom:omni-101`, `http://100.78.63.20:20128`).
|
||||
|
||||
> **Changes since 2026-07-22:** `mimo-v2.5` via **opencode-go** has been **added at position 1** of the
|
||||
> chain, and the primary changed to `meituan/longcat-2.5-preview:free` (was `freellmapi-auto`).
|
||||
> The other five entries and their order are unchanged.
|
||||
> `hermes fallback list` also prints a standing notice on every invocation:
|
||||
> *"Claude Subscription DirectSDK (Experimental): Claude Code is not installed"* — expected on this
|
||||
> box, since `claude` is not on PATH. It is a notice, not a failure of the fallback chain.
|
||||
> **Chain rewritten 2026-11-11 — the 2026-10-20 list recorded above was materially stale.**
|
||||
> Live `hermes fallback list` now reads (verbatim):
|
||||
>
|
||||
> ```
|
||||
> Primary: smart-route (via custom:omni-101)
|
||||
> Fallback chain (6 entries):
|
||||
> 1. meituan/longcat-2.5-preview:free (via nous)
|
||||
> 2. deepseek-v4.1-flash (via opencode-zen) [https://opencode.ai/zen/v1]
|
||||
> 3. deepseek-v4.1-flash (via custom:guacano)
|
||||
> 4. kilo-reflection-beam-free (via custom:litellm)
|
||||
> 5. poolside/laguna-m.1:free (via kilocode) [https://api.kilo.ai/api/gateway]
|
||||
> 6. or-inclusionai_ling-2_6-1t (via custom) [http://litellm:4000/v1]
|
||||
> ```
|
||||
>
|
||||
> What changed since the 2026-10-20 reading: the **primary moved back to OmniRoute**
|
||||
> (`smart-route` via `custom:omni-101`, matching the live `model:` block — see
|
||||
> [[concepts/model-routing]]); `mimo-v2.5` via opencode-go and `stepfun/step-3.7-flash:free` via
|
||||
> nous have **left** the chain; `qwen3.5:397b` via ollama-cloud has **left** (so `ollama.com` no
|
||||
> longer needs probing); `deepseek-v4.1-flash` via `custom:guacano` and `kilo-reflection-beam-free`
|
||||
> via `custom:litellm` have **joined** at positions 3 and 4; positions 5 and 6 are unchanged.
|
||||
> The standing *"Claude Subscription DirectSDK … Claude Code is not installed"* notice recorded on
|
||||
> 2026-10-20 **did not print** on the 2026-11-11 run.
|
||||
|
||||
### 2. Test OpenRouter-style direct access if present
|
||||
```bash
|
||||
@@ -60,7 +77,6 @@ Current install state as of last run: OpenRouter is not configured as a standalo
|
||||
for url in \
|
||||
"https://opencode.ai/zen/v1/models" \
|
||||
"https://inference-api.nousresearch.com/v1/models" \
|
||||
"https://ollama.com/v1/models" \
|
||||
"https://api.kilo.ai/api/gateway/models"; do
|
||||
printf "%s -> " "$url"
|
||||
curl -s -o /dev/null -w "%{http_code}" --max-time 8 "$url"
|
||||
@@ -68,6 +84,13 @@ for url in \
|
||||
done
|
||||
```
|
||||
|
||||
Entries 3 (`custom:guacano` → `http://ollama:11000/v1`) and 6 (`custom` → `http://litellm:4000/v1`)
|
||||
are **internal to the estate** and both now **require authentication**: a bare `curl` returns
|
||||
**401 `invalid_api_key`**. Their keys live in `~/.hermes/.env` behind each provider's `key_env`
|
||||
(e.g. `HERMES_CUSTOM_LITELLM_API_KEY`). `guacano` resolves through CT205 over the tailnet/DNS name
|
||||
`ollama`, so it is only reachable from inside the estate — a 401 there proves auth reached the
|
||||
node, not that it is down (see [[runbooks/provider-health]]).
|
||||
|
||||
### 4. Check Hermes fallback config reflects reality
|
||||
```bash
|
||||
grep -A5 "^fallback:" ~/.hermes/config.yaml
|
||||
@@ -89,7 +112,8 @@ Verify provider names and URLs in config match what the user wants to use.
|
||||
|
||||
## Last tested
|
||||
2026-07-22 — re-run 2026-10-20 against live `hermes fallback list` output (6 entries, primary
|
||||
`meituan/longcat-2.5-preview:free`). Note the CLI reports Hermes Agent **v0.22.2 (2026.9.24),
|
||||
`meituan/longcat-2.5-preview:free`), and again on **2026-11-11** against the live output recorded
|
||||
above (6 entries, primary `smart-route` via `custom:omni-101`). Note the CLI reports Hermes Agent **v0.22.2 (2026.9.24),
|
||||
local `d923a5eb487a`**, and the `/home/hermes/.local/bin/hermes` launcher is a 73-byte shim that
|
||||
delegates to the uv-managed runtime at
|
||||
`/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/` — *not* the
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Troubleshoot Tool Loop
|
||||
type: runbook
|
||||
status: unresolved
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
review_after: 2026-11-19
|
||||
verified_on: 2026-10-20
|
||||
verified_on: 2026-11-11
|
||||
confidence: medium
|
||||
tags: [runbook, hermes]
|
||||
sources: []
|
||||
@@ -94,6 +94,12 @@ delivered message contains real content.
|
||||
2026-07-22 — re-verified against live config and live incidents on 2026-10-20; the *procedure* has
|
||||
not itself been executed end-to-end in a live loop, so `confidence` was raised only to `medium`.
|
||||
|
||||
> **Guardrail config re-read 2026-11-11 — byte-for-byte unchanged.**
|
||||
> `tool_loop_guardrails`: `warnings_enabled: true`, `hard_stop_enabled: false`,
|
||||
> `warn_after {exact_failure: 2, same_tool_failure: 3, idempotent_no_progress: 2}`,
|
||||
> `hard_stop_after {exact_failure: 5, same_tool_failure: 8, idempotent_no_progress: 5}`.
|
||||
> `hard_stop_enabled: false` therefore still means a stuck tool loop only ever *warns*.
|
||||
|
||||
## Related
|
||||
- [[concepts/tool-calling]]
|
||||
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
|
||||
|
||||
Reference in New Issue
Block a user