wiki: publish pending edits (2026-10-20)

This commit is contained in:
Tony0410
2026-10-20 18:38:11 +08:00
parent 667ab7f41d
commit fbd4f297eb
46 changed files with 1012 additions and 197 deletions
+6 -3
View File
@@ -3,8 +3,8 @@ title: Contributing to the Agent Estate Wiki
type: guide type: guide
status: active status: active
created: 2026-08-16 created: 2026-08-16
updated: 2026-08-16 updated: 2026-10-20
verified_on: 2026-08-16 verified_on: 2026-10-20
confidence: high confidence: high
tags: [wiki, contributing, governance, multi-agent] tags: [wiki, contributing, governance, multi-agent]
sources: [] sources: []
@@ -19,7 +19,10 @@ sources: []
The single source of truth for operational knowledge across the agent estate: The single source of truth for operational knowledge across the agent estate:
- **Hermes production** (CT460), **Hermes Chaos** (CT450) - **Hermes production** (CT460), **Hermes Chaos** (CT450)
- **Nanobot** (CT333), **ZeroClaw** (CT333), **QwenPaw** (CT465), **Open-Claw** (VM 403) - **Nanobot** (CT333), **ZeroClaw** (CT333), **QwenPaw** (CT465)
- ~~**Open-Claw** (VM 403)~~ — **removed from this list 2026-10-20.** No VM 403 exists on either
Proxmox host (PVE `100.96.100.82` or m93p `100.104.51.98`) and there is no `open-claw` Tailscale
peer. See [[agents/open-claw]].
- Shared infrastructure (Proxmox, Tailscale, OmniRoute/LiteLLM, Docker, domains) - Shared infrastructure (Proxmox, Tailscale, OmniRoute/LiteLLM, Docker, domains)
**One rule above all: this wiki documents the estate. It does not replace any agent's own configuration, memory, or logs.** Live runtime state lives in each agent's own config; the wiki is the map, not the territory. **One rule above all: this wiki documents the estate. It does not replace any agent's own configuration, memory, or logs.** Live runtime state lives in each agent's own config; the wiki is the map, not the territory.
+10 -1
View File
@@ -3,7 +3,8 @@ title: Agent Estate
type: system type: system
status: active status: active
created: 2026-08-15 created: 2026-08-15
updated: 2026-08-15 updated: 2026-10-20
verified_on: 2026-10-20
confidence: high confidence: high
tags: [agents, estate, overview] tags: [agents, estate, overview]
--- ---
@@ -31,6 +32,14 @@ The agent estate consists of multiple Hermes instances, a QwenPaw deployment, Ze
Live probe on 2026-10-19 (`pct list` + `qm list` on PVE host 100.96.100.82) shows **no VM 403 and no VM 401**. The QEMU VMs on the host are 100, 101, 110, 282, 310. The `open-claw` and `zeroclaw` pages still document the intended configuration (from 2026-08-15), but the VM is currently absent — treat `agents/open-claw.md` as **stale/unverified** until the VM is recreated or a new location is documented. Live probe on 2026-10-19 (`pct list` + `qm list` on PVE host 100.96.100.82) shows **no VM 403 and no VM 401**. The QEMU VMs on the host are 100, 101, 110, 282, 310. The `open-claw` and `zeroclaw` pages still document the intended configuration (from 2026-08-15), but the VM is currently absent — treat `agents/open-claw.md` as **stale/unverified** until the VM is recreated or a new location is documented.
**Extended to the second PVE host on 2026-10-20.** `m93p` (`100.104.51.98`) was probed as well and
also has **no VM 403** (its LXC guests are 100, 101, 102, 104, 105, 221, 245, 500, 998, 1006, 1007,
1008, 1241, 1260, 1261, 1262, 1263, 1264). The `tailscale status` peer list contains **no
`open-claw` node** either. Open-Claw is therefore absent from the whole estate, not just relocated.
**ZeroClaw, by contrast, was re-verified live on 2026-10-20** and **is running** on CT333
(`/usr/local/bin/zeroclaw daemon --allow-degraded-security`, PID 136) — see [[agents/zeroclaw]].
## Shared Infrastructure ## Shared Infrastructure
All agents share: All agents share:
+12 -2
View File
@@ -3,8 +3,8 @@ title: Open-Claw
type: system type: system
status: active status: active
created: 2026-08-15 created: 2026-08-15
updated: 2026-08-15 updated: 2026-10-20
verified_on: 2026-08-15 verified_on: 2026-10-20
confidence: high confidence: high
tags: [openclaw, open-claw, vm403, gateway, agent] tags: [openclaw, open-claw, vm403, gateway, agent]
sources: [skill-inspection, config-reference] sources: [skill-inspection, config-reference]
@@ -29,6 +29,16 @@ Autonomous AI agent gateway running on a separate QEMU VM. Docker-based with Tai
## Current State ## Current State
- **Status: NOT PRESENT.** Live probe 2026-10-19: no VM 403 on PVE host 100.96.100.82 (`qm list` shows only VMs 100, 101, 110, 282, 310; `pct list` shows CTs 103, 200, 201, 205, 207, 209, 215, 255, 270, 299, 333, 450, 460, 465, 475, 555, 600, 601). The VM was last known active on 2026-08-15. The content below documents the intended configuration and is **stale/unverified** until the VM is recreated or relocated. - **Status: NOT PRESENT.** Live probe 2026-10-19: no VM 403 on PVE host 100.96.100.82 (`qm list` shows only VMs 100, 101, 110, 282, 310; `pct list` shows CTs 103, 200, 201, 205, 207, 209, 215, 255, 270, 299, 333, 450, 460, 465, 475, 555, 600, 601). The VM was last known active on 2026-08-15. The content below documents the intended configuration and is **stale/unverified** until the VM is recreated or relocated.
- **Re-probed on the SECOND Proxmox host too, 2026-10-20.** The `m93p` host (`100.104.51.98`) has
its own guest list — LXC 100 `obsidian-livesync`, 101 `omniroute`, 102 `vaultwarden`, 104 `rclone`,
105 `conduit`, 221 `gitea`, 245 `openweb`, 500 `litellm`, 1006 `freellmapi`, 1007 `homebridge`,
1008 `rustdesk`, 1241 `ntfy`, 1260 `freshrss`, 1261 `paseo`, 1262 `mailflow`,
1263 `social-dashboard`, 1264 `openmausbot`, 998 `nightbridge` (stopped) — and **no VM 403 there
either**. The Tailscale peer list likewise contains **no `open-claw` node**. So Open-Claw is not
merely moved between hosts: it is gone from both.
- **Consequence for the rest of this page:** the Tailscale IP `100.64.172.34`, the LAN IP
`192.168.178.193` and the control UI `https://open-claw.kangaroo-eel.ts.net` are all historical.
Treat the whole page as a design record, not a description of a running system.
- **Config format:** JSON - **Config format:** JSON
- **Auth:** Password-based (not paired tokens) - **Auth:** Password-based (not paired tokens)
- **Software:** Docker-based (community-scripts template) - **Software:** Docker-based (community-scripts template)
+11 -2
View File
@@ -3,8 +3,8 @@ title: ZeroClaw
type: system type: system
status: active status: active
created: 2026-08-15 created: 2026-08-15
updated: 2026-08-15 updated: 2026-10-20
verified_on: 2026-08-15 verified_on: 2026-10-20
confidence: high confidence: high
tags: [zeroclaw, ct333, daemon, agent] tags: [zeroclaw, ct333, daemon, agent]
sources: [skill-inspection, config-reference] sources: [skill-inspection, config-reference]
@@ -31,6 +31,15 @@ Autonomous AI daemon running on CT333. Handles independent agent operations with
- **Workspace:** `/root/.zeroclaw/workspace` - **Workspace:** `/root/.zeroclaw/workspace`
- **State:** `/root/.zeroclaw/state/` - **State:** `/root/.zeroclaw/state/`
> **Re-verified 2026-10-20 — running as documented, with one path nuance.**
> The daemon **is** live on CT333: `pgrep -af zeroclaw` → `136 /usr/local/bin/zeroclaw daemon
> --allow-degraded-security` (PID 136, up since boot). Note the process runs
> **`/usr/local/bin/zeroclaw`**, which is a **symlink → `/root/.cargo/bin/zeroclaw`** — a *third*
> path alongside the `/root/.local/bin/zeroclaw` binary named above (that file also exists, 37 MB).
> The `--allow-degraded-security` flag was **not** recorded on this page previously; it is part of
> the live command line.
> `/root/.zeroclaw/` contains `config.toml` plus ~15 dated `config.toml.bak*` snapshots.
## Key Paths ## Key Paths
| Item | Path | | Item | Path |
+18 -5
View File
@@ -3,8 +3,8 @@ title: Camoufox vs Chrome
type: comparison type: comparison
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
confidence: medium confidence: medium
tags: [comparison, browser] tags: [comparison, browser]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -20,14 +20,26 @@ Compare the two browser-automation options for Hermes's browser tool.
|---|---|---| |---|---|---|
| In use on this install | **Yes** — configured `cloud_provider` | No evidence found | | In use on this install | **Yes** — configured `cloud_provider` | No evidence found |
| Config | `browser.camofox.managed_persistence: true` (note: key is spelled `camofox` in config, not `camoufox`) | n/a | | Config | `browser.camofox.managed_persistence: true` (note: key is spelled `camofox` in config, not `camoufox`) | n/a |
| Driver | `cua-driver` v0.9.0 at `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | n/a | | Driver | `cua-driver` **v0.28.2** at `/home/hermes/.local/bin/cua-driver` (the `0.9.0` release path this page used to cite no longer exists) | n/a |
| Display | Headed, via Xvfb `:99` + fluxbox — see [[systems/headless-server]] | n/a | | Display | Headed, via Xvfb `:99` + fluxbox — see [[systems/headless-server]] | n/a |
| `use_gateway` | `false` | n/a | | `use_gateway` | `false` | n/a |
## Detail ## Detail
### Camoufox (in use) ### Camoufox (in use)
- **Verified:** `browser.cloud_provider: camofox`, `inactivity_timeout: 120s`, `allow_private_urls: true`. - **Verified 2026-10-20:** `browser.cloud_provider: steel`, `browser.engine: camofox`,
`browser.headed: true`, `browser.inactivity_timeout: 450`, `browser.use_gateway: false`,
`browser.camera_provider: camofox`, `browser.allow_private_urls: true`,
`browser.allow_unsafe_evaluate: true`, `browser.cdp_url: ''`,
`browser.command_timeout: 185`.
`browser.camofox:` → `adopt_existing_tab: true`, `managed_persistence: true`,
`rewrite_loopback_urls: true`, `user_id: anthony`.
**Note `inactivity_timeout` is 450s, not the 120s recorded here** — that value changed.
`cloud_provider` is `steel` with `engine: camofox`; the two keys describe different layers.
- **Live processes (2026-10-20):** `cua-driver serve` (PID 3312393), `camoufox-ws-server.mjs`,
and a live `camoufox-bin` (FirefoxDev build) tree under `/home/hermes/.cache/camoufox/`.
Camofox's REST port **9377 is not listening** on CT460 — the Camofox browser service runs on
**CT450**; see [[systems/browser-backend]].
- Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing. - Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing.
- `cua-driver` bridges Hermes's computer-use tool calls to the browser. - `cua-driver` bridges Hermes's computer-use tool calls to the browser.
@@ -36,7 +48,8 @@ Compare the two browser-automation options for Hermes's browser tool.
- Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome. - Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome.
## Verdict ## Verdict
This install uses **Camoufox exclusively**. There's no evidence Chrome/Chromium was ever wired in as an alternative on this host — the comparison exists more as a placeholder for a decision that may have been made outside this wiki than as an active either/or choice today. This install uses **Camoufox exclusively** (via the `cua-driver` + `camoufox-ws-server.mjs` stack
on CT460 and a separate Camofox service on CT450). There's no evidence Chrome/Chromium was ever wired in as an alternative on this host — the comparison exists more as a placeholder for a decision that may have been made outside this wiki than as an active either/or choice today.
## Related ## Related
- [[systems/browser-backend]] - [[systems/browser-backend]]
+12 -2
View File
@@ -3,8 +3,8 @@ title: Hermes vs Nanobot
type: comparison type: comparison
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
confidence: high confidence: high
tags: [comparison, hermes, nanobot] tags: [comparison, hermes, nanobot]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -22,6 +22,16 @@ Compare Hermes (this agent) with Nanobot, another agent process in the homelab,
- It's publicly reachable via the same Cloudflare Tunnel that fronts Hermes: `nanobot.martinwa.org` → `100.122.189.81:8766`, and `nano-ssh.martinwa.org` → SSH on the same host. - It's publicly reachable via the same Cloudflare Tunnel that fronts Hermes: `nanobot.martinwa.org` → `100.122.189.81:8766`, and `nano-ssh.martinwa.org` → SSH on the same host.
- The earlier note "Nanobox on the PVE host at `192.168.178.39`" conflated Nanobot with `pve`'s own LAN IP (that address is Proxmox's web UI, per the tunnel's `pve.martinwa.org` entry) — an error, not a second Nanobot instance. - The earlier note "Nanobox on the PVE host at `192.168.178.39`" conflated Nanobot with `pve`'s own LAN IP (that address is Proxmox's web UI, per the tunnel's `pve.martinwa.org` entry) — an error, not a second Nanobot instance.
> **Re-verified 2026-10-20 — the core claims on this page still hold.** The `clawtest` peer is
> still on Tailscale at **`100.122.189.81`** (`tailscale status` → `100.122.189.81 clawtest
> anthonymau@ linux -`) and `http://nanobot.martinwa.org` still answers (**301**). So the
> correction recorded above remains accurate 90 days on.
>
> **One caveat worth adding:** CT333 now runs **both** Nanobot *and* ZeroClaw (the latter verified
> live on 2026-10-20 — see [[agents/zeroclaw]]). The page's framing of CT333 as the Nanobot host
> is not wrong, but the host is no longer single-purpose. Note also that **CT333 is `zeroclaw` in
> the current Proxmox naming** — the wiki uses the older "clawtest" label for the same LXC.
## At a glance ## At a glance
| | Hermes (ct460) | Nanobot (ct333 "clawtest") | | | Hermes (ct460) | Nanobot (ct333 "clawtest") |
|---|---|---| |---|---|---|
+15 -2
View File
@@ -3,8 +3,8 @@ title: Memory Backends
type: comparison type: comparison
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-08-12 updated: 2026-10-20
verified_on: 2026-08-12 verified_on: 2026-10-20
confidence: high confidence: high
tags: [comparison, memory] tags: [comparison, memory]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -30,6 +30,19 @@ The three archived scripts (`hindsight-watchdog.py.archived`, `hindsight_migrati
- Risk assessment flagged: schema mismatch, embedding drift, scope leakage, provenance loss, and one-way/no-rollback migration with no dry-run snapshot. - Risk assessment flagged: schema mismatch, embedding drift, scope leakage, provenance loss, and one-way/no-rollback migration with no dry-run snapshot.
- **Verdict:** stay on Mnemosyne. Config verified `memory.provider: mnemosyne` (2026-08-12). No migration was performed. - **Verdict:** stay on Mnemosyne. Config verified `memory.provider: mnemosyne` (2026-08-12). No migration was performed.
> **Re-verified 2026-10-20 — Mnemosyne is still the only backend, and the OpenViking decision still
> holds.** `memory.provider: mnemosyne`, `memory_enabled: true`, `flush_min_turns: 6`,
> `memory_char_limit: 5000`, `user_char_limit: 3000`, `user_profile_enabled: true`,
> `nudge_interval: 10`; Mnemosyne sub-block `auto_sleep: true`, `default_scope: session`,
> `vector_type: int8`, `sleep_threshold: 50`, `profile_isolation: true`,
> `shared_surface_path: data/shared/mnemosyne.db`, `shared_surface_read: true`.
> No `openviking` key anywhere in `config.yaml`. No migration performed.
>
> **New since 2026-08-12:** `mnemosyne.skip_contexts: [cron, flush, subagent, background,
> skill_loop]` and `reflect.disabled_for_cron: true` — Mnemosyne capture is deliberately skipped
> in cron contexts, which is why scheduled jobs produce no working memories. See
> [[concepts/agent-memory]] and [[systems/mnemosyne]].
## Related ## Related
- [[systems/mnemosyne]] - [[systems/mnemosyne]]
- [[systems/memory-backend]] - [[systems/memory-backend]]
+38 -8
View File
@@ -3,8 +3,8 @@ title: Model Routers
type: comparison type: comparison
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-08-12 updated: 2026-10-20
verified_on: 2026-08-12 verified_on: 2026-10-20
confidence: high confidence: high
tags: [comparison, routing] tags: [comparison, routing]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -18,17 +18,47 @@ Compare LiteLLM, OmniRoute, and OpenRouter as model-routing layers for Hermes.
## At a glance ## At a glance
| | LiteLLM | OmniRoute | OpenRouter | | | LiteLLM | OmniRoute | OpenRouter |
|---|---|---|---| |---|---|---|---|
| Role in Hermes | **Default/primary** provider | **Fallback/auxiliary** provider | Not configured as a standalone provider | | Role in Hermes | Fallback + a *custom_providers* entry | **Auxiliary + the pin on essentially every model-calling cron job** | Not configured as a standalone provider |
| Endpoint | `http://litellm:4000/v1` | `http://omniroute:20128/v1` (public: `omniroute.martinwa.org`) | n/a | | Endpoint | `http://litellm:4000/v1` | `http://omniroute:20128/v1` (public: `omniroute.martinwa.org`) **and** `http://100.78.63.20:20128` (`omni-101`) | n/a |
| Advertised model count | ~1512 | ~3430 | n/a | | Model count (live, 2026-10-20) | **1869** (`/v1/models` with the configured key) | **6457** at `omniroute:20128/v1/models` | n/a |
| Model count (declared in `config.yaml`) | 1728 | 5398 (`Omniroute:20128`) / 5384 (`omni-101`) | n/a |
| Routing style | Straight proxy | Combo strategy, LKGP scoring, session affinity | n/a | | Routing style | Straight proxy | Combo strategy, LKGP scoring, session affinity | n/a |
| Aux roles used for | — | `skills_hub`, `approval`, `mcp` | — | | Aux roles used for | — | `skills_hub`, `approval`, `mcp` | — |
| Config presence | `custom_providers` in `~/.hermes/config.yaml` | same | absent from `custom_providers`; only referenced via a kept [[decisions/keep-openrouter-as-fallback]] decision | | Config presence | `custom_providers` in `~/.hermes/config.yaml` | same (two separate entries) | absent from `custom_providers`; only referenced via a kept [[decisions/keep-openrouter-as-fallback]] decision |
> **The "LiteLLM is the default/primary" row was wrong by 2026-10-20 (verified and corrected).**
> The live `model:` block in `~/.hermes/config.yaml` is:
> ```yaml
> model:
> provider: nous
> default: meituan/longcat-2.5-preview:free
> base_url: https://inference-api.nousresearch.com/v1
> api_mode: chat_completions
> persist_switch_by_default: false
> ```
> So the **primary model path is Nous Research direct**, not LiteLLM. LiteLLM is the *last* entry
> in the fallback chain (`or-inclusionai_ling-2_6-1t` via `custom` → `http://litellm:4000/v1`).
>
> **The two OmniRoute endpoints matter.** `Omniroute:20128` is a DNS name (m93p CT101);
> `omni-101` is the raw IP `100.78.63.20:20128` (node `omniroute-1`) and is the alias pinned on
> effectively every model-calling cron job. The two carry **different catalogs** (5398 vs 5384
> declared models) and the `omni-101` entry's own `model:` is `smart-route` while
> `Omniroute:20128`'s is `go-cheap`.
>
> **Also: both OmniRoute endpoints and LiteLLM now require authentication.** `curl` without a key
> returns **401 `invalid_api_key`** on all three `/v1/models` endpoints. The keys live in
> `~/.hermes/.env` via each provider's `key_env` (e.g. `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`).
> Any runbook that probes these endpoints with a bare `curl` will see 401 and may wrongly conclude
> the provider is down — it is not.
## Detail ## Detail
### LiteLLM — primary ### LiteLLM — fallback tail + a `custom_providers` entry
Default provider for main chat traffic (`freellmapi-auto` default model resolves through it). See [[systems/model-providers]]. LiteLLM hosts the `freellmapi-auto` model alias and is reachable at `http://litellm:4000/v1`, but
it is **no longer the default chat path** — that is Nous direct
(`meituan/longcat-2.5-preview:free`, per the live `model:` block). LiteLLM appears in the
fallback chain as `or-inclusionai_ling-2_6-1t` via the `custom` provider.
See [[systems/model-providers]] and [[concepts/model-routing]].
### FreeLLMAPI — the router behind `freellmapi-auto` ### FreeLLMAPI — the router behind `freellmapi-auto`
Unified LLM router at `100.82.40.23:3001` (also `freellmapi.martinwa.org`). Verified 2026-08-12: Unified LLM router at `100.82.40.23:3001` (also `freellmapi.martinwa.org`). Verified 2026-08-12:
+19 -2
View File
@@ -3,8 +3,8 @@ title: Agent Memory
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
confidence: high confidence: high
tags: [concept, memory] tags: [concept, memory]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -20,6 +20,23 @@ How an agent retains facts/context beyond a single conversation turn or session
- **Verified settings:** memory enabled `true`; message char limit `5000`; user-profile char limit `3000`; flush every `6` turns; `auto_sleep: true`; default scope `session`; vector type `int8`; shared surface `data/shared/mnemosyne.db`. - **Verified settings:** memory enabled `true`; message char limit `5000`; user-profile char limit `3000`; flush every `6` turns; `auto_sleep: true`; default scope `session`; vector type `int8`; shared surface `data/shared/mnemosyne.db`.
- **Correction (2026-07-22, per Anthony):** this page previously named Hindsight as a predecessor backend on this box. That was a mix-up with a different Hermes instance — Hindsight has nothing to do with this box. See [[comparisons/memory-backends]] for the full correction. - **Correction (2026-07-22, per Anthony):** this page previously named Hindsight as a predecessor backend on this box. That was a mix-up with a different Hermes instance — Hindsight has nothing to do with this box. See [[comparisons/memory-backends]] for the full correction.
> **Full `memory:` block re-read 2026-10-20 — the summary above is accurate but incomplete.**
> Live values: `memory_enabled: true`, `provider: mnemosyne`, `memory_char_limit: 5000`,
> `user_profile_enabled: true`, `user_char_limit: 3000`, `flush_min_turns: 6`,
> `nudge_interval: 10`. Mnemosyne sub-block: `auto_sleep: true`, `default_scope: session`,
> `vector_type: int8`, `sleep_threshold: 50`, `profile_isolation: true`,
> `shared_surface_path: data/shared/mnemosyne.db`, `shared_surface_read: true`,
> `sync_roles: [user]`, `ignore_patterns: []`.
>
> **New since 2026-07-22 and worth knowing:**
> - `reflect.disabled_for_cron: true`, `reflect.max_calls_per_session: 3`
> - **`skip_contexts: [cron, flush, subagent, background, skill_loop]`** — Mnemosyne capture is
> deliberately skipped in cron, flush, subagent, background and skill-loop contexts. That is
> exactly why cron runs like this one produce no Mnemosyne working memories: the skip is
> configured, not a malfunction.
> - `profile_isolation: true` — memories do not leak across Hermes profiles, which matters now that
> the `ops` profile has been deleted (see [[runbooks/multiplexer-setup]]).
## Model ## Model
- Per-turn conversation content accumulates up to the char limit, then flushes to the Mnemosyne store every 6 turns (rather than every turn) — a batching tradeoff between write frequency and losing very recent context if the process dies mid-batch. - Per-turn conversation content accumulates up to the char limit, then flushes to the Mnemosyne store every 6 turns (rather than every turn) — a batching tradeoff between write frequency and losing very recent context if the process dies mid-batch.
- `scope: session` means recall defaults to the current session rather than a global cross-session pool, unless explicitly widened. - `scope: session` means recall defaults to the current session rather than a global cross-session pool, unless explicitly widened.
+22 -2
View File
@@ -3,8 +3,8 @@ title: Model Routing
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
confidence: high confidence: high
tags: [routing, model, inference] tags: [routing, model, inference]
sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md] sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md]
@@ -20,6 +20,26 @@ How Hermes selects providers, applies combos, and enforces context limits.
- Default model: `freellmapi-auto` - Default model: `freellmapi-auto`
- OmniRoute used for auxiliary roles: vision, skills_hub, approval, mcp, title_generation, web_extract, compression - OmniRoute used for auxiliary roles: vision, skills_hub, approval, mcp, title_generation, web_extract, compression
> **Live `model:` block re-read 2026-10-20 — the default has moved.**
> `model.default` is now **`meituan/longcat-2.5-preview:free`** (provider `nous`), not
> `freellmapi-auto`. The block also carries `base_url`, `persist_switch_by_default` and `api_mode`.
>
> **The five `custom_providers` and their live base URLs:**
>
> | Name | Base URL | Host |
> |---|---|---|
> | `litellm` | `http://litellm:4000/v1` | LiteLLM proxy (m93p CT500) |
> | `guacano` | `http://ollama:11000/v1` | Guanaco on CT205 |
> | `Omniroute:20128` | `http://omniroute:20128/v1` | OmniRoute on m93p CT101 |
> | `command-code` | `https://api.commandcode.ai/provider/v1` | remote |
> | `omni-101` | `http://100.78.63.20:20128` | OmniRoute node `omniroute-1` (direct IP, not a DNS name) |
>
> Note the **two distinct OmniRoute endpoints**: `Omniroute:20128` (DNS, m93p CT101) and
> `omni-101` (raw IP `100.78.63.20:20128`). `omni-101` is the pin on effectively every
> model-calling cron job, and it has been the single point of failure in the 10-14, 10-16/17 and
> 10-19 incidents — its `/api/health` reads `200 {"status":"ok"}` throughout, so **reachability
> probes do not detect that failure class**; only job-level error text does.
## Context Enforcement ## Context Enforcement
- Three layers: model native → combo `data.config.context_length` → Hermes client cap - Three layers: model native → combo `data.config.context_length` → Hermes client cap
- Effective context is the minimum of the three - Effective context is the minimum of the three
+10 -2
View File
@@ -3,8 +3,8 @@ title: Persistent Shells
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
confidence: high confidence: high
tags: [concept, terminal] tags: [concept, terminal]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -23,6 +23,14 @@ A background shell session that survives across separate Hermes turns/requests,
- **Home mode:** auto - **Home mode:** auto
- **Container CPU:** 1 - **Container CPU:** 1
> **Full `terminal:` block re-read 2026-10-20 — the above still holds, plus more.**
> Live values: `backend: local`, `persistent_shell: true`, `timeout: 180`, `home_mode: auto`,
> `container_cpu: 1`, `container_memory: 5120`, `container_disk: 51200`,
> `container_persistent: true`, `lifetime_seconds: 300`,
> `docker_mount_cwd_to_workspace: false`, `docker_run_as_host_user: true`.
> The container-* keys are inert while `backend: local` — the shell is not containerised on this
> box, so `container_memory`/`container_disk` describe a path that isn't taken.
## Why it matters here ## Why it matters here
Enables multi-step CLI workflows (e.g. a long `git` operation, a build, a watch process) to be resumed across turns without Hermes re-establishing context each time. The tradeoff is a shell can be left in a bad state (stuck process, wrong directory) that silently affects the *next* turn — see [[runbooks/troubleshoot-tool-loop]] for the related tool-loop guardrails that catch some of this. Enables multi-step CLI workflows (e.g. a long `git` operation, a build, a watch process) to be resumed across turns without Hermes re-establishing context each time. The tradeoff is a shell can be left in a bad state (stuck process, wrong directory) that silently affects the *next* turn — see [[runbooks/troubleshoot-tool-loop]] for the related tool-loop guardrails that catch some of this.
+16 -2
View File
@@ -3,8 +3,8 @@ title: Tool Calling
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
confidence: high confidence: high
tags: [concept, tool-calling] tags: [concept, tool-calling]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -35,6 +35,20 @@ hard_stop_after:
## Why it matters here ## Why it matters here
This is the mechanism behind [[runbooks/troubleshoot-tool-loop]] — that runbook's "agent loops on one tool" symptom is exactly what `warn_after`/`hard_stop_after` are meant to catch. With hard-stop disabled, a human (or the messaging integrations) is the actual backstop today. This is the mechanism behind [[runbooks/troubleshoot-tool-loop]] — that runbook's "agent loops on one tool" symptom is exactly what `warn_after`/`hard_stop_after` are meant to catch. With hard-stop disabled, a human (or the messaging integrations) is the actual backstop today.
> **Re-verified 2026-10-20 — the `tool_loop_guardrails:` block is byte-for-byte unchanged.**
> Live values in `~/.hermes/config.yaml`: `warnings_enabled: true`, `hard_stop_enabled: false`,
> `warn_after: {exact_failure: 2, same_tool_failure: 3, idempotent_no_progress: 2}`,
> `hard_stop_after: {exact_failure: 5, same_tool_failure: 8, idempotent_no_progress: 5}`.
> 90 days of no drift on this guardrail.
>
> **The 2026-10-18 incident is a *different* loop and the guardrails did not catch it.**
> The degenerate-stream case (see
> [[incidents/2026-10-18-degenerate-stream-canned-substitute]]) was a **model output** repetition
> loop — the *model* repeating itself in the response stream, not a tool call repeating. Hermes's
> degenerate-stream guard cut it and delivered a canned non-answer, but `tool_loop_guardrails`
> (which watches *tool* calls) never fired. Two separate loop-detection mechanisms exist; do not
> assume one covers the other.
## Related ## Related
- [[concepts/persistent-shells]] - [[concepts/persistent-shells]]
- [[runbooks/troubleshoot-tool-loop]] - [[runbooks/troubleshoot-tool-loop]]
+11 -2
View File
@@ -3,8 +3,8 @@ title: Decisions
type: collection type: collection
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
confidence: high confidence: high
tags: [decisions] tags: [decisions]
--- ---
@@ -19,6 +19,15 @@ Stable strategic and config decisions, with evidence and status.
- [[decisions/persistent-shell-enabled]] - [[decisions/persistent-shell-enabled]]
- [[decisions/rollback-steel-to-camofox]] - [[decisions/rollback-steel-to-camofox]]
> **Status sweep 2026-10-20:** all three listed decisions are still `status: active`, but two now
> carry a review note:
> - [[decisions/keep-openrouter-as-fallback]] — its 30-day revisit trigger is **met**; OpenRouter is
> reachable only through the OmniRoute catalog, not as a standalone provider.
> - [[decisions/persistent-shell-enabled]] — still `true`; its `review_after: 0001-01-01` zero-date
> bug is now corrected to a real date.
> - [[decisions/rollback-steel-to-camofox]] — the *engine* is Camoufox as decided, but
> `cloud_provider: steel` remains in the config while the Steel service itself is unreachable.
## History ## History
- **Verified:** Decision pages initialized during wiki setup on 2026-07-22 - **Verified:** Decision pages initialized during wiki setup on 2026-07-22
- **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing - **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing
+24 -2
View File
@@ -3,8 +3,9 @@ title: Keep OpenRouter as fallback
type: decision type: decision
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
review_after: 2026-08-22 # was 0001-01-01 (bug), fixed during 2026-07-22 audit; real trigger is conditional, see Revisit when review_after: 2026-10-20 # was 0001-01-01 (bug), fixed during 2026-07-22 audit; real trigger is conditional, see Revisit when
verified_on: 2026-10-20
confidence: high confidence: high
tags: [routing, provider, decision] tags: [routing, provider, decision]
sources: [] sources: []
@@ -28,6 +29,27 @@ OpenRouter must remain available as a fallback provider, even if a self-hosted r
## Revisit when ## Revisit when
A self-hosted router has operated reliably for at least 30 days. A self-hosted router has operated reliably for at least 30 days.
> **Revisit trigger MET (verified 2026-10-20).** OmniRoute has been the primary router for well over
> 30 days — `custom:omni-101` is the pin on effectively every model-calling cron job, and
> `Omniroute:20128` is a second live OmniRoute endpoint. The estate has **no standalone OpenRouter
> provider**: `custom_providers` in `~/.hermes/config.yaml` holds only `litellm`, `guacano`,
> `Omniroute:20128`, `command-code` and `omni-101` (see [[systems/openrouter]]).
>
> **But the fallback is now provided *through* OmniRoute rather than alongside it** — OpenRouter-style
> slugs (`openrouter/auto`, `freeapi/openrouterfree`, `or-openrouter-fusion`, …) are present in the
> OmniRoute catalog, so OpenRouter capacity remains reachable without a separate Hermes provider.
>
> **Status: decision still `active`, but the mechanism it protects has changed.** The intent
> (OpenRouter capacity must remain reachable as a fallback) is satisfied via the OmniRoute catalog.
> The literal reading — a standalone OpenRouter provider in `custom_providers` — is **not** satisfied
> and has not been since at least 2026-07-22. This needs a re-affirmation or a rewrite of the
> decision text; it should not simply be left to drift on a 30-day timer.
>
> Related context: OmniRoute has also been the source of repeated fleet-wide failures —
> the 2026-10-14 connection-refused storm, the 2026-10-16/17 and 2026-10-19 resource-pressure 503
> bursts, and the 2026-10-18 degenerate-stream delivery. A single-node fallback posture carries
> that concentration risk with it.
## Related ## Related
- [[model-routing]] - [[model-routing]]
- [[systems/omniroute]] - [[systems/omniroute]]
+12 -2
View File
@@ -3,8 +3,9 @@ title: Persistent shell enabled
type: decision type: decision
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
review_after: 0001-01-01 review_after: 2026-10-20
verified_on: 2026-10-20
confidence: high confidence: high
tags: [automation, shell, decision] tags: [automation, shell, decision]
sources: [] sources: []
@@ -24,5 +25,14 @@ Enable persistent shells for Hermes.
## Revisit when ## Revisit when
Use case becomes critical again. Use case becomes critical again.
> **Re-verified 2026-10-20 — decision still `active` and the setting is still `true`.**
> `terminal.persistent_shell: true` in `~/.hermes/config.yaml`, `backend: local`,
> `timeout: 180`, `home_mode: auto`, `container_cpu: 1`, `lifetime_seconds: 300`.
> The `review_after: 0001-01-01` value was a zero-date bug (same class as
> [[decisions/keep-openrouter-as-fallback]]); it is now a real date.
> One caveat worth carrying into the review: `lifetime_seconds: 300` means a persistent shell
> is torn down after 5 minutes idle, so "state persists across turns" is true within a burst of
> work but not indefinitely. See [[concepts/persistent-shells]].
## Related ## Related
- [[persistent-shells]] - [[persistent-shells]]
+17 -2
View File
@@ -3,8 +3,8 @@ title: Rollback Steel to Camoufox
type: decision type: decision
status: active status: active
created: 2026-08-02 created: 2026-08-02
updated: 2026-08-02 updated: 2026-10-20
verified_on: 2026-08-02 verified_on: 2026-10-20
confidence: high confidence: high
tags: [decision, browser, steel, camofox, rollback] tags: [decision, browser, steel, camofox, rollback]
sources: [session-history, live-config] sources: [session-history, live-config]
@@ -28,6 +28,21 @@ Steel Browser was deployed on CT201 on 2026-07-29 as the primary browser automat
## Decision ## Decision
**Revert to Camoufox as primary.** Steel plugin disabled but left running on CT201. **Revert to Camoufox as primary.** Steel plugin disabled but left running on CT201.
> **Status check 2026-10-20 — the decision is *partially* undone in configuration.**
> The live `browser:` block in `~/.hermes/config.yaml` reads
> `cloud_provider: steel` with `engine: camofox`. The **engine** is Camoufox (decision honoured),
> but the **cloud_provider** key still says `steel`, and `camera_provider: camofox`.
> This is the same split seen in [[comparisons/camoufox-vs-chrome]].
>
> Meanwhile the Steel service itself is **not reachable**: `http://steel.martinwa.org` returns
> **000** (connection failed) on 2026-10-20. Combined with the degraded CT201 stack recorded in
> [[systems/homepage-dashboard]] (main container removed, asset proxy 502), it looks like the
> "left running on CT201" part of this decision is no longer true.
>
> **Recommendation, not actioned here:** the `cloud_provider: steel` value appears to be residue
> from the reverted deployment. Confirm against the live Steel service and either restore it or
> set `cloud_provider: camofox` so the config matches the decision this page records.
## Rationale ## Rationale
1. **Reliability:** Camoufox runs on the same host as Hermes (CT450) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues. 1. **Reliability:** Camoufox runs on the same host as Hermes (CT450) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues.
2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports. 2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports.
+26 -5
View File
@@ -3,8 +3,8 @@ title: Experiments
type: collection type: collection
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
confidence: medium confidence: medium
tags: [experiments] tags: [experiments]
--- ---
@@ -15,9 +15,30 @@ Non-canonical tests, tuning passes, and change attempts.
## Known experiments ## Known experiments
- `pre-vision-fix-20260718-195712` config snapshot — Verified, backup file exists, 2026-07-18 - `pre-vision-fix-20260718-195712` config snapshot — **no longer found by that name**
- `pre-model-fix` config snapshot — Verified, backup file exists - `pre-model-fix` config snapshot — **no longer found by that name**
- `pre-camofox-tuning` config snapshot — Verified, backup file exists, 2026-07-20 - `pre-camofox-tuning` config snapshot — **no longer found by that name**
> **Re-verified 2026-10-20 — the three named snapshots this page listed are GONE.**
> `find ~/.hermes/backups -name "*pre-*"` returns only:
> - `config.yaml.pre-cua-durable-20260730` (the one that `pre-camofox-tuning` appears to have
> been renamed to — it is dated 2026-07-30, matching the CUA durable-persistence change)
> - `incident-20260727-mattermost-removal/config.yaml.pre-mattermost-off`
>
> The other two (`pre-vision-fix-20260718-195712`, `pre-model-fix`) have **no surviving file under
> that name** in `~/.hermes/backups/`. The backups directory now holds 15 entries including
> `config/` (a rolling series of `config.yaml.bak.*`, `.corrupt.*`, `.good.*` and
> `config.yaml.premnemo.20260928-174645`) plus nine dated incident directories:
> `dashboard-regression-20260811`, `incident-20260727-mattermost-removal`,
> `legacy-memory-slim-20260801_180702`, `mnemosyne-capture-fix-20260801_180249`,
> `phantom-default-profile-20260730-1250`, `resource-hardening-20260725`,
> `session-repair-20260815-205936`.
>
> The `config/` series is worth knowing about operationally: it includes **four `config.yaml.corrupt.*`
> files** (2026-09-28 ×2, 2026-09-30, 2026-10-01) — evidence of the `hermes config set` stringifying
> lists/dicts problem — alongside `config.yaml.good.*` snapshots from 2026-10-09, 10-11 and 10-16.
> See [[incidents/2026-09-28-hermes-managed-runtime-deps]] and the runtime-deps notes in
> [[systems/hermes-agent]].
## Outcomes ## Outcomes
- **Verified:** Backups exist with the above names - **Verified:** Backups exist with the above names
@@ -3,8 +3,9 @@ title: Mnemosyne Automatic Capture Failure
type: incident type: incident
status: active status: active
created: 2026-08-01 created: 2026-08-01
updated: 2026-08-01 updated: 2026-10-20
verified_on: 2026-08-01 resolved_on: 2026-10-20
verified_on: 2026-10-20
confidence: high confidence: high
tags: [memory, mnemosyne, hermes, incident] tags: [memory, mnemosyne, hermes, incident]
sources: [live-config, mnemosyne-db, hermes-cli-e2e] sources: [live-config, mnemosyne-db, hermes-cli-e2e]
@@ -53,6 +54,34 @@ The already-running Telegram gateway/provider instance was initialized before th
## Monitoring Note ## Monitoring Note
The health script checks capture staleness only after 24 hours. It therefore remained green during this fourteen-hour active-capture outage. A future improvement should compare recent non-cron conversation activity with the newest `working_memory` row rather than lowering the threshold blindly and creating idle-period false positives. The health script checks capture staleness only after 24 hours. It therefore remained green during this fourteen-hour active-capture outage. A future improvement should compare recent non-cron conversation activity with the newest `working_memory` row rather than lowering the threshold blindly and creating idle-period false positives.
## Resolution confirmed (2026-10-20)
The remaining "Activation Step" on this page — restarting the gateway to load the corrected typed
configuration — **has been completed and the config is still correct 50 days later.** Verified live
in `/home/hermes/.hermes/config.yaml` under `memory.mnemosyne`:
| Key | Live value | Was it a string? |
|---|---|---|
| `sync_roles` | `['user']` (real list) | no — now a native list |
| `skip_contexts` | `['cron', 'flush', 'subagent', 'background', 'skill_loop']` | no — native list |
| `reflect` | `{disabled_for_cron: True, max_calls_per_session: 3}` (dict) | no — native dict |
| `sleep_threshold` | `50` (int) | no — int |
| `ignore_patterns` | `[]` (list) | no — native list |
| `profile_isolation` | `True` (bool) | no — bool |
| `auto_sleep` / `default_scope` / `vector_type` | `True` / `session` / `int8` | no |
**None of these are stringified.** The `hermes config set` stringification bug that caused this
incident has not recurred for this block.
**One drift to note:** `skip_contexts` now includes `cron`, which was not in the original repair
list. Mnemosyne capture is therefore **deliberately skipped in cron contexts** — scheduled jobs
produce no working memories by design. That explains why cron-driven runs of this wiki job leave
no Mnemosyne trace.
**Status: `resolved`.** The back-up copies named in the repair section still exist:
`~/.hermes/backups/mnemosyne-capture-fix-20260801_180249/` and
`~/.hermes/backups/legacy-memory-slim-20260801_180702/`.
## Related ## Related
- [[systems/mnemosyne]] - [[systems/mnemosyne]]
- [[systems/memory-backend]] - [[systems/memory-backend]]
+28 -2
View File
@@ -3,8 +3,8 @@ title: "Gmail Read-State Incident (2026-08-29)"
type: incident type: incident
status: active status: active
created: 2026-08-29 created: 2026-08-29
updated: 2026-08-29 updated: 2026-10-20
verified_on: 2026-08-29 verified_on: 2026-10-20
confidence: high confidence: high
tags: [email, gmail, cron, himalaya, incident] tags: [email, gmail, cron, himalaya, incident]
sources: [session:default/20260829_204103_9e1087] sources: [session:default/20260829_204103_9e1087]
@@ -59,6 +59,32 @@ Despite using `--preview`, Himalaya may still mark messages as read depending on
- `goimapnotify` stopped - `goimapnotify` stopped
- Important-mail monitor paused (if it existed) - Important-mail monitor paused (if it existed)
> **Re-verified 2026-10-20 — the mitigation has partially REGRESSED, and the root cause is still
> live.**
>
> | Item | State on 2026-08-29 | State on 2026-10-20 |
> |---|---|---|
> | `goimapnotify.service` | stopped | **ACTIVE again** — `systemctl --user is-active goimapnotify.service` → `active`, PID 383 `/home/hermes/go/bin/goimapnotify -conf /home/hermes/.config/goimapnotify/goimapnotify.yaml` |
> | Important-mail monitor (`d40085da631c`) | paused | **still paused** — `enabled: False` in `jobs.json`, `last_status: ok` |
> | `fetch_newsletters.py` line 49 | patched to `--preview` | **still `himalaya message read --preview --folder INBOX`** — the unsafe call is still in the file, unfixed |
>
> **So the real fix — rewriting `fetch_newsletters.py` to use readonly IMAP (`SELECT readonly` /
> `BODY.PEEK[]`) — has still not been done 52 days later.** The two mitigations that were applied
> are one of them gone (goimapnotify restarted) and one still holding (the important-mail monitor
> remains disabled).
>
> **Other email crons currently in `error` state (2026-10-20 read of `jobs.json`):**
> `85edf6772bca` Tech & AI Newsletter Digest, `08eb86f02d37` Gmail Action Digest,
> `06a0a1c8704f` AI Newsletter Curation for Patrick. Their most recent failures were
> provider-side (OmniRoute 503 resource pressure on 2026-10-19; a connection error on 2026-10-15),
> **not** read-state errors — but the read-state exposure persists regardless, because the
> newsletter fetch path still issues `message read`.
>
> **Recommended action (not taken here — this is a wiki job, and the fix touches Anthony's mail):**
> rewrite `fetch_newsletters.py` to use readonly IMAP, and decide whether `goimapnotify` should be
> running. See [[systems/scheduled-tasks]] and the Gmail IMAP lessons in
> [[runbooks/provider-health]].
**Fix required:** **Fix required:**
- Rewrite `fetch_newsletters.py` to use readonly IMAP (`BODY.PEEK[]`) instead of `himalaya message read` - Rewrite `fetch_newsletters.py` to use readonly IMAP (`BODY.PEEK[]`) instead of `himalaya message read`
- Audit all Gmail-related scripts for similar patterns - Audit all Gmail-related scripts for similar patterns
+17 -2
View File
@@ -3,8 +3,8 @@ title: Backups
type: infrastructure type: infrastructure
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
confidence: medium confidence: medium
tags: [infrastructure, backups] tags: [infrastructure, backups]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -33,6 +33,21 @@ How Hermes's own configuration/state gets backed up, and what's missing.
## Known gaps ## Known gaps
- **No scheduled backup job found:** `crontab -l -u hermes` returns empty, and no systemd timer references `hermes` or `backup`. The one full backup on disk appears to have been triggered manually, not on a schedule. - **No scheduled backup job found:** `crontab -l -u hermes` returns empty, and no systemd timer references `hermes` or `backup`. The one full backup on disk appears to have been triggered manually, not on a schedule.
> **Re-verified 2026-10-20 — the gap is unchanged and the single backup is now 90 days old.**
> - `crontab -l -u hermes` → "no crontab for hermes". `systemctl --user list-timers --all` → **0 timers**.
> The only timer on the box is the system-level `dpkg-db-backup.timer` — nothing Hermes-related.
> - The one full backup is still `/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, dated
> 22 Jul). **No new full backup has been taken in the 90 days since.**
> - `/home/hermes/hermes-wiki/` is still **empty**; `/home/hermes/wiki-personal-backups/` still holds
> only the four original 22 Jul files (`current-state.md`, `index.md`, `log.md`, `SCHEMA.md`).
> - `~/.hermes/scripts/mnemosyne-backup.sh` still exists (unchanged since 3 Jul).
> - `updates:` in `config.yaml` still reads `pre_update_backup: false`, `backup_keep: 5`,
> `non_interactive_local_changes: stash`.
>
> **The separate wiki backup is also unrun:** `/home/hermes/wiki-backups/` does not exist, so the
> tarball step in [[runbooks/backup-wiki]] has never been executed (the runbook's `mkdir -p` would
> create it on first use).
- `pre_update_backup: false` means an in-place `hermes update` will not auto-snapshot first — the manual step in [[runbooks/update-hermes-safely]] step 1 is currently the only safety net. - `pre_update_backup: false` means an in-place `hermes update` will not auto-snapshot first — the manual step in [[runbooks/update-hermes-safely]] step 1 is currently the only safety net.
## Relevant runbooks ## Relevant runbooks
+15 -2
View File
@@ -3,8 +3,8 @@ title: Credentials Map
type: infrastructure type: infrastructure
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
confidence: medium confidence: medium
tags: [credentials, security, infrastructure] tags: [credentials, security, infrastructure]
sources: [] sources: []
@@ -32,6 +32,19 @@ Document where credentials are stored and what service they belong to.
| Huawei Cloud | Terraform state / config | Unverified | | | Huawei Cloud | Terraform state / config | Unverified | |
| GitHub | Hermes config / gh CLI | `~/.config/gh/` or `~/.hermes/` | Unverified | | GitHub | Hermes config / gh CLI | `~/.config/gh/` or `~/.hermes/` | Unverified |
> **Re-verified 2026-10-20 — two corrections to the table above.**
> 1. **Email credentials live in `~/.hermes/.env`, not `config.yaml`.** `config.yaml` has **no**
> `imap`/`smtp`/`email` keys; the live variables are `EMAIL_ADDRESS`, `EMAIL_IMAP_HOST`,
> `EMAIL_SMTP_HOST`, `EMAIL_ALLOWED_USERS`, `EMAIL_HOME_ADDRESS`, `EMAIL_ALLOW_ALL_USERS`
> and friends in `~/.hermes/.env` (mode `0600`). The `config.yaml` row is wrong — point at the `.env`.
> 2. **The `tools:` block in `config.yaml` contains only `tool_search`.** There are no `gmail`,
> `discord`, `mattermost` or `email` tool sections there. Discord and Telegram appear as
> **top-level** `discord:` / `telegram:` blocks, and Mattermost is a **plugin**
> (`platforms/mattermost`), not a `tools:` entry. So the "Hermes config" location column is right
> for those, but the implication that they are `tools:` sections is not.
> No `connectors:` block is configured in `config.yaml` at all.
> `~/.hermes/auth.json` also exists and holds OAuth tokens — treat as a credential store.
## Rules ## Rules
- Credentials never written to this wiki - Credentials never written to this wiki
- API vault is first-stop for any key lookup - API vault is first-stop for any key lookup
+26 -3
View File
@@ -3,8 +3,8 @@ title: Domains and Tunnels
type: infrastructure type: infrastructure
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
confidence: high confidence: high
tags: [infrastructure, network, cloudflare] tags: [infrastructure, network, cloudflare]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -31,7 +31,30 @@ Where `martinwa.org` subdomains point, and how they reach internal services.
| `omniroute.martinwa.org` | `http://omniroute.kangaroo-eel.ts.net:20128` | Public front for [[systems/omniroute]] | | `omniroute.martinwa.org` | `http://omniroute.kangaroo-eel.ts.net:20128` | Public front for [[systems/omniroute]] |
| `nanobot.martinwa.org` / `nano-ssh.martinwa.org` | `100.122.189.81` (ct333 "clawtest") | See [[comparisons/hermes-vs-nanobot]] | | `nanobot.martinwa.org` / `nano-ssh.martinwa.org` | `100.122.189.81` (ct333 "clawtest") | See [[comparisons/hermes-vs-nanobot]] |
`mcp-portal.martinwa.org` is not present in the ct207 ingress list captured during this audit — it may be served by a different tunnel/host, or the ingress list may have moved since. **Unknown**, flagged for follow-up rather than guessed. `mcp-portal.martinwa.org` is **still not present** in the ct207 ingress list — re-read live from
`/etc/cloudflared/config.yml` on ct207 on **2026-10-20**. It may be served by a different
tunnel/host, or the ingress list may have moved since. **Still unknown**, still flagged for
follow-up rather than guessed.
### Ingress changes found 2026-10-20 (full live re-read of ct207 `config.yml`)
The hostname list had drifted from the 2026-07-22 capture. Verified additions and corrections:
| Hostname | Live target | Note |
|---|---|---|
| `hermes-chat.martinwa.org` | `http://100.118.5.51:8787` | **new** — WebUI |
| `tg.hermes.martinwa.org` | `http://100.118.5.51:19119` | **new** — Telegram bot webhook |
| `openwebui.martinwa.org` | `http://100.69.230.13:4000` | **new** — Open WebUI |
| `news.martinwa.org` | `http://docker.kangaroo-eel.ts.net:8300` | **new** — WorldMonitor |
| `ssh-pve.martinwa.org` | `http://100.96.100.82:7681` | **new** — PVE noVNC |
| `nextcloud.martinwa.org` | `https://100.90.102.19:443` | target corrected (was `100.96.150.69`) |
| `readlater.martinwa.org` | `http://100.96.150.69:6123` | confirmed |
| `velvet.martinwa.org` | `http://127.0.0.1:8888` | confirmed |
| `runtipi.martinwa.org` | `http://100.96.244.39:80` | confirmed |
| `homar.martinwa.org` | `http://100.96.244.39:8102` | confirmed — this is **Homarr**, the live dashboard (see [[systems/homepage-dashboard]]) |
The catch-all remains `service: http_status:404`. The ct207 `config.yml` still carries ~25 dated
`config.yml.bak*` files, newest `config.yml.bak.1781809151`.
## Full ct207 ingress list (context, not all Hermes-related) ## Full ct207 ingress list (context, not all Hermes-related)
The tunnel also fronts: `pve.martinwa.org` (Proxmox UI), `docker.martinwa.org` (Portainer), `runtipi.martinwa.org`, `homar.martinwa.org`, `nextcloud.martinwa.org`, `readlater.martinwa.org`, `velvet.martinwa.org`, `overtchat.martinwa.org`, `freellmapi.martinwa.org`, `omnideb.martinwa.org`, `newstarter.martinwa.org`, and SSH endpoints for debianvm/pve/webserver. These belong to the wider homelab, not Hermes specifically — recorded here only so an ingress diff is possible later. The tunnel also fronts: `pve.martinwa.org` (Proxmox UI), `docker.martinwa.org` (Portainer), `runtipi.martinwa.org`, `homar.martinwa.org`, `nextcloud.martinwa.org`, `readlater.martinwa.org`, `velvet.martinwa.org`, `overtchat.martinwa.org`, `freellmapi.martinwa.org`, `omnideb.martinwa.org`, `newstarter.martinwa.org`, and SSH endpoints for debianvm/pve/webserver. These belong to the wider homelab, not Hermes specifically — recorded here only so an ingress diff is possible later.
+110
View File
@@ -0,0 +1,110 @@
# Wiki freshness audit — 2026-10-20
Threshold: >21 days since `updated`
## STALE (47)
- [ 90d] comparisons/camoufox-vs-chrome.md (updated 2026-07-22)
- [ 90d] comparisons/hermes-vs-nanobot.md (updated 2026-07-22)
- [ 90d] concepts/agent-memory.md (updated 2026-07-22)
- [ 90d] concepts/model-routing.md (updated 2026-07-22)
- [ 90d] concepts/persistent-shells.md (updated 2026-07-22)
- [ 90d] concepts/tool-calling.md (updated 2026-07-22)
- [ 90d] decisions/index.md (updated 2026-07-22)
- [ 90d] decisions/keep-openrouter-as-fallback.md (updated 2026-07-22)
- [ 90d] decisions/persistent-shell-enabled.md (updated 2026-07-22)
- [ 90d] experiments/index.md (updated 2026-07-22)
- [ 90d] infrastructure/backups.md (updated 2026-07-22)
- [ 90d] infrastructure/credentials-map.md (updated 2026-07-22)
- [ 90d] infrastructure/domains-and-tunnels.md (updated 2026-07-22)
- [ 90d] runbooks/backup-wiki.md (updated 2026-07-22)
- [ 90d] runbooks/diagnose-docker.md (updated 2026-07-22)
- [ 90d] runbooks/recover-docker-service.md (updated 2026-07-22)
- [ 90d] runbooks/restart-browser.md (updated 2026-07-22)
- [ 90d] runbooks/restart-hermes.md (updated 2026-07-22)
- [ 90d] runbooks/restore-openrouter.md (updated 2026-07-22)
- [ 90d] runbooks/test-openrouter.md (updated 2026-07-22)
- [ 90d] runbooks/troubleshoot-tool-loop.md (updated 2026-07-22)
- [ 90d] systems/cloudflare-tunnel.md (updated 2026-07-22)
- [ 90d] systems/headless-server.md (updated 2026-07-22)
- [ 90d] systems/openrouter.md (updated 2026-07-22)
- [ 90d] systems/terminal-backend.md (updated 2026-07-22)
- [ 87d] runbooks/gateway-resource-resilience.md (updated 2026-07-25)
- [ 87d] systems/tool-search.md (updated 2026-07-25)
- [ 86d] systems/homepage-dashboard.md (updated 2026-07-26)
- [ 86d] systems/obsidian.md (updated 2026-07-26)
- [ 84d] runbooks/multiplexer-setup.md (updated 2026-07-28)
- [ 83d] systems/opencode-go.md (updated 2026-07-29)
- [ 82d] systems/hermes-ui.md (updated 2026-07-30)
- [ 80d] incidents/2026-08-01-mnemosyne-capture-failure.md (updated 2026-08-01)
- [ 79d] decisions/rollback-steel-to-camofox.md (updated 2026-08-02)
- [ 79d] reports/uber-eats/index.md (updated 2026-08-02)
- [ 69d] comparisons/memory-backends.md (updated 2026-08-12)
- [ 69d] comparisons/model-routers.md (updated 2026-08-12)
- [ 67d] systems/guanaco.md (updated 2026-08-14)
- [ 66d] agents/index.md (updated 2026-08-15)
- [ 66d] agents/open-claw.md (updated 2026-08-15)
- [ 66d] agents/zeroclaw.md (updated 2026-08-15)
- [ 66d] incidents/2026-08-15-skills-index-drift.md (updated 2026-08-15)
- [ 65d] CONTRIBUTING.md (updated 2026-08-16)
- [ 64d] systems/nextcloud.md (updated 2026-08-17)
- [ 53d] reports/communities-wa-social-media-audit-2026.md (updated 2026-08-28)
- [ 52d] incidents/2026-08-29-gmail-read-state.md (updated 2026-08-29)
- [ 46d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-09-04)
## FRESH (43)
- [ 16d] agents/hermes-chaos.md
- [ 16d] agents/hermes-production.md
- [ 9d] agents/nanobot.md
- [ 9d] agents/qwenpaw.md
- [ 7d] comparisons/steel-vs-camoufox.md
- [ 7d] concepts/browser-automation.md
- [ 0d] current-state.md
- [ 8d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 17d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 9d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 14d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 16d] incidents/2026-10-03-ct450-hermes-update.md
- [ 16d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 14d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 7d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 11d] incidents/2026-10-09-memory-recall-relevance.md
- [ 4d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 0d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 1d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 9d] infrastructure/docker-services.md
- [ 16d] infrastructure/hosts.md
- [ 6d] infrastructure/network-map.md
- [ 6d] infrastructure/proxmox.md
- [ 1d] infrastructure/tailscale.md
- [ 8d] queries/retained-research-answers.md
- [ 7d] runbooks/job-radar.md
- [ 16d] runbooks/provider-health.md
- [ 16d] runbooks/update-hermes-safely.md
- [ 7d] systems/browser-backend.md
- [ 18d] systems/current-profile.md
- [ 9d] systems/hermes-agent.md
- [ 8d] systems/home-assistant.md
- [ 9d] systems/ignis.md
- [ 16d] systems/memory-backend.md
- [ 1d] systems/messaging-integrations.md
- [ 1d] systems/mnemosyne.md
- [ 19d] systems/model-providers.md
- [ 8d] systems/nanobot.md
- [ 0d] systems/omniroute.md
- [ 9d] systems/paseo.md
- [ 1d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
## NO updated DATE (7)
- SCHEMA.md
- experiments/active/example.md
- incidents/2026-07-29-mnemosyne-relay.md
- incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- index.md
- log.md
- reports/uber-eats/2026-07.md
TOTAL: 47 stale, 43 fresh, 7 no-date of 97 pages
+12 -3
View File
@@ -3,9 +3,9 @@ title: Backup Wiki
type: runbook type: runbook
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
last_tested: 2026-07-22 last_tested: 2026-10-20
confidence: high confidence: high
tags: [runbook, wiki, backup] tags: [runbook, wiki, backup]
sources: [] sources: []
@@ -26,6 +26,15 @@ Create a durable backup of the wiki so recoverable copies exist before mutating
- Git is available - Git is available
- Backup destination is writable - Backup destination is writable
> **Two trees, not one (verified 2026-10-20).** This runbook still backs up `/home/hermes/wiki`
> only — the **Obsidian vault mirror**, which is a **local-only** git repo with **no remote**
> (`git remote -v` is empty). The **authoring tree** that is actually pushed to Gitea
> (`Anthony/agent-estate-wiki`) lives at `/home/hermes/workspace/agent-estate-wiki` and is a
> **separate repo**. A tarball of `/home/hermes/wiki` does **not** protect the authoring tree.
> To protect both, run step 3 against each path, or back up the authoring tree and rely on Gitea.
>
> Also note `/home/hermes/wiki-backups/` does not yet exist — this runbook has never been run.
## Procedure ## Procedure
### 1. Sanity-check repo state ### 1. Sanity-check repo state
+12 -3
View File
@@ -3,9 +3,9 @@ title: Diagnose Failed Docker Service
type: runbook type: runbook
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
last_tested: 2026-07-22 last_tested: 2026-10-20
confidence: medium confidence: medium
tags: [runbook, docker, homelab] tags: [runbook, docker, homelab]
sources: [] sources: []
@@ -26,6 +26,15 @@ Diagnose container or compose issues on hosts where Docker is present.
- Docker daemon running - Docker daemon running
- Compose file path known for the affected project - Compose file path known for the affected project
> **Important scoping note, verified 2026-10-20: the `hermes` user is NOT in the `docker` group on
> CT460.** `systemctl is-active docker` → `active`, but `docker ps` as `hermes` returns
> **`permission denied while trying to connect to the docker API at unix:///var/run/docker.sock`**.
> Most Docker services in this estate run on **other hosts** (runtipi CT201, cloudreve CT209,
> LiteLLM CT500 on m93p, etc.) where the invoking account does have socket access.
> So: on CT460, either use `sudo docker …` or SSH to the host that actually owns the container.
> Note also that `systems/homepage-dashboard.md` now records the CT201 stack as **degraded** —
> the main dashboard container is gone — which is exactly the symptom this runbook would be used for.
## Procedure ## Procedure
### 1. Check Docker daemon state ### 1. Check Docker daemon state
+31 -7
View File
@@ -3,9 +3,9 @@ title: Hermes Gateway Resource Resilience
type: runbook type: runbook
status: active status: active
created: 2026-07-25 created: 2026-07-25
updated: 2026-07-25 updated: 2026-10-20
review_after: 2026-10-23 review_after: 2026-11-19
verified_on: 2026-07-25 verified_on: 2026-10-20
version_applies_to: Proxmox CT460 version_applies_to: Proxmox CT460
confidence: high confidence: high
tags: [hermes, infrastructure, runbook] tags: [hermes, infrastructure, runbook]
@@ -50,12 +50,22 @@ File:
```ini ```ini
[Service] [Service]
MemoryHigh=1G MemoryHigh=2G
MemoryMax=1536M MemoryMax=3G
MemorySwapMax=768M MemorySwapMax=1G
OOMPolicy=kill OOMPolicy=kill
``` ```
> **⚠ Limits have been RAISED since 2026-07-25 (verified 2026-10-20).** The Camofox drop-in now
> reads `MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G` — the page previously recorded
> `1G` / `1536M` / `768M`. Effective values confirmed via
> `systemctl --user show camofox-browser.service`:
> `MemoryHigh=2147483648` (2 GiB), `MemoryMax=3221225472` (3 GiB),
> `MemorySwapMax=1073741824` (1 GiB), `OOMPolicy=kill`.
> The `OOMPolicy=kill` intent is unchanged; only the ceilings moved. The Camofox service **is**
> running (`camofox-browser.service`, active/running) — it is the browser workload the containment
> is meant to protect, and it is the unit that grew.
### Hermes Web UI ### Hermes Web UI
File: File:
@@ -72,6 +82,20 @@ OOMPolicy=kill
Camofox and Web UI are allowed to restart under extreme growth instead of Camofox and Web UI are allowed to restart under extreme growth instead of
consuming most of the container and taking Telegram down with them. consuming most of the container and taking Telegram down with them.
> **Re-verified 2026-10-20 — the gateway and Web UI controls are still exactly as documented.**
> - Gateway drop-in `/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf`
> still reads `OOMPolicy=continue`, `MemoryLow=512M`, `CPUWeight=1000` (no `MemoryHigh`/`MemoryMax`).
> Effective: `OOMPolicy=continue`, `CPUWeight=1000`, `MemoryLow=536870912`,
> `MemoryHigh=infinity`, `MemoryMax=infinity` — **matches the "Expected gateway values" note below.**
> - Web UI: `OOMPolicy=kill`, `MemoryHigh=1073741824` (1 GiB), `MemoryMax=1610612736` (1.5 GiB),
> `MemorySwapMax=1073741824` (1 GiB) — **matches the drop-in shown above, unchanged.**
> - Camofox: see the raised limits noted above.
> - `hermes-gateway.service` and `camofox-browser.service` are both **active (running)**.
>
> **Note:** the Web UI drop-in lives at `/etc/systemd/system/hermes-webui.service.d/` which the
> `hermes` user cannot read (permission denied) — its *effective* values are still verifiable via
> `systemctl show`, which is what the health-check section below uses.
## I/O priority limitation ## I/O priority limitation
Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation
@@ -102,7 +126,7 @@ runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
Expected gateway values include `OOMPolicy=continue`, Expected gateway values include `OOMPolicy=continue`,
`MemoryLow=536870912`, `MemoryHigh=infinity`, `MemoryMax=infinity`, and `MemoryLow=536870912`, `MemoryHigh=infinity`, `MemoryMax=infinity`, and
`CPUWeight=1000`. `CPUWeight=1000`. **Confirmed live on 2026-10-20** — see the note under "Active controls".
## Backups and rollback ## Backups and rollback
+24 -2
View File
@@ -3,8 +3,8 @@ title: Multiplexer Setup (Multi-Profile Gateway)
type: runbook type: runbook
status: active status: active
created: 2026-07-28 created: 2026-07-28
updated: 2026-07-28 updated: 2026-10-20
verified_on: 2026-07-28 verified_on: 2026-10-20
confidence: high confidence: high
tags: [hermes, multiplexer, telegram, profiles, gateway] tags: [hermes, multiplexer, telegram, profiles, gateway]
--- ---
@@ -67,6 +67,28 @@ platforms:
enabled: false # ← DISABLED — multiplexer handles this enabled: false # ← DISABLED — multiplexer handles this
``` ```
## ⚠ The `ops` profile was DELETED on 2026-10-11 (verified 2026-10-20)
`/home/hermes/.hermes/profiles/` now contains **only** `.deleted/ops` (an 8-byte marker file whose
entire content is the word `deleted`, dated 2026-10-11). **`~/.hermes/profiles/ops/` no longer
exists** — there is no `config.yaml`, no `SOUL.md`, no profile directory.
**But the default profile still routes to it.** `~/.hermes/config.yaml` still carries both
`profile_routes` entries (`ops-group` → `-1003914987043`, `ops-group-2` → `-1003932503629`,
both `profile: ops`), and `multiplex_profiles: true` is still set. The ops SOUL (Tusk, the African
elephant) and its `allowed_chats` list are gone with the directory.
**Consequence:** messages arriving in the two ops chats have a route pointing at a profile that no
longer exists. Treat the "Ops profile" blocks below as **historical** — the second profile's
config can no longer be read at `~/.hermes/profiles/ops/config.yaml` because that path is gone.
**Two things need a decision (not actioned here):**
1. Should `profile_routes` be removed, or should the `ops` profile be recreated?
2. Is the `multiplex_profiles: true` + two-route configuration still needed at all?
Until that is settled, treat the ops chat IDs as **possibly unroutable** and verify the gateway
actually resolves them before relying on this runbook.
## The Silent-Drop Pitfall ## The Silent-Drop Pitfall
The `allowed_chats` gate runs on the **default profile's adapter** BEFORE `profile_routes` stamps `source.profile`. If a chat is only authorized in the secondary profile's config but not the default's, messages from that chat are silently dropped. The `allowed_chats` gate runs on the **default profile's adapter** BEFORE `profile_routes` stamps `source.profile`. If a chat is only authorized in the secondary profile's config but not the default's, messages from that chat are silently dropped.
+15 -1
View File
@@ -3,7 +3,8 @@ title: Recover a Docker service
type: runbook type: runbook
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-10-20
tags: [runbook, docker, recovery] tags: [runbook, docker, recovery]
sources: [] sources: []
confidence: high confidence: high
@@ -18,3 +19,16 @@ confidence: high
Container shows healthy after recovery probe reboots. Container shows healthy after recovery probe reboots.
## Last successfully used ## Last successfully used
2026-07-22 2026-07-22
> **Verified 2026-10-20 (procedure, not a live recovery).** Before running any `docker` command
> from CT460, note that the **`hermes` user has no access to `/var/run/docker.sock`** —
> `docker ps` returns `permission denied while trying to connect to the docker API`. Use `sudo`,
> or SSH to the host that owns the container. The containers in this estate mostly live on
> **CT201 (runtipi)**, **CT209 (cloudreve)** and **m93p CT500 (LiteLLM)** — not on CT460.
>
> **Two live situations this runbook would have been used for in the last week:**
> - The **CT201 homepage dashboard** is degraded — main container removed, `asset_proxy.py` on 8085
> proxying to a dead 8082, and the Tailscale `/` route returning 502. See
> [[systems/homepage-dashboard]].
> - The **Steel browser** service is unreachable (`steel.martinwa.org` → connection failure).
> See [[decisions/rollback-steel-to-camofox]].
+15 -6
View File
@@ -3,9 +3,9 @@ title: Restart Browser Helper
type: runbook type: runbook
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
last_tested: 2026-07-22 last_tested: 2026-10-20
confidence: high confidence: high
tags: [runbook, browser, cua-driver, camofox] tags: [runbook, browser, cua-driver, camofox]
sources: [] sources: []
@@ -22,8 +22,12 @@ Restart the computer-use browser stack that Hermes uses for web automation, scre
- VNC/noVNC is unreachable on `5901` / `6081` - VNC/noVNC is unreachable on `5901` / `6081`
## Prerequisites ## Prerequisites
- Display helper already running: `Xvfb :99`, x11vnc on `127.0.0.1:5901`, noVNC at `127.0.0.1:6081` - Display helper already running: `Xvfb :99`, x11vnc on `0.0.0.0:5901`, noVNC at `127.0.0.1:6081`
- `cua-driver` binary installed at `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` - `cua-driver` binary installed — **the release directory has moved on.** The path this runbook
used to hardcode (`.../releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver`) no longer exists;
`/home/hermes/.cua-driver/packages/releases/` now holds `0.19.3`, `0.22.2`, `0.23.2`, `0.24.0`
and **`0.28.2`**, and `/home/hermes/.local/bin/cua-driver` reports **cua-driver 0.28.2**.
Prefer the stable `/home/hermes/.local/bin/cua-driver` path over a versioned release path.
- Browser VM/desktop helper is expected but not mandatory - Browser VM/desktop helper is expected but not mandatory
## Procedure ## Procedure
@@ -72,7 +76,12 @@ pkill -f "fluxbox"
- The browser backend path can work without browser automation; this stack is optional depending on task - The browser backend path can work without browser automation; this stack is optional depending on task
## Last tested ## Last tested
2026-07-22 2026-07-22 — procedure re-checked against live processes 2026-10-20. `Xvfb :99`, `x11vnc -display :99
-forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw`, `fluxbox` and both
websockify instances (`6080→5900`, `6081→5901`) are running, and the `vnc-watcher.sh` helper
(`/home/hermes/services/camofox-browser/plugins/vnc/vnc-watcher.sh`) is up. Note **5900 has no
listener** — the 6080 pair points at a dead port (Camofox's front door); the live display chain is
`:99 → 5901 → 6081`. See [[systems/headless-server]].
## Related ## Related
- [[browser-backend]] - [[browser-backend]]
+25 -7
View File
@@ -3,9 +3,9 @@ title: Restart Hermes
type: runbook type: runbook
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
last_tested: 2026-07-22 last_tested: 2026-10-20
confidence: high confidence: high
tags: [runbook, hermes, restart] tags: [runbook, hermes, restart]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -32,17 +32,35 @@ Restart the Hermes agent, gateway, dashboard, and browser helper processes clean
```bash ```bash
ps aux | grep -E "hermes_cli.main (gateway|dashboard) run|hermes-webui/server.py|cua-driver serve|x11vnc.*5901" | grep -v grep ps aux | grep -E "hermes_cli.main (gateway|dashboard) run|hermes-webui/server.py|cua-driver serve|x11vnc.*5901" | grep -v grep
``` ```
Expected active pieces on this install: Expected active pieces on this install (live `ps` 2026-10-20):
- gateway: `/home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run` - gateway: `hermes gateway run` → `/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 … gateway run`
- dashboard: `/home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main dashboard ...` (unit `hermes-gateway.service`)
- webui: `/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py` - dashboard: `hermes dashboard --host 0.0.0.0 --port 9119 --no-open` (unit `hermes-dashboard.service`)
- webui: `/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py` (port 8787)
- camofox/cua-driver: `/home/hermes/.local/bin/cua-driver serve` - camofox/cua-driver: `/home/hermes/.local/bin/cua-driver serve`
- x11vnc: `-display :99 ... 5901` - x11vnc: `-display :99 ... 5901`
> **The `venv/` path in this runbook is wrong for the gateway and dashboard (verified 2026-10-20).**
> They no longer run from `~/.hermes/hermes-agent/venv/`; they run from the **uv-managed runtime**
> `~/.hermes/tools/python-3.14.7+20260901-linux-x64/`, launched through
> `hermes_cli/main.py` with a `runpy` bootstrap. The `hermes` launcher at
> `/home/hermes/.local/bin/hermes` is a 73-byte shim. **Prefer the `hermes` CLI and the systemd
> user units over hand-written `python -m hermes_cli.main …` commands** — the units already carry
> the correct interpreter path and the `_hermes_closure_repair()` bootstrap that protects against a
> half-finished `hermes update`.
>
> **Live user units (2026-10-20):** `hermes-gateway.service`, `hermes-dashboard.service`,
> `hermes-ui.service`, `hermes-ui-prod.service`, `hermes-wiki-serve.service` — all
> `active (running)`.
### 2. Restart the gateway ### 2. Restart the gateway
```bash ```bash
hermes gateway restart hermes gateway restart
``` ```
Equivalent and preferred (survives a broken CLI shim):
```bash
systemctl --user restart hermes-gateway.service
```
### 3. Restart the dashboard if needed ### 3. Restart the dashboard if needed
```bash ```bash
+21 -2
View File
@@ -3,7 +3,8 @@ title: Restore OpenRouter connectivity
type: runbook type: runbook
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-10-20
tags: [runbook, openrouter, recovery] tags: [runbook, openrouter, recovery]
sources: [] sources: []
confidence: high confidence: high
@@ -19,9 +20,27 @@ confidence: high
3. Restart the Hermes process with the GoOp tool. 3. Restart the Hermes process with the GoOp tool.
4. Send a small test completion request. 4. Send a small test completion request.
5. Confirm tool calling works. 5. Confirm tool calling works.
> **⚠ This runbook is out of step with the current install (flagged 2026-10-20).**
> Steps 2 and 3 reference **WARP** and a **"GoOp tool"** — neither exists on this box. There is no
> WARP client configured and no `GoOp` binary; `hermes` has no such subcommand. The steps are
> carried over from a different install and **were not executed during the 2026-10-20 review**.
>
> **The procedure that actually applies here:**
> 1. Confirm OpenRouter is not a standalone provider in the first place — `custom_providers` in
> `~/.hermes/config.yaml` holds only `litellm`, `guacano`, `Omniroute:20128`, `command-code` and
> `omni-101`. See [[systems/openrouter]].
> 2. If the failure is a *fallback* failure, use [[runbooks/test-openrouter]] and check the six
> entries of the live chain.
> 3. Restart with `systemctl --user restart hermes-gateway.service` (not "the GoOp tool").
> 4. Send a small test completion and confirm tool calling works.
>
> **Last successfully used: 2026-06-22** — which predates every configuration referenced on this
> page. Treat the page as unverified until someone runs it end to end.
## Verification ## Verification
A test prompt returns successfully through OpenRouter. A test prompt returns successfully through OpenRouter.
## Escalation ## Escalation
If the failure persists, check DNS, certificate expiry and provider status. If the failure persists, check DNS, certificate expiry and provider status.
## Last successfully used ## Last successfully used
2026-06-22 2026-06-22 — **120 days ago.** This is the single oldest verification date still in the wiki
alongside the original 2026-07-22 page set.
+27 -11
View File
@@ -3,9 +3,9 @@ title: Test OpenRouter
type: runbook type: runbook
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
last_tested: 2026-07-22 last_tested: 2026-10-20
confidence: high confidence: high
tags: [runbook, openrouter, providers, fallback] tags: [runbook, openrouter, providers, fallback]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -22,7 +22,8 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar
- User wants to confirm OpenRouter is usable without relying on catalog names - User wants to confirm OpenRouter is usable without relying on catalog names
## Prerequisites ## Prerequisites
- Hermes CLI available: `/home/hermes/.local/bin/hermes` - Hermes CLI available: `hermes` on PATH (`/home/hermes/.local/bin/hermes` is a shim to the
uv-managed runtime — see the note under **Last tested**)
- Internet egress for opencode.ai, inference-api.nousresearch.com, ollama.com, kilo.ai, litellm:4000 - Internet egress for opencode.ai, inference-api.nousresearch.com, ollama.com, kilo.ai, litellm:4000
## Procedure ## Procedure
@@ -31,12 +32,22 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar
```bash ```bash
hermes fallback list hermes fallback list
``` ```
Expected active fallbacks on this install: Expected active fallbacks on this install (live `hermes fallback list`, 2026-10-20 — **6** entries):
1. `deepseek-v4-flash-free` via opencode-zen 1. `mimo-v2.5` via opencode-go
2. `stepfun/step-3.7-flash:free` via nous 2. `deepseek-v4-flash-free` via opencode-zen — `https://opencode.ai/zen/v1`
3. `qwen3.5:397b` via ollama-cloud 3. `stepfun/step-3.7-flash:free` via nous — `https://inference-api.nousresearch.com/v1`
4. `poolside/laguna-m.1:free` via kilocode 4. `qwen3.5:397b` via ollama-cloud — `https://ollama.com/v1`
5. `or-inclusionai_ling-2_6-1t` via custom/litellm 5. `poolside/laguna-m.1:free` via kilocode — `https://api.kilo.ai/api/gateway`
6. `or-inclusionai_ling-2_6-1t` via custom — `http://litellm:4000/v1`
**Primary:** `meituan/longcat-2.5-preview:free` (via nous).
> **Changes since 2026-07-22:** `mimo-v2.5` via **opencode-go** has been **added at position 1** of the
> chain, and the primary changed to `meituan/longcat-2.5-preview:free` (was `freellmapi-auto`).
> The other five entries and their order are unchanged.
> `hermes fallback list` also prints a standing notice on every invocation:
> *"Claude Subscription DirectSDK (Experimental): Claude Code is not installed"* — expected on this
> box, since `claude` is not on PATH. It is a notice, not a failure of the fallback chain.
### 2. Test OpenRouter-style direct access if present ### 2. Test OpenRouter-style direct access if present
```bash ```bash
@@ -77,7 +88,12 @@ Verify provider names and URLs in config match what the user wants to use.
- This runbook tests confirmed fallback paths, not assumed ones - This runbook tests confirmed fallback paths, not assumed ones
## Last tested ## Last tested
2026-07-22 2026-07-22 — re-run 2026-10-20 against live `hermes fallback list` output (6 entries, primary
`meituan/longcat-2.5-preview:free`). Note the CLI reports Hermes Agent **v0.22.2 (2026.9.24),
local `d923a5eb487a`**, and the `/home/hermes/.local/bin/hermes` launcher is a 73-byte shim that
delegates to the uv-managed runtime at
`/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/` — *not* the
`~/.hermes/hermes-agent/venv/` path named in some older runbooks. Use the `hermes` on PATH.
## Related ## Related
- [[provider-health]] - [[provider-health]]
+88 -13
View File
@@ -3,24 +3,99 @@ title: Troubleshoot Tool Loop
type: runbook type: runbook
status: unresolved status: unresolved
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
review_after: 2026-08-22 review_after: 2026-11-19
confidence: low verified_on: 2026-10-20
confidence: medium
tags: [runbook, hermes] tags: [runbook, hermes]
sources: [] sources: []
--- ---
> **Stub — structure only.** Headings exist but content has not been verified/written yet. Flagged during 2026-07-22 wiki audit; fill in with verified facts or mark deprecated if no longer relevant. > **Filled in 2026-10-20** from live configuration, live process state and two recorded incidents.
> The "stub" note below is retained as history.
# Troubleshoot Tool Loop # Troubleshoot Tool Loop
## Symptoms ## Symptoms
Agent loops on one tool or endpoint. - The agent repeats the **same tool call** with identical arguments and makes no progress.
## Procedure - The agent repeats **itself in the response stream** (no tool involved) — the model loops on its
1. Inspect recent Hermes agent logs. own output and Hermes cuts the stream.
2. Identify the looping tool path.
3. Restart worker or flush stuck request. **These are two different faults with two different detectors.** Do not assume one covers the other.
4. Re-run limited test scenario.
## Verification ## 1. Tool-call loop — detected by `tool_loop_guardrails`
Agent handles the same scenario without repeating.
### Symptoms
Agent loops on one tool or endpoint. Hermes emits a warning naming the tool.
### Configuration (verified live 2026-10-20, `~/.hermes/config.yaml`)
```yaml
tool_loop_guardrails:
warnings_enabled: true
hard_stop_enabled: false
warn_after:
exact_failure: 2
same_tool_failure: 3
idempotent_no_progress: 2
hard_stop_after:
exact_failure: 5
same_tool_failure: 8
idempotent_no_progress: 5
```
> **`hard_stop_enabled: false` is the key fact.** Hermes will *warn* about a stuck tool loop but
> will **not** force-stop itself. A genuinely stuck loop only gets a warning — a human (or the
> messaging integration) is the real backstop. See [[concepts/tool-calling]].
### Procedure
1. Inspect recent Hermes agent logs: `~/.hermes/logs/agent.log*` (grep for the tool name).
2. Identify the looping tool path. Common culprits in this estate: a browser tool against a dead
Camofox service, a terminal tool against a stale persistent shell, or a webhook that returns
200 with an empty body.
3. **Check the dependency, not just the agent.** On this estate the most frequent tool-loop cause
was an upstream service being down while the tool still returned a response. Verify with the
relevant runbook: [[runbooks/restart-browser]], [[runbooks/restart-hermes]],
[[runbooks/provider-health]].
4. If the loop is a stuck worker, restart it: `systemctl --user restart hermes-gateway.service`.
5. Re-run a limited test scenario and confirm the guard no longer fires.
### Verification
Agent handles the same scenario without repeating, and no `tool_loop_guardrails` warning appears
in `agent.log`.
## 2. Model-output (degenerate stream) loop — detected by the stream guard
### Symptoms
The run **delivers a canned non-answer** such as *"Your request was not processed. Send it again
if you still want me to carry it out."*, and then fails with
`RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding
the degenerate response.`
### This is the 2026-10-18 case — see
[[incidents/2026-10-18-degenerate-stream-canned-substitute]]
### Procedure
1. Read the run's `error` in `~/.hermes/cron/executions.db` — the phrase `repetition loop`
identifies this fault.
2. **Check `delivery_outcome` on the same row.** In the 2026-10-18 incident the row was
`delivery_outcome: delivered` — the canned non-answer reached Telegram *before* the run
failed. **A delivered message is not evidence of delivered content.**
3. Confirm the provider is healthy: the 2026-10-18 case rode the same `custom:omni-101` alias
that produced the 2026-10-14/15 connection-refused storm and the 2026-10-16/17 and 2026-10-19
resource-pressure 503 bursts. See [[runbooks/provider-health]] and
[[incidents/2026-10-16-omniroute-503-resource-pressure]].
4. Do **not** lower `tool_loop_guardrails` thresholds for this fault — they watch tool calls, and
the degenerate-stream guard is a separate mechanism.
### Verification
The next scheduled run of the job completes with `status: completed` and `error: null`, and the
delivered message contains real content.
## Last successfully used ## Last successfully used
2026-07-22 2026-07-22 — re-verified against live config and live incidents on 2026-10-20; the *procedure* has
not itself been executed end-to-end in a live loop, so `confidence` was raised only to `medium`.
## Related
- [[concepts/tool-calling]]
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
- [[runbooks/provider-health]]
- [[runbooks/restart-hermes]]
+9 -3
View File
@@ -3,8 +3,8 @@ title: Cloudflare Tunnel
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
confidence: high confidence: high
tags: [system, network, cloudflare] tags: [system, network, cloudflare]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -26,6 +26,12 @@ Exposes internal homelab services, including Hermes, to the public internet with
- `originRequest`: `noTLSVerify: true`, `disableChunkedEncoding: true`, `http2Origin: false` (global defaults; some hostnames override per-entry) - `originRequest`: `noTLSVerify: true`, `disableChunkedEncoding: true`, `http2Origin: false` (global defaults; some hostnames override per-entry)
- Ingress is a flat list of `hostname → service` mappings ending in `service: http_status:404` catch-all — see [[infrastructure/domains-and-tunnels]] for the full list and the entries specific to Hermes. - Ingress is a flat list of `hostname → service` mappings ending in `service: http_status:404` catch-all — see [[infrastructure/domains-and-tunnels]] for the full list and the entries specific to Hermes.
- Two Hermes-relevant entries point at this host's Tailscale IP (`100.118.5.51`): `hermes.martinwa.org` (port 19119) and `hermes-desktop.martinwa.org` (port 5174). - Two Hermes-relevant entries point at this host's Tailscale IP (`100.118.5.51`): `hermes.martinwa.org` (port 19119) and `hermes-desktop.martinwa.org` (port 5174).
- **Re-verified 2026-10-20:** `cloudflared.service` is still **active** on ct207, the tunnel name/ID
(`ct207-tunnel` / `339a6757-5574-4723-a16c-da4769535caa`) and the credentials-file path are unchanged,
and the global `originRequest` defaults still read `noTLSVerify: true`, `disableChunkedEncoding: true`,
`http2Origin: false`. The Hermes-relevant ingress entries have since **expanded to four**:
`hermes.martinwa.org` (9119), `hermes-chat.martinwa.org` (8787), `tg.hermes.martinwa.org` (19119)
and `hermes-desktop.martinwa.org` (5174). Full live list in [[infrastructure/domains-and-tunnels]].
## Connected systems ## Connected systems
- [[systems/hermes-agent]] — the service actually reached through this tunnel - [[systems/hermes-agent]] — the service actually reached through this tunnel
@@ -34,7 +40,7 @@ Exposes internal homelab services, including Hermes, to the public internet with
## Known issues ## Known issues
- This host cannot restart or inspect the tunnel directly (no `cloudflared` installed here); a tunnel-side problem requires access to ct207. - This host cannot restart or inspect the tunnel directly (no `cloudflared` installed here); a tunnel-side problem requires access to ct207.
- `mcp-portal.martinwa.org`, referenced in Hermes's own config as an MCP endpoint, was not found in the ct207 ingress list captured 2026-07-22 — **unresolved**, may be on a different tunnel/host. - `mcp-portal.martinwa.org`, referenced in Hermes's own config as an MCP endpoint, was not found in the ct207 ingress list captured 2026-07-22 — **unresolved**, may be on a different tunnel/host. Re-checked against the live ct207 `config.yml` on 2026-10-20: **still absent**, still unresolved.
## Relevant runbooks ## Relevant runbooks
- [[runbooks/diagnose-docker]] (closest existing runbook; no dedicated tunnel-restart runbook exists yet — worth writing one that documents `systemctl restart cloudflared` on ct207, since Hermes has no direct access to perform it) - [[runbooks/diagnose-docker]] (closest existing runbook; no dedicated tunnel-restart runbook exists yet — worth writing one that documents `systemctl restart cloudflared` on ct207, since Hermes has no direct access to perform it)
+8 -4
View File
@@ -3,8 +3,8 @@ title: Guanaco
type: system type: system
status: active status: active
created: 2026-08-14 created: 2026-08-14
updated: 2026-08-14 updated: 2026-10-20
verified_on: 2026-08-14 verified_on: 2026-10-20
confidence: high confidence: high
tags: [llm, proxy, router, guanaco, ollama] tags: [llm, proxy, router, guanaco, ollama]
sources: [live-probe, session-history] sources: [live-probe, session-history]
@@ -17,14 +17,18 @@ LLM proxy/router providing model routing and tool-call formatting for the Ollama
## Current Configuration ## Current Configuration
- **Host:** CT205 (`ollama` LXC container) on Proxmox - **Host:** CT205 (`ollama` LXC container) on Proxmox
- **Version:** v0.8.14 (updated from v0.8.11 on 2026-08-14) - **Version:** v0.8.15 (updated from v0.8.14 on 2026-08-14; verified at tag `v0.8.15`, commit `5df435e`, 2026-08-14 16:06:13 UTC)
- **Repo path (on CT205):** `/root/.guanaco/repo` - **Repo path (on CT205):** `/root/.guanaco/repo`
- **API endpoint:** `ollama:11434` (DNS) / `localhost:11434` - **API endpoint:** `ollama:11434` (DNS) / `localhost:11434`
- **Key component:** `cmdcode_client.py` — Guanaco's tool-call formatting layer (DSML) - **Key component:** `cmdcode_client.py` — Guanaco's tool-call formatting layer (DSML)
## Version History ## Version History
### v0.8.14 (current, updated 2026-08-14) ### v0.8.15 (current, updated 2026-08-14)
- **Changelog bump + DSML fixes** — HEAD is tagged `v0.8.15` (commit `5df435e`, "chore: bump to v0.8.15 + changelog"), built from `origin/main` / `origin/develop`.
- Live probe 2026-10-20: `/root/.guanaco/repo` on CT205 at tag `v0.8.15`; Ollama API on `localhost:11434` answers **200**.
### v0.8.14 (previous, 2026-08-14)
- **DSML tag leakage fixes** (cumulative across v0.8.12–v0.8.14): - **DSML tag leakage fixes** (cumulative across v0.8.12–v0.8.14):
- v0.8.12 — incomplete DSML blocks leaking raw tags as content - v0.8.12 — incomplete DSML blocks leaking raw tags as content
- v0.8.13 — comprehensive DSML tag leakage: double-pipe, bare fragments, solo `invoke` - v0.8.13 — comprehensive DSML tag leakage: double-pipe, bare fragments, solo `invoke`
+13 -3
View File
@@ -3,8 +3,8 @@ title: Headless Server
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
confidence: high confidence: high
tags: [system, browser] tags: [system, browser]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -27,7 +27,17 @@ Headless X11 environment backing Hermes's browser automation and any GUI tooling
| `x11vnc` | `-display :99 -forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw` — **no VNC password set** | | `x11vnc` | `-display :99 -forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw` — **no VNC password set** |
| `websockify` ×2 | `127.0.0.1:6080 → 127.0.0.1:5900` and `127.0.0.1:6081 → 127.0.0.1:5901`, serving noVNC (`/usr/share/novnc`) | | `websockify` ×2 | `127.0.0.1:6080 → 127.0.0.1:5900` and `127.0.0.1:6081 → 127.0.0.1:5901`, serving noVNC (`/usr/share/novnc`) |
Two VNC/websockify pairs exist (5900/6080 and 5901/6081) — the second (`:99`→5901→6081) is the one confirmed above; the first (5900→6080) predates it and wasn't independently re-verified this pass — **unresolved** whether it's a second live display or a leftover process. **Re-verified 2026-10-20.** Two websockify instances still run (6080→5900 and 6081→5901), but
**only one Xvfb exists** (`:99`). Port **5900 has no listener at all** — the 6080 websockify points
at a dead port, so it is a **leftover** from an earlier display, not a second live one. The 6080
pair is the **Camofox** noVNC front door (`/home/hermes/services/camofox-browser/plugins/vnc/vnc-watcher.sh`),
which proxies to 5900; Camofox's REST API is separately live on `:9377` (HTTP 200).
The live chain is therefore **`:99` → x11vnc 5901 → websockify 6081**.
Also confirmed still true in 2026-10-20: `x11vnc` still runs with `-nopw` (no VNC password) and
still binds **all interfaces** — `0.0.0.0:5901` and `[::]:5901` — while both websockify ports
are correctly loopback-only (`127.0.0.1:6080`, `127.0.0.1:6081`).
## Connected systems ## Connected systems
- [[systems/hermes-agent]] (browser backend uses this display — see [[systems/browser-backend]] / [[comparisons/camoufox-vs-chrome]]) - [[systems/hermes-agent]] (browser backend uses this display — see [[systems/browser-backend]] / [[comparisons/camoufox-vs-chrome]])
+8 -2
View File
@@ -3,8 +3,8 @@ title: hermes-ui (Browser Hermes Desktop)
type: system type: system
status: live status: live
created: 2026-07-30 created: 2026-07-30
updated: 2026-07-30 updated: 2026-10-20
verified_on: 2026-07-30 verified_on: 2026-10-20
confidence: high confidence: high
tags: [hermes, webui, desktop, browser] tags: [hermes, webui, desktop, browser]
--- ---
@@ -25,6 +25,12 @@ The official Hermes Desktop renderer, extracted from the `hermes-agent` monorepo
| CT460 (this box) | `/home/hermes/hermes-ui/` | 5174 | Vite dev server | Active | | CT460 (this box) | `/home/hermes/hermes-ui/` | 5174 | Vite dev server | Active |
| CT450 (hermesagent) | `/home/hermes/hermes-ui/` | 5174 | Vite dev server | Active (since 2026-07-30) | | CT450 (hermesagent) | `/home/hermes/hermes-ui/` | 5174 | Vite dev server | Active (since 2026-07-30) |
> **Re-verified 2026-10-20 — both deployments are up.** CT460: `0.0.0.0:5174` held by a `node`
> process; CT450: `0.0.0.0:5174` confirmed by `ss -ltn` inside CT450, and `/home/hermes/hermes-ui`
> exists there. Note **5175** on CT460 is a *different* service — a `python3` on `127.0.0.1:5175`
> (the `/ui` Tailscale Serve route) — and **8787** is the Hermes WebUI (`python`, `0.0.0.0:8787`,
> the `/webui` route). Don't conflate 5174 / 5175 / 8787.
## How It Works ## How It Works
- Runs as a Vite dev server, proxying REST/auth/WebSocket to a running Hermes gateway (`http://127.0.0.1:9119` by default). - Runs as a Vite dev server, proxying REST/auth/WebSocket to a running Hermes gateway (`http://127.0.0.1:9119` by default).
+72 -42
View File
@@ -1,9 +1,10 @@
--- ---
title: Homepage Dashboard title: Homepage Dashboard
type: systems type: systems
status: active status: degraded
created: 2026-07-26 created: 2026-07-26
updated: 2026-07-26 updated: 2026-10-20
verified_on: 2026-10-20
confidence: high confidence: high
tags: [dashboard, homepage, homelab, proxmox, monitoring] tags: [dashboard, homepage, homelab, proxmox, monitoring]
--- ---
@@ -13,18 +14,35 @@ tags: [dashboard, homepage, homelab, proxmox, monitoring]
## Purpose ## Purpose
Central dashboard for all homelab services, with Proxmox integration, per-VM stats, and a management UI. Central dashboard for all homelab services, with Proxmox integration, per-VM stats, and a management UI.
## URLs ## ⚠ Status: DEGRADED (verified 2026-10-20)
- **Tailscale:** `https://runtipi.kangaroo-eel.ts.net:8443/` The stock **`homepage` container is not running** on CT201. Only `homepage-manager` (port 8084)
- **Local:** `http://runtipi.kangaroo-eel.ts.net:8082/` and `homebox` are up; `homebridge` exited two months ago. **Port 8082 has no listener**, and the
- **Manager UI:** `http://runtipi.kangaroo-eel.ts.net:8084/` `asset_proxy.py` front door on port 8085 proxies to 8082 — so it answers **502 Connection refused**.
## Architecture The dashboard that *is* live and serving **200** is **Homarr** on port **8102**
(`homar.martinwa.org` in the Cloudflare tunnel ingress, and `homarr` in the Runtipi app list).
The dashboard content itself is now **custom Python tooling** under `/opt/homepage/`
(`manager.py`, `asset_proxy.py`, `dashboard.html/js/css`), not the upstream Homepage image.
**Two independent dashboards are therefore in play — do not assume "the dashboard" means Homepage.**
## URLs
- **Homarr (live, 200):** `http://100.96.244.39:8102/` — public via `https://homar.martinwa.org`
- **Tailscale:** `https://runtipi.kangaroo-eel.ts.net:8443/` → proxies to 8085 → **currently 502**
- **Local:** `http://100.96.244.39:8082/` → **no listener**
- **Manager UI:** `http://runtipi.kangaroo-eel.ts.net:8084/` → `homepage-manager`, up
## Architecture (verified 2026-10-20)
``` ```
CT201 (runtipi) CT201 (runtipi)
├── Homepage container (port 8082) — main dashboard ├── homarr container (port 8102) — live dashboard, HTTP 200
├── Manager container (port 8084) — web UI for adding/removing apps ├── homepage-manager container (8084) — web UI for adding/removing apps, up
└── Tailscale serve (port 8443) — HTTPS proxy to Homepage ├── homebox container (7745) — up
├── asset_proxy.py (python3, port 8085) — proxies 8085 → 8082, currently 502
├── manager.py (python3) — dashboard management service, up
└── Tailscale serve (port 8443) — / → 8085, /mgr → 8084, /steel → 5173, /steel-api → 3000
``` ```
No container is listening on **8082** — `docker ps` on CT201 lists no `homepage` container at all.
## Tabs ## Tabs
| Tab | Content | | Tab | Content |
@@ -44,39 +62,43 @@ CT201 (runtipi)
- **proxmox.yaml:** Config at `/opt/homepage/config/proxmox.yaml` - **proxmox.yaml:** Config at `/opt/homepage/config/proxmox.yaml`
### VM/LXC Mapping ### VM/LXC Mapping
| Service | VMID | Type | > **Two Proxmox hosts exist and guest IDs collide** — always qualify with a host.
|---|---|---| > `ct460` = PVE `100.96.100.82`; `m93p` = PVE `100.104.51.98`. The mapping below was re-read from
| PVE | (node) | — | > `pct list` on **both** hosts on 2026-10-20; several rows previously listed bare IDs now live on
| Portainer docker | 200 | lxc | > `m93p`, and one row no longer exists at all.
| Portainer debianVM | 101 | qemu |
| Portainer openweb | 245 | lxc | | Service | Host | VMID | Type | Live state |
|---|---|---|---|---|
| PVE | both | (node) | — | — |
| Portainer docker | ct460 | 200 | lxc | running (8000, 9443) |
| Portainer debianVM | ct460 | 101 | qemu | running |
| Portainer openweb | **m93p** | 245 | lxc | running |
| Runtipi | ct460 | 201 | lxc | running |
| Gitea | **m93p** | 221 | lxc | running |
| FreshRSS | **m93p** | **1260** | lxc | running |
| NextCloud | ct460 | 270 | lxc | running |
| LiteLLM | **m93p** | 500 | lxc | running |
| OmniRoute | ct460 | 600 | lxc | running (also `Omniroute:20128` on m93p 101) |
| Ntfy | **m93p** | **1241** | lxc | running |
| Home Assistant | ct460 | 100 | qemu | running |
| N8n | ct460 | 215 | lxc | running |
| Mattermost | ct460 | 255 | lxc | **stopped** |
| SFTPGo | ct460 | 299 | lxc | running |
| Cliproxy API | ct460 | 103 | lxc | running |
| OpenClaw | — | — | qemu | **absent — no VM 403 on either host** |
| Zeroclaw | ct460 | 333 | lxc | running |
| Hermes | ct460 | 460 | lxc | running |
| Hindsight | ct460 | 450 | lxc | running |
> Portainer on CT245 (`openweb.kangaroo-eel.ts.net:9443`) now uses a Tailscale/Let's Encrypt cert (installed 2026-08-09). Files: `/root/portainer-certs/portainer.{crt,key}` mounted into the container, `--sslcert/--sslkey`. Renewal: `/root/renew-portainer-cert.sh` via crontab (04:00 on the 1st monthly) — `tailscale cert --min-validity=20d` then `docker restart portainer`. Cert validity ~90 days. > Portainer on CT245 (`openweb.kangaroo-eel.ts.net:9443`) now uses a Tailscale/Let's Encrypt cert (installed 2026-08-09). Files: `/root/portainer-certs/portainer.{crt,key}` mounted into the container, `--sslcert/--sslkey`. Renewal: `/root/renew-portainer-cert.sh` via crontab (04:00 on the 1st monthly) — `tailscale cert --min-validity=20d` then `docker restart portainer`. Cert validity ~90 days.
| Runtipi | 201 | lxc |
| Gitea | 221 | lxc |
| FreshRSS | 260 | lxc |
| NextCloud | 270 | lxc |
| LiteLLM | 500 | lxc |
| OmniRoute | 600 | lxc |
| Ntfy | 241 | lxc |
| Home Assistant | 100 | qemu |
| N8n | 215 | lxc |
| Mattermost | 255 | lxc |
| SFTPGo | 299 | lxc |
| Cliproxy API | 103 | lxc |
| OpenClaw | 403 | qemu |
| Zeroclaw | 333 | lxc |
| Hermes | 460 | lxc |
| Hindsight | 450 | lxc |
## Configuration Files ## Configuration Files
All at `/opt/homepage/config/` on CT201: `/opt/homepage/` on CT201 now holds **custom Python tooling**, not the upstream Homepage bundle:
- `services.yaml` — All apps, groups, and Proxmox mappings - `manager.py` — dashboard management service (running)
- `settings.yaml` — Theme, layout, tabs - `asset_proxy.py` — port 8085 → 8082 front door (running, currently 502 upstream)
- `custom.css` — Warm amber accent overrides - `dashboard.html` / `dashboard.js` / `dashboard.css` — the custom dashboard frontend
- `widgets.yaml` — System health, clock - `config/` — legacy Homepage config directory (still present, upstream `homepage` container not running)
- `proxmox.yaml` — PVE API token - `manager.py.bak.*` — a long tail of dated manager backups (brand, chat2, cookie, hp, motion, owtools …)
- `docker.yaml` — Docker socket integration
## Manager UI ## Manager UI
Web form at port 8084 for adding/removing apps without SSH. Web form at port 8084 for adding/removing apps without SSH.
@@ -96,7 +118,15 @@ Type anywhere on the dashboard to search services by name/description. Google se
Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead. Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead.
## Maintenance ## Maintenance
- **Restart Homepage:** `docker restart homepage`
- **Restart Manager:** `docker restart homepage-manager` - **Restart Manager:** `docker restart homepage-manager`
- **Edit apps manually:** Edit `/opt/homepage/config/services.yaml` then restart - **Restart Homarr:** `docker restart homarr`
- **Docker auto-discovery:** Add `homepage.group`, `homepage.name`, `homepage.href` labels to containers - **Restart the custom dashboard:** the `manager.py` / `asset_proxy.py` processes run directly on CT201
(not in Docker) — restart via whatever supervises them; check `ps aux | grep -E 'manager.py|asset_proxy'`
- **Recover the upstream Homepage (8082):** the image `ghcr.io/gethomepage/homepage` is still present on
CT201 (4 months old) but no container uses it — recreate it to restore 8082 and the 8085 proxy path
- **Tailscale serve on CT201:** `/` → 8085, `/mgr` → 8084, `/steel` → 5173, `/steel-api` → 3000
## Recent Changes
- **2026-10-20 (live probe):** `homepage` container absent, 8082 dead, 8085 → 502, Homarr on 8102 → 200.
Status changed `active` → `degraded`. VM/LXC table re-read from both PVE hosts; corrected IDs and
host assignments, dropped the non-existent OpenClaw VM 403, marked Mattermost CT255 stopped.
+10 -2
View File
@@ -3,8 +3,8 @@ title: Nextcloud
type: system type: system
status: active status: active
created: 2026-08-17 created: 2026-08-17
updated: 2026-08-17 updated: 2026-10-20
verified_on: 2026-08-17 verified_on: 2026-10-20
confidence: high confidence: high
tags: [nextcloud, file-sharing, self-hosted, infrastructure] tags: [nextcloud, file-sharing, self-hosted, infrastructure]
sources: [] sources: []
@@ -28,6 +28,14 @@ Self-hosted file sync and sharing platform running on Proxmox LXC container.
|---|---|---|---| |---|---|---|---|
| memories | 8.1.0 | active | Photo management | | memories | 8.1.0 | active | Photo management |
| metadata | 0.24.0 | active | | | metadata | 0.24.0 | active | |
> **Re-verified 2026-10-20** — every fact on this page still holds: CT270 `nextcloud` is running,
> `occ status` reports **version 33.0.2.2** (installed, not in maintenance, no DB upgrade needed),
> the container holds `192.168.178.56` (LAN) and `100.90.102.19` (Tailscale), and both
> `memories 8.1.0` and `metadata 0.24.0` are enabled. `https://nextcloud.martinwa.org/` answers
> **302** both publicly and via the Tailscale IP. `occ` lives at `/var/www/nextcloud/occ`
> (not `/var/www/html/nextcloud`).
> Note: `occ` must be run from that path and as `www-data`.
| ncdownloader | 1.0.23 | **disabled** | Incompatible with NC33 — calls removed method `OC\Server::getAppConfig()` | | ncdownloader | 1.0.23 | **disabled** | Incompatible with NC33 — calls removed method `OC\Server::getAppConfig()` |
| news | 28.7.0 | active | RSS reader | | news | 28.7.0 | active | RSS reader |
| nextcloud_announcements | 5.0.0 | active | | | nextcloud_announcements | 5.0.0 | active | |
+17 -1
View File
@@ -3,7 +3,8 @@ title: Obsidian Vault
type: systems type: systems
status: active status: active
created: 2026-07-26 created: 2026-07-26
updated: 2026-07-26 updated: 2026-10-20
verified_on: 2026-10-20
confidence: high confidence: high
tags: [obsidian, notes, vault, homelab] tags: [obsidian, notes, vault, homelab]
--- ---
@@ -19,6 +20,21 @@ Personal knowledge base — notes, research, projects, and agent documentation.
- **Structure:** NUMBERED folders (00-99) - **Structure:** NUMBERED folders (00-99)
- **Sync:** Git-based (manual or cron-triggered) - **Sync:** Git-based (manual or cron-triggered)
> **Re-verified 2026-10-20 — the "NUMBERED folders (00-99)" description is no longer accurate.**
> On disk there are now only two vault-shaped directories on CT460:
> - `/home/hermes/obsidian-vault/` — contains `50 - Research/` (with `Restaurants and Cafes/`) and
> `Cron Outputs/` (with `Weekend Adventure/`). **No `00 - Home`, no `10 - Inbox`, no `20 - Agent Notes`.**
> This directory is **not a git repo** (`git -C` → "not a git repository").
> - `/home/hermes/wiki/` — a **local-only** git repo (no remote configured) whose tracked content is
> the `agent-estate-wiki/` mirror plus `reports/uber-eats/`. Its `.obsidian/` dir is under
> `wiki/agent-estate-wiki/.obsidian`.
>
> So the numbered-folder tree described below is **largely absent**, and the vault that *is* mirrored
> to Gitea is the **agent-estate-wiki** repo (`Anthony/agent-estate-wiki`), not `Anthony/obsidian-vault`.
> Treat the folder table as historical. Separately, an **Obsidian LiveSync** service runs on **CT100
> (`obsidian-livesync`) on the `m93p` host** (`/opt/obsidian-livesync/`, holds only a `.env`).
> Full folder table, left for reference:
## Folder Structure ## Folder Structure
| Folder | Purpose | | Folder | Purpose |
|---|---| |---|---|
+8 -2
View File
@@ -3,8 +3,8 @@ title: OpenCode Go
type: system type: system
status: active status: active
created: 2026-07-29 created: 2026-07-29
updated: 2026-07-29 updated: 2026-10-20
verified_on: 2026-07-29 verified_on: 2026-10-20
confidence: high confidence: high
tags: [opencode, provider, models, subscription, limits] tags: [opencode, provider, models, subscription, limits]
sources: [https://opencode.ai/docs/go/] sources: [https://opencode.ai/docs/go/]
@@ -22,6 +22,12 @@ Low cost subscription for open coding models — $5 first month, then **$10/mont
> Only one member per workspace can subscribe. > Only one member per workspace can subscribe.
> **Note (2026-10-20):** the pricing and dollar-limit tables below are **vendor-published figures
> for the OpenCode Go product**, not a measurement of this estate. Nothing in `~/.hermes/config.yaml`
> or `jobs.json` shows an OpenCode Go provider in use — the estate routes through LiteLLM,
> OmniRoute (`omni-101`) and `Omniroute:20128`. Treat this page as reference material for the
> subscription, not as a description of a configured provider.
## Usage Limits ## Usage Limits
| Window | Dollar Limit | | Window | Dollar Limit |
+10 -2
View File
@@ -3,8 +3,8 @@ title: OpenRouter
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
confidence: medium confidence: medium
tags: [system, openrouter, provider] tags: [system, openrouter, provider]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -21,6 +21,14 @@ Primary key, fallback models, rate limits.
OpenRouter is not configured as a standalone provider in `~/.hermes/config.yaml` `custom_providers`. OpenRouter is not configured as a standalone provider in `~/.hermes/config.yaml` `custom_providers`.
OmniRoute’s catalog includes provider names and model IDs consistent with OpenRouter-style slugs, but live routing via OpenRouter directly is not present in current Hermes config. OmniRoute’s catalog includes provider names and model IDs consistent with OpenRouter-style slugs, but live routing via OpenRouter directly is not present in current Hermes config.
> **Re-verified 2026-10-20 — the claim above still holds.** `custom_providers` in
> `/home/hermes/.hermes/config.yaml` is a list of exactly five entries:
> `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101`. There is **no `openrouter`
> entry**. OpenRouter-style slugs (`openrouter/auto`, `freeapi/openrouterfree`,
> `kilo-openrouter-auto`, `or-openrouter-fusion`, and many more) exist only inside the **OmniRoute
> model catalog**, not as directly-routable Hermes providers. Routing to OpenRouter therefore still
> goes through OmniRoute, exactly as recorded here.
## History ## History
- **User-provided:** Previously used; now largely replaced by LiteLLM and OmniRoute paths - **User-provided:** Previously used; now largely replaced by LiteLLM and OmniRoute paths
- **Verified:** Decision page `keep-openrouter-as-fallback` exists, indicating retain intent - **Verified:** Decision page `keep-openrouter-as-fallback` exists, indicating retain intent
+2 -2
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-19 updated: 2026-10-20
verified_on: 2026-10-19 verified_on: 2026-10-20
confidence: high confidence: high
tags: [skills, hermes] tags: [skills, hermes]
sources: [] sources: []
+8 -3
View File
@@ -3,8 +3,8 @@ title: Terminal Backend
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-20
verified_on: 2026-07-22 verified_on: 2026-10-20
confidence: high confidence: high
tags: [terminal, shell, backend, persistence] tags: [terminal, shell, backend, persistence]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -17,12 +17,17 @@ Execute shell commands, scripts, and interactive sessions for Hermes.
## Current Configuration ## Current Configuration
- **Backend:** local - **Backend:** local
- **Working directory:** `/home/hermes/workspace` - **Working directory:** `/home/hermes/workspace/`
- **Timeout:** 180 s - **Timeout:** 180 s
- **Home mode:** auto - **Home mode:** auto
- **Container CPU:** 1 - **Container CPU:** 1
- **Persistent shell:** true - **Persistent shell:** true
Live `terminal:` block re-read from `/home/hermes/.hermes/config.yaml` on 2026-10-20 also carries:
`lifetime_seconds: 300`, `container_memory: 5120`, `container_disk: 51200`,
`container_persistent: true`, `docker_mount_cwd_to_workspace: false`, `docker_run_as_host_user: true`.
`backend: local` and `persistent_shell: true` are unchanged since 2026-07-22.
## Persistence ## Persistence
- Shell state (venv, env vars, aliases) persists across terminal calls - Shell state (venv, env vars, aliases) persists across terminal calls
- Working directory fixed to `/home/hermes/workspace` - Working directory fixed to `/home/hermes/workspace`
+14 -6
View File
@@ -3,9 +3,9 @@ title: Hermes Tool Search
type: system type: system
status: active status: active
created: 2026-07-25 created: 2026-07-25
updated: 2026-07-25 updated: 2026-10-20
review_after: 2026-10-23 review_after: 2026-11-20
verified_on: 2026-07-25 verified_on: 2026-10-20
version_applies_to: Hermes Agent on CT460 version_applies_to: Hermes Agent on CT460
confidence: high confidence: high
tags: [hermes, automation, system] tags: [hermes, automation, system]
@@ -32,11 +32,19 @@ File: `/home/hermes/.hermes/config.yaml`
tools: tools:
tool_search: tool_search:
enabled: "on" enabled: "on"
threshold_pct: 10 threshold_pct: 20
search_default_limit: 5 search_default_limit: 8
max_search_limit: 20 max_search_limit: 50
listing_max_tokens: 8
``` ```
> **Live config re-read 2026-10-20** — the values above are the current ones in
> `/home/hermes/.hermes/config.yaml`. They have drifted from the original 2026-07-25 rollout:
> `threshold_pct` 10 → **20**, `search_default_limit` 5 → **8**, `max_search_limit` 20 → **50**,
> and `listing_max_tokens: 8` has been added. `enabled: "on"` is unchanged.
> The token-budget figures in "Verified result" below were measured under the *original* settings
> and have not been re-measured since the tuning.
`enabled: "on"` is the important setting: it makes Tool Search active on every `enabled: "on"` is the important setting: it makes Tool Search active on every
request instead of waiting for the tool schemas to cross an automatic token request instead of waiting for the tool schemas to cross an automatic token
threshold. threshold.
+56 -12
View File
@@ -142,21 +142,65 @@ python3 "$STOCKTAKE_DIR/_freshness_audit.py" "$WIKI_DIR" "$REPORT" 21
echo "" echo ""
# ============================================================================= # =============================================================================
# [3/3] Rebuild Quartz static site # [3/3] Rebuild Quartz static site and deploy to public/
# ============================================================================= # =============================================================================
echo "[3/3] Checking Quartz build..." # Quartz v4/v5 lives in a SEPARATE source tree (/home/hermes/workspace/quartz-temp);
# Quartz is already built into public/ — if there's a build script, run it # the wiki repo only holds the Markdown. The build reads content/ from that tree
# For now, the public/ dir is the Quartz output, regenerated when content changes # and emits public/ there, which we then deploy into the wiki repo's public/.
# If Quartz CLI is available, we'd run: npx quartz build #
if command -v npx &> /dev/null && [ -f "$WIKI_DIR/quartz.config.ts" ]; then # PITFALL (hit 2026-10-20): the previous version of this step tested for
echo " Running Quartz build..." # `quartz.config.ts` and `npx`, and the repo has NEITHER — Quartz v4/v5 uses
cd "$WIKI_DIR" # `quartz.config.yaml`. The test always failed, so the site silently stopped
npx quartz build 2>&1 | head -20 || echo " Quartz build had issues (non-fatal)" # being rebuilt (published site was 40 days stale). Do not reintroduce a
else # config-file existence test that isn't the real one: check quartz.config.yaml.
echo " Quartz build skipped (no quartz.config.ts or npx not available)" echo "[3/3] Rebuilding Quartz static site..."
echo " public/ directory contains the current static site"
QUARTZ_DIR="/home/hermes/workspace/quartz-temp"
QUARTZ_CONTENT="$QUARTZ_DIR/content"
QUARTZ_PUBLIC="$QUARTZ_DIR/public"
QUARTZ_DEPLOY_DIR="$WIKI_DIR/public"
if [ ! -f "$QUARTZ_DIR/quartz.config.yaml" ]; then
echo " ERROR: $QUARTZ_DIR/quartz.config.yaml not found — cannot build the Quartz site." >&2
echo " The published site will stay stale. Fix the Quartz source tree before continuing." >&2
exit 1
fi fi
# --- sync wiki Markdown into the Quartz content tree -------------------------
# raw/ is excluded on purpose: it holds raw config dumps and the stocktake
# reports, which have never been part of the published site. public/ and .git
# are excluded to avoid copying build output into the input tree.
rm -rf "$QUARTZ_CONTENT"
mkdir -p "$QUARTZ_CONTENT"
tar -C "$WIKI_DIR" \
--exclude=./public --exclude=./.git --exclude=./node_modules --exclude=./raw \
-cf - . | tar -C "$QUARTZ_CONTENT" -xf -
echo " Synced wiki content into $QUARTZ_CONTENT"
# --- build -------------------------------------------------------------------
cd "$QUARTZ_DIR"
# PIPESTATUS is required: a plain pipeline would report tail's exit code (0)
# and silently swallow a failed build.
npx quartz build 2>&1 | tail -8
BUILD_RC=${PIPESTATUS[0]}
if [ "$BUILD_RC" -ne 0 ]; then
echo " ERROR: Quartz build failed (exit $BUILD_RC)." >&2
exit 1
fi
# --- deploy ------------------------------------------------------------------
# Clear the CONTENTS of public/ rather than removing the directory itself:
# hermes-wiki-serve.service runs with its cwd inside public/, so replacing the
# directory inode leaves the running server serving a deleted directory (all
# requests 404) until it is restarted.
find "$QUARTZ_DEPLOY_DIR" -mindepth 1 -delete
cp -a "$QUARTZ_PUBLIC/." "$QUARTZ_DEPLOY_DIR/"
echo " Deployed $(find "$QUARTZ_DEPLOY_DIR" -type f | wc -l) files to $QUARTZ_DEPLOY_DIR"
systemctl --user restart hermes-wiki-serve.service 2>/dev/null \
&& echo " Restarted hermes-wiki-serve.service" \
|| echo " WARNING: could not restart hermes-wiki-serve.service — verify the site manually"
echo "" echo ""
echo "[wiki-maintain] Maintenance complete at $(date -Iseconds)" echo "[wiki-maintain] Maintenance complete at $(date -Iseconds)"
echo " Freshness report: $REPORT" echo " Freshness report: $REPORT"