From fbd4f297eb28d3454c2250cd4cf5d906e362747f Mon Sep 17 00:00:00 2001 From: Tony0410 <47945770+Tony0410@users.noreply.github.com> Date: Tue, 20 Oct 2026 18:38:11 +0800 Subject: [PATCH] wiki: publish pending edits (2026-10-20) --- CONTRIBUTING.md | 9 +- agents/index.md | 11 +- agents/open-claw.md | 14 ++- agents/zeroclaw.md | 13 +- comparisons/camoufox-vs-chrome.md | 23 +++- comparisons/hermes-vs-nanobot.md | 14 ++- comparisons/memory-backends.md | 17 ++- comparisons/model-routers.md | 46 +++++-- concepts/agent-memory.md | 21 +++- concepts/model-routing.md | 24 +++- concepts/persistent-shells.md | 12 +- concepts/tool-calling.md | 18 ++- decisions/index.md | 13 +- decisions/keep-openrouter-as-fallback.md | 26 +++- decisions/persistent-shell-enabled.md | 14 ++- decisions/rollback-steel-to-camofox.md | 19 ++- experiments/index.md | 31 ++++- .../2026-08-01-mnemosyne-capture-failure.md | 33 ++++- incidents/2026-08-29-gmail-read-state.md | 30 ++++- infrastructure/backups.md | 19 ++- infrastructure/credentials-map.md | 17 ++- infrastructure/domains-and-tunnels.md | 29 ++++- raw/stocktake/freshness-20261020.txt | 110 +++++++++++++++++ runbooks/backup-wiki.md | 15 ++- runbooks/diagnose-docker.md | 15 ++- runbooks/gateway-resource-resilience.md | 38 ++++-- runbooks/multiplexer-setup.md | 26 +++- runbooks/recover-docker-service.md | 16 ++- runbooks/restart-browser.md | 21 +++- runbooks/restart-hermes.md | 32 +++-- runbooks/restore-openrouter.md | 23 +++- runbooks/test-openrouter.md | 38 ++++-- runbooks/troubleshoot-tool-loop.md | 101 ++++++++++++++-- systems/cloudflare-tunnel.md | 12 +- systems/guanaco.md | 12 +- systems/headless-server.md | 16 ++- systems/hermes-ui.md | 10 +- systems/homepage-dashboard.md | 114 +++++++++++------- systems/nextcloud.md | 12 +- systems/obsidian.md | 18 ++- systems/opencode-go.md | 10 +- systems/openrouter.md | 12 +- systems/skills-index.md | 4 +- systems/terminal-backend.md | 11 +- systems/tool-search.md | 20 ++- wiki-maintain.sh | 70 +++++++++-- 46 files changed, 1012 insertions(+), 197 deletions(-) create mode 100644 raw/stocktake/freshness-20261020.txt diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 16dc0e5..81a3e04 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -3,8 +3,8 @@ title: Contributing to the Agent Estate Wiki type: guide status: active created: 2026-08-16 -updated: 2026-08-16 -verified_on: 2026-08-16 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [wiki, contributing, governance, multi-agent] sources: [] @@ -19,7 +19,10 @@ sources: [] The single source of truth for operational knowledge across the agent estate: - **Hermes production** (CT460), **Hermes Chaos** (CT450) -- **Nanobot** (CT333), **ZeroClaw** (CT333), **QwenPaw** (CT465), **Open-Claw** (VM 403) +- **Nanobot** (CT333), **ZeroClaw** (CT333), **QwenPaw** (CT465) +- ~~**Open-Claw** (VM 403)~~ — **removed from this list 2026-10-20.** No VM 403 exists on either + Proxmox host (PVE `100.96.100.82` or m93p `100.104.51.98`) and there is no `open-claw` Tailscale + peer. See [[agents/open-claw]]. - Shared infrastructure (Proxmox, Tailscale, OmniRoute/LiteLLM, Docker, domains) **One rule above all: this wiki documents the estate. It does not replace any agent's own configuration, memory, or logs.** Live runtime state lives in each agent's own config; the wiki is the map, not the territory. diff --git a/agents/index.md b/agents/index.md index fcd64e5..dccec4f 100644 --- a/agents/index.md +++ b/agents/index.md @@ -3,7 +3,8 @@ title: Agent Estate type: system status: active created: 2026-08-15 -updated: 2026-08-15 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [agents, estate, overview] --- @@ -31,6 +32,14 @@ The agent estate consists of multiple Hermes instances, a QwenPaw deployment, Ze Live probe on 2026-10-19 (`pct list` + `qm list` on PVE host 100.96.100.82) shows **no VM 403 and no VM 401**. The QEMU VMs on the host are 100, 101, 110, 282, 310. The `open-claw` and `zeroclaw` pages still document the intended configuration (from 2026-08-15), but the VM is currently absent — treat `agents/open-claw.md` as **stale/unverified** until the VM is recreated or a new location is documented. +**Extended to the second PVE host on 2026-10-20.** `m93p` (`100.104.51.98`) was probed as well and +also has **no VM 403** (its LXC guests are 100, 101, 102, 104, 105, 221, 245, 500, 998, 1006, 1007, +1008, 1241, 1260, 1261, 1262, 1263, 1264). The `tailscale status` peer list contains **no +`open-claw` node** either. Open-Claw is therefore absent from the whole estate, not just relocated. + +**ZeroClaw, by contrast, was re-verified live on 2026-10-20** and **is running** on CT333 +(`/usr/local/bin/zeroclaw daemon --allow-degraded-security`, PID 136) — see [[agents/zeroclaw]]. + ## Shared Infrastructure All agents share: diff --git a/agents/open-claw.md b/agents/open-claw.md index d84c8b5..b37cdb6 100644 --- a/agents/open-claw.md +++ b/agents/open-claw.md @@ -3,8 +3,8 @@ title: Open-Claw type: system status: active created: 2026-08-15 -updated: 2026-08-15 -verified_on: 2026-08-15 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [openclaw, open-claw, vm403, gateway, agent] sources: [skill-inspection, config-reference] @@ -29,6 +29,16 @@ Autonomous AI agent gateway running on a separate QEMU VM. Docker-based with Tai ## Current State - **Status: NOT PRESENT.** Live probe 2026-10-19: no VM 403 on PVE host 100.96.100.82 (`qm list` shows only VMs 100, 101, 110, 282, 310; `pct list` shows CTs 103, 200, 201, 205, 207, 209, 215, 255, 270, 299, 333, 450, 460, 465, 475, 555, 600, 601). The VM was last known active on 2026-08-15. The content below documents the intended configuration and is **stale/unverified** until the VM is recreated or relocated. +- **Re-probed on the SECOND Proxmox host too, 2026-10-20.** The `m93p` host (`100.104.51.98`) has + its own guest list — LXC 100 `obsidian-livesync`, 101 `omniroute`, 102 `vaultwarden`, 104 `rclone`, + 105 `conduit`, 221 `gitea`, 245 `openweb`, 500 `litellm`, 1006 `freellmapi`, 1007 `homebridge`, + 1008 `rustdesk`, 1241 `ntfy`, 1260 `freshrss`, 1261 `paseo`, 1262 `mailflow`, + 1263 `social-dashboard`, 1264 `openmausbot`, 998 `nightbridge` (stopped) — and **no VM 403 there + either**. The Tailscale peer list likewise contains **no `open-claw` node**. So Open-Claw is not + merely moved between hosts: it is gone from both. +- **Consequence for the rest of this page:** the Tailscale IP `100.64.172.34`, the LAN IP + `192.168.178.193` and the control UI `https://open-claw.kangaroo-eel.ts.net` are all historical. + Treat the whole page as a design record, not a description of a running system. - **Config format:** JSON - **Auth:** Password-based (not paired tokens) - **Software:** Docker-based (community-scripts template) diff --git a/agents/zeroclaw.md b/agents/zeroclaw.md index d2e6a4b..037844e 100644 --- a/agents/zeroclaw.md +++ b/agents/zeroclaw.md @@ -3,8 +3,8 @@ title: ZeroClaw type: system status: active created: 2026-08-15 -updated: 2026-08-15 -verified_on: 2026-08-15 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [zeroclaw, ct333, daemon, agent] sources: [skill-inspection, config-reference] @@ -31,6 +31,15 @@ Autonomous AI daemon running on CT333. Handles independent agent operations with - **Workspace:** `/root/.zeroclaw/workspace` - **State:** `/root/.zeroclaw/state/` +> **Re-verified 2026-10-20 — running as documented, with one path nuance.** +> The daemon **is** live on CT333: `pgrep -af zeroclaw` → `136 /usr/local/bin/zeroclaw daemon +> --allow-degraded-security` (PID 136, up since boot). Note the process runs +> **`/usr/local/bin/zeroclaw`**, which is a **symlink → `/root/.cargo/bin/zeroclaw`** — a *third* +> path alongside the `/root/.local/bin/zeroclaw` binary named above (that file also exists, 37 MB). +> The `--allow-degraded-security` flag was **not** recorded on this page previously; it is part of +> the live command line. +> `/root/.zeroclaw/` contains `config.toml` plus ~15 dated `config.toml.bak*` snapshots. + ## Key Paths | Item | Path | diff --git a/comparisons/camoufox-vs-chrome.md b/comparisons/camoufox-vs-chrome.md index baf49fe..9c4c099 100644 --- a/comparisons/camoufox-vs-chrome.md +++ b/comparisons/camoufox-vs-chrome.md @@ -3,8 +3,8 @@ title: Camoufox vs Chrome type: comparison status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: medium tags: [comparison, browser] sources: [raw/configs/hermes-config-sanitized.txt] @@ -20,14 +20,26 @@ Compare the two browser-automation options for Hermes's browser tool. |---|---|---| | In use on this install | **Yes** — configured `cloud_provider` | No evidence found | | Config | `browser.camofox.managed_persistence: true` (note: key is spelled `camofox` in config, not `camoufox`) | n/a | -| Driver | `cua-driver` v0.9.0 at `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | n/a | +| Driver | `cua-driver` **v0.28.2** at `/home/hermes/.local/bin/cua-driver` (the `0.9.0` release path this page used to cite no longer exists) | n/a | | Display | Headed, via Xvfb `:99` + fluxbox — see [[systems/headless-server]] | n/a | | `use_gateway` | `false` | n/a | ## Detail ### Camoufox (in use) -- **Verified:** `browser.cloud_provider: camofox`, `inactivity_timeout: 120s`, `allow_private_urls: true`. +- **Verified 2026-10-20:** `browser.cloud_provider: steel`, `browser.engine: camofox`, + `browser.headed: true`, `browser.inactivity_timeout: 450`, `browser.use_gateway: false`, + `browser.camera_provider: camofox`, `browser.allow_private_urls: true`, + `browser.allow_unsafe_evaluate: true`, `browser.cdp_url: ''`, + `browser.command_timeout: 185`. + `browser.camofox:` → `adopt_existing_tab: true`, `managed_persistence: true`, + `rewrite_loopback_urls: true`, `user_id: anthony`. + **Note `inactivity_timeout` is 450s, not the 120s recorded here** — that value changed. + `cloud_provider` is `steel` with `engine: camofox`; the two keys describe different layers. +- **Live processes (2026-10-20):** `cua-driver serve` (PID 3312393), `camoufox-ws-server.mjs`, + and a live `camoufox-bin` (FirefoxDev build) tree under `/home/hermes/.cache/camoufox/`. + Camofox's REST port **9377 is not listening** on CT460 — the Camofox browser service runs on + **CT450**; see [[systems/browser-backend]]. - Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing. - `cua-driver` bridges Hermes's computer-use tool calls to the browser. @@ -36,7 +48,8 @@ Compare the two browser-automation options for Hermes's browser tool. - Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome. ## Verdict -This install uses **Camoufox exclusively**. There's no evidence Chrome/Chromium was ever wired in as an alternative on this host — the comparison exists more as a placeholder for a decision that may have been made outside this wiki than as an active either/or choice today. +This install uses **Camoufox exclusively** (via the `cua-driver` + `camoufox-ws-server.mjs` stack +on CT460 and a separate Camofox service on CT450). There's no evidence Chrome/Chromium was ever wired in as an alternative on this host — the comparison exists more as a placeholder for a decision that may have been made outside this wiki than as an active either/or choice today. ## Related - [[systems/browser-backend]] diff --git a/comparisons/hermes-vs-nanobot.md b/comparisons/hermes-vs-nanobot.md index 7b4afdd..fe4abb9 100644 --- a/comparisons/hermes-vs-nanobot.md +++ b/comparisons/hermes-vs-nanobot.md @@ -3,8 +3,8 @@ title: Hermes vs Nanobot type: comparison status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [comparison, hermes, nanobot] sources: [raw/configs/hermes-config-sanitized.txt] @@ -22,6 +22,16 @@ Compare Hermes (this agent) with Nanobot, another agent process in the homelab, - It's publicly reachable via the same Cloudflare Tunnel that fronts Hermes: `nanobot.martinwa.org` → `100.122.189.81:8766`, and `nano-ssh.martinwa.org` → SSH on the same host. - The earlier note "Nanobox on the PVE host at `192.168.178.39`" conflated Nanobot with `pve`'s own LAN IP (that address is Proxmox's web UI, per the tunnel's `pve.martinwa.org` entry) — an error, not a second Nanobot instance. +> **Re-verified 2026-10-20 — the core claims on this page still hold.** The `clawtest` peer is +> still on Tailscale at **`100.122.189.81`** (`tailscale status` → `100.122.189.81 clawtest +> anthonymau@ linux -`) and `http://nanobot.martinwa.org` still answers (**301**). So the +> correction recorded above remains accurate 90 days on. +> +> **One caveat worth adding:** CT333 now runs **both** Nanobot *and* ZeroClaw (the latter verified +> live on 2026-10-20 — see [[agents/zeroclaw]]). The page's framing of CT333 as the Nanobot host +> is not wrong, but the host is no longer single-purpose. Note also that **CT333 is `zeroclaw` in +> the current Proxmox naming** — the wiki uses the older "clawtest" label for the same LXC. + ## At a glance | | Hermes (ct460) | Nanobot (ct333 "clawtest") | |---|---|---| diff --git a/comparisons/memory-backends.md b/comparisons/memory-backends.md index 8bd82e0..b39db9f 100644 --- a/comparisons/memory-backends.md +++ b/comparisons/memory-backends.md @@ -3,8 +3,8 @@ title: Memory Backends type: comparison status: active created: 2026-07-22 -updated: 2026-08-12 -verified_on: 2026-08-12 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [comparison, memory] sources: [raw/configs/hermes-config-sanitized.txt] @@ -30,6 +30,19 @@ The three archived scripts (`hindsight-watchdog.py.archived`, `hindsight_migrati - Risk assessment flagged: schema mismatch, embedding drift, scope leakage, provenance loss, and one-way/no-rollback migration with no dry-run snapshot. - **Verdict:** stay on Mnemosyne. Config verified `memory.provider: mnemosyne` (2026-08-12). No migration was performed. +> **Re-verified 2026-10-20 — Mnemosyne is still the only backend, and the OpenViking decision still +> holds.** `memory.provider: mnemosyne`, `memory_enabled: true`, `flush_min_turns: 6`, +> `memory_char_limit: 5000`, `user_char_limit: 3000`, `user_profile_enabled: true`, +> `nudge_interval: 10`; Mnemosyne sub-block `auto_sleep: true`, `default_scope: session`, +> `vector_type: int8`, `sleep_threshold: 50`, `profile_isolation: true`, +> `shared_surface_path: data/shared/mnemosyne.db`, `shared_surface_read: true`. +> No `openviking` key anywhere in `config.yaml`. No migration performed. +> +> **New since 2026-08-12:** `mnemosyne.skip_contexts: [cron, flush, subagent, background, +> skill_loop]` and `reflect.disabled_for_cron: true` — Mnemosyne capture is deliberately skipped +> in cron contexts, which is why scheduled jobs produce no working memories. See +> [[concepts/agent-memory]] and [[systems/mnemosyne]]. + ## Related - [[systems/mnemosyne]] - [[systems/memory-backend]] diff --git a/comparisons/model-routers.md b/comparisons/model-routers.md index 7afada6..afd0078 100644 --- a/comparisons/model-routers.md +++ b/comparisons/model-routers.md @@ -3,8 +3,8 @@ title: Model Routers type: comparison status: active created: 2026-07-22 -updated: 2026-08-12 -verified_on: 2026-08-12 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [comparison, routing] sources: [raw/configs/hermes-config-sanitized.txt] @@ -18,17 +18,47 @@ Compare LiteLLM, OmniRoute, and OpenRouter as model-routing layers for Hermes. ## At a glance | | LiteLLM | OmniRoute | OpenRouter | |---|---|---|---| -| Role in Hermes | **Default/primary** provider | **Fallback/auxiliary** provider | Not configured as a standalone provider | -| Endpoint | `http://litellm:4000/v1` | `http://omniroute:20128/v1` (public: `omniroute.martinwa.org`) | n/a | -| Advertised model count | ~1512 | ~3430 | n/a | +| Role in Hermes | Fallback + a *custom_providers* entry | **Auxiliary + the pin on essentially every model-calling cron job** | Not configured as a standalone provider | +| Endpoint | `http://litellm:4000/v1` | `http://omniroute:20128/v1` (public: `omniroute.martinwa.org`) **and** `http://100.78.63.20:20128` (`omni-101`) | n/a | +| Model count (live, 2026-10-20) | **1869** (`/v1/models` with the configured key) | **6457** at `omniroute:20128/v1/models` | n/a | +| Model count (declared in `config.yaml`) | 1728 | 5398 (`Omniroute:20128`) / 5384 (`omni-101`) | n/a | | Routing style | Straight proxy | Combo strategy, LKGP scoring, session affinity | n/a | | Aux roles used for | — | `skills_hub`, `approval`, `mcp` | — | -| Config presence | `custom_providers` in `~/.hermes/config.yaml` | same | absent from `custom_providers`; only referenced via a kept [[decisions/keep-openrouter-as-fallback]] decision | +| Config presence | `custom_providers` in `~/.hermes/config.yaml` | same (two separate entries) | absent from `custom_providers`; only referenced via a kept [[decisions/keep-openrouter-as-fallback]] decision | + +> **The "LiteLLM is the default/primary" row was wrong by 2026-10-20 (verified and corrected).** +> The live `model:` block in `~/.hermes/config.yaml` is: +> ```yaml +> model: +> provider: nous +> default: meituan/longcat-2.5-preview:free +> base_url: https://inference-api.nousresearch.com/v1 +> api_mode: chat_completions +> persist_switch_by_default: false +> ``` +> So the **primary model path is Nous Research direct**, not LiteLLM. LiteLLM is the *last* entry +> in the fallback chain (`or-inclusionai_ling-2_6-1t` via `custom` → `http://litellm:4000/v1`). +> +> **The two OmniRoute endpoints matter.** `Omniroute:20128` is a DNS name (m93p CT101); +> `omni-101` is the raw IP `100.78.63.20:20128` (node `omniroute-1`) and is the alias pinned on +> effectively every model-calling cron job. The two carry **different catalogs** (5398 vs 5384 +> declared models) and the `omni-101` entry's own `model:` is `smart-route` while +> `Omniroute:20128`'s is `go-cheap`. +> +> **Also: both OmniRoute endpoints and LiteLLM now require authentication.** `curl` without a key +> returns **401 `invalid_api_key`** on all three `/v1/models` endpoints. The keys live in +> `~/.hermes/.env` via each provider's `key_env` (e.g. `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`). +> Any runbook that probes these endpoints with a bare `curl` will see 401 and may wrongly conclude +> the provider is down — it is not. ## Detail -### LiteLLM — primary -Default provider for main chat traffic (`freellmapi-auto` default model resolves through it). See [[systems/model-providers]]. +### LiteLLM — fallback tail + a `custom_providers` entry +LiteLLM hosts the `freellmapi-auto` model alias and is reachable at `http://litellm:4000/v1`, but +it is **no longer the default chat path** — that is Nous direct +(`meituan/longcat-2.5-preview:free`, per the live `model:` block). LiteLLM appears in the +fallback chain as `or-inclusionai_ling-2_6-1t` via the `custom` provider. +See [[systems/model-providers]] and [[concepts/model-routing]]. ### FreeLLMAPI — the router behind `freellmapi-auto` Unified LLM router at `100.82.40.23:3001` (also `freellmapi.martinwa.org`). Verified 2026-08-12: diff --git a/concepts/agent-memory.md b/concepts/agent-memory.md index 2f41ce9..c6d7e1d 100644 --- a/concepts/agent-memory.md +++ b/concepts/agent-memory.md @@ -3,8 +3,8 @@ title: Agent Memory type: concept status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [concept, memory] sources: [raw/configs/hermes-config-sanitized.txt] @@ -20,6 +20,23 @@ How an agent retains facts/context beyond a single conversation turn or session - **Verified settings:** memory enabled `true`; message char limit `5000`; user-profile char limit `3000`; flush every `6` turns; `auto_sleep: true`; default scope `session`; vector type `int8`; shared surface `data/shared/mnemosyne.db`. - **Correction (2026-07-22, per Anthony):** this page previously named Hindsight as a predecessor backend on this box. That was a mix-up with a different Hermes instance — Hindsight has nothing to do with this box. See [[comparisons/memory-backends]] for the full correction. +> **Full `memory:` block re-read 2026-10-20 — the summary above is accurate but incomplete.** +> Live values: `memory_enabled: true`, `provider: mnemosyne`, `memory_char_limit: 5000`, +> `user_profile_enabled: true`, `user_char_limit: 3000`, `flush_min_turns: 6`, +> `nudge_interval: 10`. Mnemosyne sub-block: `auto_sleep: true`, `default_scope: session`, +> `vector_type: int8`, `sleep_threshold: 50`, `profile_isolation: true`, +> `shared_surface_path: data/shared/mnemosyne.db`, `shared_surface_read: true`, +> `sync_roles: [user]`, `ignore_patterns: []`. +> +> **New since 2026-07-22 and worth knowing:** +> - `reflect.disabled_for_cron: true`, `reflect.max_calls_per_session: 3` +> - **`skip_contexts: [cron, flush, subagent, background, skill_loop]`** — Mnemosyne capture is +> deliberately skipped in cron, flush, subagent, background and skill-loop contexts. That is +> exactly why cron runs like this one produce no Mnemosyne working memories: the skip is +> configured, not a malfunction. +> - `profile_isolation: true` — memories do not leak across Hermes profiles, which matters now that +> the `ops` profile has been deleted (see [[runbooks/multiplexer-setup]]). + ## Model - Per-turn conversation content accumulates up to the char limit, then flushes to the Mnemosyne store every 6 turns (rather than every turn) — a batching tradeoff between write frequency and losing very recent context if the process dies mid-batch. - `scope: session` means recall defaults to the current session rather than a global cross-session pool, unless explicitly widened. diff --git a/concepts/model-routing.md b/concepts/model-routing.md index 6c94a6d..fbe1074 100644 --- a/concepts/model-routing.md +++ b/concepts/model-routing.md @@ -3,8 +3,8 @@ title: Model Routing type: concept status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [routing, model, inference] sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md] @@ -20,6 +20,26 @@ How Hermes selects providers, applies combos, and enforces context limits. - Default model: `freellmapi-auto` - OmniRoute used for auxiliary roles: vision, skills_hub, approval, mcp, title_generation, web_extract, compression +> **Live `model:` block re-read 2026-10-20 — the default has moved.** +> `model.default` is now **`meituan/longcat-2.5-preview:free`** (provider `nous`), not +> `freellmapi-auto`. The block also carries `base_url`, `persist_switch_by_default` and `api_mode`. +> +> **The five `custom_providers` and their live base URLs:** +> +> | Name | Base URL | Host | +> |---|---|---| +> | `litellm` | `http://litellm:4000/v1` | LiteLLM proxy (m93p CT500) | +> | `guacano` | `http://ollama:11000/v1` | Guanaco on CT205 | +> | `Omniroute:20128` | `http://omniroute:20128/v1` | OmniRoute on m93p CT101 | +> | `command-code` | `https://api.commandcode.ai/provider/v1` | remote | +> | `omni-101` | `http://100.78.63.20:20128` | OmniRoute node `omniroute-1` (direct IP, not a DNS name) | +> +> Note the **two distinct OmniRoute endpoints**: `Omniroute:20128` (DNS, m93p CT101) and +> `omni-101` (raw IP `100.78.63.20:20128`). `omni-101` is the pin on effectively every +> model-calling cron job, and it has been the single point of failure in the 10-14, 10-16/17 and +> 10-19 incidents — its `/api/health` reads `200 {"status":"ok"}` throughout, so **reachability +> probes do not detect that failure class**; only job-level error text does. + ## Context Enforcement - Three layers: model native → combo `data.config.context_length` → Hermes client cap - Effective context is the minimum of the three diff --git a/concepts/persistent-shells.md b/concepts/persistent-shells.md index 55bfda3..1005aa8 100644 --- a/concepts/persistent-shells.md +++ b/concepts/persistent-shells.md @@ -3,8 +3,8 @@ title: Persistent Shells type: concept status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [concept, terminal] sources: [raw/configs/hermes-config-sanitized.txt] @@ -23,6 +23,14 @@ A background shell session that survives across separate Hermes turns/requests, - **Home mode:** auto - **Container CPU:** 1 +> **Full `terminal:` block re-read 2026-10-20 — the above still holds, plus more.** +> Live values: `backend: local`, `persistent_shell: true`, `timeout: 180`, `home_mode: auto`, +> `container_cpu: 1`, `container_memory: 5120`, `container_disk: 51200`, +> `container_persistent: true`, `lifetime_seconds: 300`, +> `docker_mount_cwd_to_workspace: false`, `docker_run_as_host_user: true`. +> The container-* keys are inert while `backend: local` — the shell is not containerised on this +> box, so `container_memory`/`container_disk` describe a path that isn't taken. + ## Why it matters here Enables multi-step CLI workflows (e.g. a long `git` operation, a build, a watch process) to be resumed across turns without Hermes re-establishing context each time. The tradeoff is a shell can be left in a bad state (stuck process, wrong directory) that silently affects the *next* turn — see [[runbooks/troubleshoot-tool-loop]] for the related tool-loop guardrails that catch some of this. diff --git a/concepts/tool-calling.md b/concepts/tool-calling.md index ab3fc77..a8aac65 100644 --- a/concepts/tool-calling.md +++ b/concepts/tool-calling.md @@ -3,8 +3,8 @@ title: Tool Calling type: concept status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [concept, tool-calling] sources: [raw/configs/hermes-config-sanitized.txt] @@ -35,6 +35,20 @@ hard_stop_after: ## Why it matters here This is the mechanism behind [[runbooks/troubleshoot-tool-loop]] — that runbook's "agent loops on one tool" symptom is exactly what `warn_after`/`hard_stop_after` are meant to catch. With hard-stop disabled, a human (or the messaging integrations) is the actual backstop today. +> **Re-verified 2026-10-20 — the `tool_loop_guardrails:` block is byte-for-byte unchanged.** +> Live values in `~/.hermes/config.yaml`: `warnings_enabled: true`, `hard_stop_enabled: false`, +> `warn_after: {exact_failure: 2, same_tool_failure: 3, idempotent_no_progress: 2}`, +> `hard_stop_after: {exact_failure: 5, same_tool_failure: 8, idempotent_no_progress: 5}`. +> 90 days of no drift on this guardrail. +> +> **The 2026-10-18 incident is a *different* loop and the guardrails did not catch it.** +> The degenerate-stream case (see +> [[incidents/2026-10-18-degenerate-stream-canned-substitute]]) was a **model output** repetition +> loop — the *model* repeating itself in the response stream, not a tool call repeating. Hermes's +> degenerate-stream guard cut it and delivered a canned non-answer, but `tool_loop_guardrails` +> (which watches *tool* calls) never fired. Two separate loop-detection mechanisms exist; do not +> assume one covers the other. + ## Related - [[concepts/persistent-shells]] - [[runbooks/troubleshoot-tool-loop]] diff --git a/decisions/index.md b/decisions/index.md index 63eb585..f656170 100644 --- a/decisions/index.md +++ b/decisions/index.md @@ -3,8 +3,8 @@ title: Decisions type: collection status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [decisions] --- @@ -19,6 +19,15 @@ Stable strategic and config decisions, with evidence and status. - [[decisions/persistent-shell-enabled]] - [[decisions/rollback-steel-to-camofox]] +> **Status sweep 2026-10-20:** all three listed decisions are still `status: active`, but two now +> carry a review note: +> - [[decisions/keep-openrouter-as-fallback]] — its 30-day revisit trigger is **met**; OpenRouter is +> reachable only through the OmniRoute catalog, not as a standalone provider. +> - [[decisions/persistent-shell-enabled]] — still `true`; its `review_after: 0001-01-01` zero-date +> bug is now corrected to a real date. +> - [[decisions/rollback-steel-to-camofox]] — the *engine* is Camoufox as decided, but +> `cloud_provider: steel` remains in the config while the Steel service itself is unreachable. + ## History - **Verified:** Decision pages initialized during wiki setup on 2026-07-22 - **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing diff --git a/decisions/keep-openrouter-as-fallback.md b/decisions/keep-openrouter-as-fallback.md index eba1390..bee8d80 100644 --- a/decisions/keep-openrouter-as-fallback.md +++ b/decisions/keep-openrouter-as-fallback.md @@ -3,8 +3,9 @@ title: Keep OpenRouter as fallback type: decision status: active created: 2026-07-22 -updated: 2026-07-22 -review_after: 2026-08-22 # was 0001-01-01 (bug), fixed during 2026-07-22 audit; real trigger is conditional, see Revisit when +updated: 2026-10-20 +review_after: 2026-10-20 # was 0001-01-01 (bug), fixed during 2026-07-22 audit; real trigger is conditional, see Revisit when +verified_on: 2026-10-20 confidence: high tags: [routing, provider, decision] sources: [] @@ -28,6 +29,27 @@ OpenRouter must remain available as a fallback provider, even if a self-hosted r ## Revisit when A self-hosted router has operated reliably for at least 30 days. +> **Revisit trigger MET (verified 2026-10-20).** OmniRoute has been the primary router for well over +> 30 days — `custom:omni-101` is the pin on effectively every model-calling cron job, and +> `Omniroute:20128` is a second live OmniRoute endpoint. The estate has **no standalone OpenRouter +> provider**: `custom_providers` in `~/.hermes/config.yaml` holds only `litellm`, `guacano`, +> `Omniroute:20128`, `command-code` and `omni-101` (see [[systems/openrouter]]). +> +> **But the fallback is now provided *through* OmniRoute rather than alongside it** — OpenRouter-style +> slugs (`openrouter/auto`, `freeapi/openrouterfree`, `or-openrouter-fusion`, …) are present in the +> OmniRoute catalog, so OpenRouter capacity remains reachable without a separate Hermes provider. +> +> **Status: decision still `active`, but the mechanism it protects has changed.** The intent +> (OpenRouter capacity must remain reachable as a fallback) is satisfied via the OmniRoute catalog. +> The literal reading — a standalone OpenRouter provider in `custom_providers` — is **not** satisfied +> and has not been since at least 2026-07-22. This needs a re-affirmation or a rewrite of the +> decision text; it should not simply be left to drift on a 30-day timer. +> +> Related context: OmniRoute has also been the source of repeated fleet-wide failures — +> the 2026-10-14 connection-refused storm, the 2026-10-16/17 and 2026-10-19 resource-pressure 503 +> bursts, and the 2026-10-18 degenerate-stream delivery. A single-node fallback posture carries +> that concentration risk with it. + ## Related - [[model-routing]] - [[systems/omniroute]] diff --git a/decisions/persistent-shell-enabled.md b/decisions/persistent-shell-enabled.md index 68cca98..9dde45c 100644 --- a/decisions/persistent-shell-enabled.md +++ b/decisions/persistent-shell-enabled.md @@ -3,8 +3,9 @@ title: Persistent shell enabled type: decision status: active created: 2026-07-22 -updated: 2026-07-22 -review_after: 0001-01-01 +updated: 2026-10-20 +review_after: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [automation, shell, decision] sources: [] @@ -24,5 +25,14 @@ Enable persistent shells for Hermes. ## Revisit when Use case becomes critical again. +> **Re-verified 2026-10-20 — decision still `active` and the setting is still `true`.** +> `terminal.persistent_shell: true` in `~/.hermes/config.yaml`, `backend: local`, +> `timeout: 180`, `home_mode: auto`, `container_cpu: 1`, `lifetime_seconds: 300`. +> The `review_after: 0001-01-01` value was a zero-date bug (same class as +> [[decisions/keep-openrouter-as-fallback]]); it is now a real date. +> One caveat worth carrying into the review: `lifetime_seconds: 300` means a persistent shell +> is torn down after 5 minutes idle, so "state persists across turns" is true within a burst of +> work but not indefinitely. See [[concepts/persistent-shells]]. + ## Related - [[persistent-shells]] diff --git a/decisions/rollback-steel-to-camofox.md b/decisions/rollback-steel-to-camofox.md index 1d49f16..3204a01 100644 --- a/decisions/rollback-steel-to-camofox.md +++ b/decisions/rollback-steel-to-camofox.md @@ -3,8 +3,8 @@ title: Rollback Steel to Camoufox type: decision status: active created: 2026-08-02 -updated: 2026-08-02 -verified_on: 2026-08-02 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [decision, browser, steel, camofox, rollback] sources: [session-history, live-config] @@ -28,6 +28,21 @@ Steel Browser was deployed on CT201 on 2026-07-29 as the primary browser automat ## Decision **Revert to Camoufox as primary.** Steel plugin disabled but left running on CT201. +> **Status check 2026-10-20 — the decision is *partially* undone in configuration.** +> The live `browser:` block in `~/.hermes/config.yaml` reads +> `cloud_provider: steel` with `engine: camofox`. The **engine** is Camoufox (decision honoured), +> but the **cloud_provider** key still says `steel`, and `camera_provider: camofox`. +> This is the same split seen in [[comparisons/camoufox-vs-chrome]]. +> +> Meanwhile the Steel service itself is **not reachable**: `http://steel.martinwa.org` returns +> **000** (connection failed) on 2026-10-20. Combined with the degraded CT201 stack recorded in +> [[systems/homepage-dashboard]] (main container removed, asset proxy 502), it looks like the +> "left running on CT201" part of this decision is no longer true. +> +> **Recommendation, not actioned here:** the `cloud_provider: steel` value appears to be residue +> from the reverted deployment. Confirm against the live Steel service and either restore it or +> set `cloud_provider: camofox` so the config matches the decision this page records. + ## Rationale 1. **Reliability:** Camoufox runs on the same host as Hermes (CT450) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues. 2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports. diff --git a/experiments/index.md b/experiments/index.md index 31d5fd0..9cb4122 100644 --- a/experiments/index.md +++ b/experiments/index.md @@ -3,8 +3,8 @@ title: Experiments type: collection status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: medium tags: [experiments] --- @@ -15,9 +15,30 @@ Non-canonical tests, tuning passes, and change attempts. ## Known experiments -- `pre-vision-fix-20260718-195712` config snapshot — Verified, backup file exists, 2026-07-18 -- `pre-model-fix` config snapshot — Verified, backup file exists -- `pre-camofox-tuning` config snapshot — Verified, backup file exists, 2026-07-20 +- `pre-vision-fix-20260718-195712` config snapshot — **no longer found by that name** +- `pre-model-fix` config snapshot — **no longer found by that name** +- `pre-camofox-tuning` config snapshot — **no longer found by that name** + +> **Re-verified 2026-10-20 — the three named snapshots this page listed are GONE.** +> `find ~/.hermes/backups -name "*pre-*"` returns only: +> - `config.yaml.pre-cua-durable-20260730` (the one that `pre-camofox-tuning` appears to have +> been renamed to — it is dated 2026-07-30, matching the CUA durable-persistence change) +> - `incident-20260727-mattermost-removal/config.yaml.pre-mattermost-off` +> +> The other two (`pre-vision-fix-20260718-195712`, `pre-model-fix`) have **no surviving file under +> that name** in `~/.hermes/backups/`. The backups directory now holds 15 entries including +> `config/` (a rolling series of `config.yaml.bak.*`, `.corrupt.*`, `.good.*` and +> `config.yaml.premnemo.20260928-174645`) plus nine dated incident directories: +> `dashboard-regression-20260811`, `incident-20260727-mattermost-removal`, +> `legacy-memory-slim-20260801_180702`, `mnemosyne-capture-fix-20260801_180249`, +> `phantom-default-profile-20260730-1250`, `resource-hardening-20260725`, +> `session-repair-20260815-205936`. +> +> The `config/` series is worth knowing about operationally: it includes **four `config.yaml.corrupt.*` +> files** (2026-09-28 ×2, 2026-09-30, 2026-10-01) — evidence of the `hermes config set` stringifying +> lists/dicts problem — alongside `config.yaml.good.*` snapshots from 2026-10-09, 10-11 and 10-16. +> See [[incidents/2026-09-28-hermes-managed-runtime-deps]] and the runtime-deps notes in +> [[systems/hermes-agent]]. ## Outcomes - **Verified:** Backups exist with the above names diff --git a/incidents/2026-08-01-mnemosyne-capture-failure.md b/incidents/2026-08-01-mnemosyne-capture-failure.md index 3ba3fe7..18bbe6a 100644 --- a/incidents/2026-08-01-mnemosyne-capture-failure.md +++ b/incidents/2026-08-01-mnemosyne-capture-failure.md @@ -3,8 +3,9 @@ title: Mnemosyne Automatic Capture Failure type: incident status: active created: 2026-08-01 -updated: 2026-08-01 -verified_on: 2026-08-01 +updated: 2026-10-20 +resolved_on: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [memory, mnemosyne, hermes, incident] sources: [live-config, mnemosyne-db, hermes-cli-e2e] @@ -53,6 +54,34 @@ The already-running Telegram gateway/provider instance was initialized before th ## Monitoring Note The health script checks capture staleness only after 24 hours. It therefore remained green during this fourteen-hour active-capture outage. A future improvement should compare recent non-cron conversation activity with the newest `working_memory` row rather than lowering the threshold blindly and creating idle-period false positives. +## Resolution confirmed (2026-10-20) + +The remaining "Activation Step" on this page — restarting the gateway to load the corrected typed +configuration — **has been completed and the config is still correct 50 days later.** Verified live +in `/home/hermes/.hermes/config.yaml` under `memory.mnemosyne`: + +| Key | Live value | Was it a string? | +|---|---|---| +| `sync_roles` | `['user']` (real list) | no — now a native list | +| `skip_contexts` | `['cron', 'flush', 'subagent', 'background', 'skill_loop']` | no — native list | +| `reflect` | `{disabled_for_cron: True, max_calls_per_session: 3}` (dict) | no — native dict | +| `sleep_threshold` | `50` (int) | no — int | +| `ignore_patterns` | `[]` (list) | no — native list | +| `profile_isolation` | `True` (bool) | no — bool | +| `auto_sleep` / `default_scope` / `vector_type` | `True` / `session` / `int8` | no | + +**None of these are stringified.** The `hermes config set` stringification bug that caused this +incident has not recurred for this block. + +**One drift to note:** `skip_contexts` now includes `cron`, which was not in the original repair +list. Mnemosyne capture is therefore **deliberately skipped in cron contexts** — scheduled jobs +produce no working memories by design. That explains why cron-driven runs of this wiki job leave +no Mnemosyne trace. + +**Status: `resolved`.** The back-up copies named in the repair section still exist: +`~/.hermes/backups/mnemosyne-capture-fix-20260801_180249/` and +`~/.hermes/backups/legacy-memory-slim-20260801_180702/`. + ## Related - [[systems/mnemosyne]] - [[systems/memory-backend]] diff --git a/incidents/2026-08-29-gmail-read-state.md b/incidents/2026-08-29-gmail-read-state.md index b63efae..4dc3e91 100644 --- a/incidents/2026-08-29-gmail-read-state.md +++ b/incidents/2026-08-29-gmail-read-state.md @@ -3,8 +3,8 @@ title: "Gmail Read-State Incident (2026-08-29)" type: incident status: active created: 2026-08-29 -updated: 2026-08-29 -verified_on: 2026-08-29 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [email, gmail, cron, himalaya, incident] sources: [session:default/20260829_204103_9e1087] @@ -59,6 +59,32 @@ Despite using `--preview`, Himalaya may still mark messages as read depending on - `goimapnotify` stopped - Important-mail monitor paused (if it existed) +> **Re-verified 2026-10-20 — the mitigation has partially REGRESSED, and the root cause is still +> live.** +> +> | Item | State on 2026-08-29 | State on 2026-10-20 | +> |---|---|---| +> | `goimapnotify.service` | stopped | **ACTIVE again** — `systemctl --user is-active goimapnotify.service` → `active`, PID 383 `/home/hermes/go/bin/goimapnotify -conf /home/hermes/.config/goimapnotify/goimapnotify.yaml` | +> | Important-mail monitor (`d40085da631c`) | paused | **still paused** — `enabled: False` in `jobs.json`, `last_status: ok` | +> | `fetch_newsletters.py` line 49 | patched to `--preview` | **still `himalaya message read --preview --folder INBOX`** — the unsafe call is still in the file, unfixed | +> +> **So the real fix — rewriting `fetch_newsletters.py` to use readonly IMAP (`SELECT readonly` / +> `BODY.PEEK[]`) — has still not been done 52 days later.** The two mitigations that were applied +> are one of them gone (goimapnotify restarted) and one still holding (the important-mail monitor +> remains disabled). +> +> **Other email crons currently in `error` state (2026-10-20 read of `jobs.json`):** +> `85edf6772bca` Tech & AI Newsletter Digest, `08eb86f02d37` Gmail Action Digest, +> `06a0a1c8704f` AI Newsletter Curation for Patrick. Their most recent failures were +> provider-side (OmniRoute 503 resource pressure on 2026-10-19; a connection error on 2026-10-15), +> **not** read-state errors — but the read-state exposure persists regardless, because the +> newsletter fetch path still issues `message read`. +> +> **Recommended action (not taken here — this is a wiki job, and the fix touches Anthony's mail):** +> rewrite `fetch_newsletters.py` to use readonly IMAP, and decide whether `goimapnotify` should be +> running. See [[systems/scheduled-tasks]] and the Gmail IMAP lessons in +> [[runbooks/provider-health]]. + **Fix required:** - Rewrite `fetch_newsletters.py` to use readonly IMAP (`BODY.PEEK[]`) instead of `himalaya message read` - Audit all Gmail-related scripts for similar patterns diff --git a/infrastructure/backups.md b/infrastructure/backups.md index d4a155c..5442d2c 100644 --- a/infrastructure/backups.md +++ b/infrastructure/backups.md @@ -3,8 +3,8 @@ title: Backups type: infrastructure status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: medium tags: [infrastructure, backups] sources: [raw/configs/hermes-config-sanitized.txt] @@ -33,6 +33,21 @@ How Hermes's own configuration/state gets backed up, and what's missing. ## Known gaps - **No scheduled backup job found:** `crontab -l -u hermes` returns empty, and no systemd timer references `hermes` or `backup`. The one full backup on disk appears to have been triggered manually, not on a schedule. + +> **Re-verified 2026-10-20 — the gap is unchanged and the single backup is now 90 days old.** +> - `crontab -l -u hermes` → "no crontab for hermes". `systemctl --user list-timers --all` → **0 timers**. +> The only timer on the box is the system-level `dpkg-db-backup.timer` — nothing Hermes-related. +> - The one full backup is still `/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, dated +> 22 Jul). **No new full backup has been taken in the 90 days since.** +> - `/home/hermes/hermes-wiki/` is still **empty**; `/home/hermes/wiki-personal-backups/` still holds +> only the four original 22 Jul files (`current-state.md`, `index.md`, `log.md`, `SCHEMA.md`). +> - `~/.hermes/scripts/mnemosyne-backup.sh` still exists (unchanged since 3 Jul). +> - `updates:` in `config.yaml` still reads `pre_update_backup: false`, `backup_keep: 5`, +> `non_interactive_local_changes: stash`. +> +> **The separate wiki backup is also unrun:** `/home/hermes/wiki-backups/` does not exist, so the +> tarball step in [[runbooks/backup-wiki]] has never been executed (the runbook's `mkdir -p` would +> create it on first use). - `pre_update_backup: false` means an in-place `hermes update` will not auto-snapshot first — the manual step in [[runbooks/update-hermes-safely]] step 1 is currently the only safety net. ## Relevant runbooks diff --git a/infrastructure/credentials-map.md b/infrastructure/credentials-map.md index 4e41118..7357fab 100644 --- a/infrastructure/credentials-map.md +++ b/infrastructure/credentials-map.md @@ -3,8 +3,8 @@ title: Credentials Map type: infrastructure status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: medium tags: [credentials, security, infrastructure] sources: [] @@ -32,6 +32,19 @@ Document where credentials are stored and what service they belong to. | Huawei Cloud | Terraform state / config | Unverified | | | GitHub | Hermes config / gh CLI | `~/.config/gh/` or `~/.hermes/` | Unverified | +> **Re-verified 2026-10-20 — two corrections to the table above.** +> 1. **Email credentials live in `~/.hermes/.env`, not `config.yaml`.** `config.yaml` has **no** +> `imap`/`smtp`/`email` keys; the live variables are `EMAIL_ADDRESS`, `EMAIL_IMAP_HOST`, +> `EMAIL_SMTP_HOST`, `EMAIL_ALLOWED_USERS`, `EMAIL_HOME_ADDRESS`, `EMAIL_ALLOW_ALL_USERS` +> and friends in `~/.hermes/.env` (mode `0600`). The `config.yaml` row is wrong — point at the `.env`. +> 2. **The `tools:` block in `config.yaml` contains only `tool_search`.** There are no `gmail`, +> `discord`, `mattermost` or `email` tool sections there. Discord and Telegram appear as +> **top-level** `discord:` / `telegram:` blocks, and Mattermost is a **plugin** +> (`platforms/mattermost`), not a `tools:` entry. So the "Hermes config" location column is right +> for those, but the implication that they are `tools:` sections is not. +> No `connectors:` block is configured in `config.yaml` at all. +> `~/.hermes/auth.json` also exists and holds OAuth tokens — treat as a credential store. + ## Rules - Credentials never written to this wiki - API vault is first-stop for any key lookup diff --git a/infrastructure/domains-and-tunnels.md b/infrastructure/domains-and-tunnels.md index 8148cc8..1e2ad4e 100644 --- a/infrastructure/domains-and-tunnels.md +++ b/infrastructure/domains-and-tunnels.md @@ -3,8 +3,8 @@ title: Domains and Tunnels type: infrastructure status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [infrastructure, network, cloudflare] sources: [raw/configs/hermes-config-sanitized.txt] @@ -31,7 +31,30 @@ Where `martinwa.org` subdomains point, and how they reach internal services. | `omniroute.martinwa.org` | `http://omniroute.kangaroo-eel.ts.net:20128` | Public front for [[systems/omniroute]] | | `nanobot.martinwa.org` / `nano-ssh.martinwa.org` | `100.122.189.81` (ct333 "clawtest") | See [[comparisons/hermes-vs-nanobot]] | -`mcp-portal.martinwa.org` is not present in the ct207 ingress list captured during this audit — it may be served by a different tunnel/host, or the ingress list may have moved since. **Unknown**, flagged for follow-up rather than guessed. +`mcp-portal.martinwa.org` is **still not present** in the ct207 ingress list — re-read live from +`/etc/cloudflared/config.yml` on ct207 on **2026-10-20**. It may be served by a different +tunnel/host, or the ingress list may have moved since. **Still unknown**, still flagged for +follow-up rather than guessed. + +### Ingress changes found 2026-10-20 (full live re-read of ct207 `config.yml`) + +The hostname list had drifted from the 2026-07-22 capture. Verified additions and corrections: + +| Hostname | Live target | Note | +|---|---|---| +| `hermes-chat.martinwa.org` | `http://100.118.5.51:8787` | **new** — WebUI | +| `tg.hermes.martinwa.org` | `http://100.118.5.51:19119` | **new** — Telegram bot webhook | +| `openwebui.martinwa.org` | `http://100.69.230.13:4000` | **new** — Open WebUI | +| `news.martinwa.org` | `http://docker.kangaroo-eel.ts.net:8300` | **new** — WorldMonitor | +| `ssh-pve.martinwa.org` | `http://100.96.100.82:7681` | **new** — PVE noVNC | +| `nextcloud.martinwa.org` | `https://100.90.102.19:443` | target corrected (was `100.96.150.69`) | +| `readlater.martinwa.org` | `http://100.96.150.69:6123` | confirmed | +| `velvet.martinwa.org` | `http://127.0.0.1:8888` | confirmed | +| `runtipi.martinwa.org` | `http://100.96.244.39:80` | confirmed | +| `homar.martinwa.org` | `http://100.96.244.39:8102` | confirmed — this is **Homarr**, the live dashboard (see [[systems/homepage-dashboard]]) | + +The catch-all remains `service: http_status:404`. The ct207 `config.yml` still carries ~25 dated +`config.yml.bak*` files, newest `config.yml.bak.1781809151`. ## Full ct207 ingress list (context, not all Hermes-related) The tunnel also fronts: `pve.martinwa.org` (Proxmox UI), `docker.martinwa.org` (Portainer), `runtipi.martinwa.org`, `homar.martinwa.org`, `nextcloud.martinwa.org`, `readlater.martinwa.org`, `velvet.martinwa.org`, `overtchat.martinwa.org`, `freellmapi.martinwa.org`, `omnideb.martinwa.org`, `newstarter.martinwa.org`, and SSH endpoints for debianvm/pve/webserver. These belong to the wider homelab, not Hermes specifically — recorded here only so an ingress diff is possible later. diff --git a/raw/stocktake/freshness-20261020.txt b/raw/stocktake/freshness-20261020.txt new file mode 100644 index 0000000..f066e34 --- /dev/null +++ b/raw/stocktake/freshness-20261020.txt @@ -0,0 +1,110 @@ +# Wiki freshness audit — 2026-10-20 +Threshold: >21 days since `updated` + +## STALE (47) + +- [ 90d] comparisons/camoufox-vs-chrome.md (updated 2026-07-22) +- [ 90d] comparisons/hermes-vs-nanobot.md (updated 2026-07-22) +- [ 90d] concepts/agent-memory.md (updated 2026-07-22) +- [ 90d] concepts/model-routing.md (updated 2026-07-22) +- [ 90d] concepts/persistent-shells.md (updated 2026-07-22) +- [ 90d] concepts/tool-calling.md (updated 2026-07-22) +- [ 90d] decisions/index.md (updated 2026-07-22) +- [ 90d] decisions/keep-openrouter-as-fallback.md (updated 2026-07-22) +- [ 90d] decisions/persistent-shell-enabled.md (updated 2026-07-22) +- [ 90d] experiments/index.md (updated 2026-07-22) +- [ 90d] infrastructure/backups.md (updated 2026-07-22) +- [ 90d] infrastructure/credentials-map.md (updated 2026-07-22) +- [ 90d] infrastructure/domains-and-tunnels.md (updated 2026-07-22) +- [ 90d] runbooks/backup-wiki.md (updated 2026-07-22) +- [ 90d] runbooks/diagnose-docker.md (updated 2026-07-22) +- [ 90d] runbooks/recover-docker-service.md (updated 2026-07-22) +- [ 90d] runbooks/restart-browser.md (updated 2026-07-22) +- [ 90d] runbooks/restart-hermes.md (updated 2026-07-22) +- [ 90d] runbooks/restore-openrouter.md (updated 2026-07-22) +- [ 90d] runbooks/test-openrouter.md (updated 2026-07-22) +- [ 90d] runbooks/troubleshoot-tool-loop.md (updated 2026-07-22) +- [ 90d] systems/cloudflare-tunnel.md (updated 2026-07-22) +- [ 90d] systems/headless-server.md (updated 2026-07-22) +- [ 90d] systems/openrouter.md (updated 2026-07-22) +- [ 90d] systems/terminal-backend.md (updated 2026-07-22) +- [ 87d] runbooks/gateway-resource-resilience.md (updated 2026-07-25) +- [ 87d] systems/tool-search.md (updated 2026-07-25) +- [ 86d] systems/homepage-dashboard.md (updated 2026-07-26) +- [ 86d] systems/obsidian.md (updated 2026-07-26) +- [ 84d] runbooks/multiplexer-setup.md (updated 2026-07-28) +- [ 83d] systems/opencode-go.md (updated 2026-07-29) +- [ 82d] systems/hermes-ui.md (updated 2026-07-30) +- [ 80d] incidents/2026-08-01-mnemosyne-capture-failure.md (updated 2026-08-01) +- [ 79d] decisions/rollback-steel-to-camofox.md (updated 2026-08-02) +- [ 79d] reports/uber-eats/index.md (updated 2026-08-02) +- [ 69d] comparisons/memory-backends.md (updated 2026-08-12) +- [ 69d] comparisons/model-routers.md (updated 2026-08-12) +- [ 67d] systems/guanaco.md (updated 2026-08-14) +- [ 66d] agents/index.md (updated 2026-08-15) +- [ 66d] agents/open-claw.md (updated 2026-08-15) +- [ 66d] agents/zeroclaw.md (updated 2026-08-15) +- [ 66d] incidents/2026-08-15-skills-index-drift.md (updated 2026-08-15) +- [ 65d] CONTRIBUTING.md (updated 2026-08-16) +- [ 64d] systems/nextcloud.md (updated 2026-08-17) +- [ 53d] reports/communities-wa-social-media-audit-2026.md (updated 2026-08-28) +- [ 52d] incidents/2026-08-29-gmail-read-state.md (updated 2026-08-29) +- [ 46d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-09-04) + +## FRESH (43) + +- [ 16d] agents/hermes-chaos.md +- [ 16d] agents/hermes-production.md +- [ 9d] agents/nanobot.md +- [ 9d] agents/qwenpaw.md +- [ 7d] comparisons/steel-vs-camoufox.md +- [ 7d] concepts/browser-automation.md +- [ 0d] current-state.md +- [ 8d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 17d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 9d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 14d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 16d] incidents/2026-10-03-ct450-hermes-update.md +- [ 16d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 14d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 7d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 11d] incidents/2026-10-09-memory-recall-relevance.md +- [ 4d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 0d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 1d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 9d] infrastructure/docker-services.md +- [ 16d] infrastructure/hosts.md +- [ 6d] infrastructure/network-map.md +- [ 6d] infrastructure/proxmox.md +- [ 1d] infrastructure/tailscale.md +- [ 8d] queries/retained-research-answers.md +- [ 7d] runbooks/job-radar.md +- [ 16d] runbooks/provider-health.md +- [ 16d] runbooks/update-hermes-safely.md +- [ 7d] systems/browser-backend.md +- [ 18d] systems/current-profile.md +- [ 9d] systems/hermes-agent.md +- [ 8d] systems/home-assistant.md +- [ 9d] systems/ignis.md +- [ 16d] systems/memory-backend.md +- [ 1d] systems/messaging-integrations.md +- [ 1d] systems/mnemosyne.md +- [ 19d] systems/model-providers.md +- [ 8d] systems/nanobot.md +- [ 0d] systems/omniroute.md +- [ 9d] systems/paseo.md +- [ 1d] systems/push-notifications.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md + +## NO updated DATE (7) + +- SCHEMA.md +- experiments/active/example.md +- incidents/2026-07-29-mnemosyne-relay.md +- incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- index.md +- log.md +- reports/uber-eats/2026-07.md + +TOTAL: 47 stale, 43 fresh, 7 no-date of 97 pages \ No newline at end of file diff --git a/runbooks/backup-wiki.md b/runbooks/backup-wiki.md index 4e5a346..ae239ea 100644 --- a/runbooks/backup-wiki.md +++ b/runbooks/backup-wiki.md @@ -3,9 +3,9 @@ title: Backup Wiki type: runbook status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 -last_tested: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 +last_tested: 2026-10-20 confidence: high tags: [runbook, wiki, backup] sources: [] @@ -26,6 +26,15 @@ Create a durable backup of the wiki so recoverable copies exist before mutating - Git is available - Backup destination is writable +> **Two trees, not one (verified 2026-10-20).** This runbook still backs up `/home/hermes/wiki` +> only — the **Obsidian vault mirror**, which is a **local-only** git repo with **no remote** +> (`git remote -v` is empty). The **authoring tree** that is actually pushed to Gitea +> (`Anthony/agent-estate-wiki`) lives at `/home/hermes/workspace/agent-estate-wiki` and is a +> **separate repo**. A tarball of `/home/hermes/wiki` does **not** protect the authoring tree. +> To protect both, run step 3 against each path, or back up the authoring tree and rely on Gitea. +> +> Also note `/home/hermes/wiki-backups/` does not yet exist — this runbook has never been run. + ## Procedure ### 1. Sanity-check repo state diff --git a/runbooks/diagnose-docker.md b/runbooks/diagnose-docker.md index b1aacca..bcf5a4f 100644 --- a/runbooks/diagnose-docker.md +++ b/runbooks/diagnose-docker.md @@ -3,9 +3,9 @@ title: Diagnose Failed Docker Service type: runbook status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 -last_tested: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 +last_tested: 2026-10-20 confidence: medium tags: [runbook, docker, homelab] sources: [] @@ -26,6 +26,15 @@ Diagnose container or compose issues on hosts where Docker is present. - Docker daemon running - Compose file path known for the affected project +> **Important scoping note, verified 2026-10-20: the `hermes` user is NOT in the `docker` group on +> CT460.** `systemctl is-active docker` → `active`, but `docker ps` as `hermes` returns +> **`permission denied while trying to connect to the docker API at unix:///var/run/docker.sock`**. +> Most Docker services in this estate run on **other hosts** (runtipi CT201, cloudreve CT209, +> LiteLLM CT500 on m93p, etc.) where the invoking account does have socket access. +> So: on CT460, either use `sudo docker …` or SSH to the host that actually owns the container. +> Note also that `systems/homepage-dashboard.md` now records the CT201 stack as **degraded** — +> the main dashboard container is gone — which is exactly the symptom this runbook would be used for. + ## Procedure ### 1. Check Docker daemon state diff --git a/runbooks/gateway-resource-resilience.md b/runbooks/gateway-resource-resilience.md index 8161606..14ff18d 100644 --- a/runbooks/gateway-resource-resilience.md +++ b/runbooks/gateway-resource-resilience.md @@ -3,9 +3,9 @@ title: Hermes Gateway Resource Resilience type: runbook status: active created: 2026-07-25 -updated: 2026-07-25 -review_after: 2026-10-23 -verified_on: 2026-07-25 +updated: 2026-10-20 +review_after: 2026-11-19 +verified_on: 2026-10-20 version_applies_to: Proxmox CT460 confidence: high tags: [hermes, infrastructure, runbook] @@ -50,12 +50,22 @@ File: ```ini [Service] -MemoryHigh=1G -MemoryMax=1536M -MemorySwapMax=768M +MemoryHigh=2G +MemoryMax=3G +MemorySwapMax=1G OOMPolicy=kill ``` +> **⚠ Limits have been RAISED since 2026-07-25 (verified 2026-10-20).** The Camofox drop-in now +> reads `MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G` — the page previously recorded +> `1G` / `1536M` / `768M`. Effective values confirmed via +> `systemctl --user show camofox-browser.service`: +> `MemoryHigh=2147483648` (2 GiB), `MemoryMax=3221225472` (3 GiB), +> `MemorySwapMax=1073741824` (1 GiB), `OOMPolicy=kill`. +> The `OOMPolicy=kill` intent is unchanged; only the ceilings moved. The Camofox service **is** +> running (`camofox-browser.service`, active/running) — it is the browser workload the containment +> is meant to protect, and it is the unit that grew. + ### Hermes Web UI File: @@ -72,6 +82,20 @@ OOMPolicy=kill Camofox and Web UI are allowed to restart under extreme growth instead of consuming most of the container and taking Telegram down with them. +> **Re-verified 2026-10-20 — the gateway and Web UI controls are still exactly as documented.** +> - Gateway drop-in `/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf` +> still reads `OOMPolicy=continue`, `MemoryLow=512M`, `CPUWeight=1000` (no `MemoryHigh`/`MemoryMax`). +> Effective: `OOMPolicy=continue`, `CPUWeight=1000`, `MemoryLow=536870912`, +> `MemoryHigh=infinity`, `MemoryMax=infinity` — **matches the "Expected gateway values" note below.** +> - Web UI: `OOMPolicy=kill`, `MemoryHigh=1073741824` (1 GiB), `MemoryMax=1610612736` (1.5 GiB), +> `MemorySwapMax=1073741824` (1 GiB) — **matches the drop-in shown above, unchanged.** +> - Camofox: see the raised limits noted above. +> - `hermes-gateway.service` and `camofox-browser.service` are both **active (running)**. +> +> **Note:** the Web UI drop-in lives at `/etc/systemd/system/hermes-webui.service.d/` which the +> `hermes` user cannot read (permission denied) — its *effective* values are still verifiable via +> `systemctl show`, which is what the health-check section below uses. + ## I/O priority limitation Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation @@ -102,7 +126,7 @@ runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \ Expected gateway values include `OOMPolicy=continue`, `MemoryLow=536870912`, `MemoryHigh=infinity`, `MemoryMax=infinity`, and -`CPUWeight=1000`. +`CPUWeight=1000`. **Confirmed live on 2026-10-20** — see the note under "Active controls". ## Backups and rollback diff --git a/runbooks/multiplexer-setup.md b/runbooks/multiplexer-setup.md index c1989fd..009ea8b 100644 --- a/runbooks/multiplexer-setup.md +++ b/runbooks/multiplexer-setup.md @@ -3,8 +3,8 @@ title: Multiplexer Setup (Multi-Profile Gateway) type: runbook status: active created: 2026-07-28 -updated: 2026-07-28 -verified_on: 2026-07-28 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [hermes, multiplexer, telegram, profiles, gateway] --- @@ -67,6 +67,28 @@ platforms: enabled: false # ← DISABLED — multiplexer handles this ``` +## ⚠ The `ops` profile was DELETED on 2026-10-11 (verified 2026-10-20) + +`/home/hermes/.hermes/profiles/` now contains **only** `.deleted/ops` (an 8-byte marker file whose +entire content is the word `deleted`, dated 2026-10-11). **`~/.hermes/profiles/ops/` no longer +exists** — there is no `config.yaml`, no `SOUL.md`, no profile directory. + +**But the default profile still routes to it.** `~/.hermes/config.yaml` still carries both +`profile_routes` entries (`ops-group` → `-1003914987043`, `ops-group-2` → `-1003932503629`, +both `profile: ops`), and `multiplex_profiles: true` is still set. The ops SOUL (Tusk, the African +elephant) and its `allowed_chats` list are gone with the directory. + +**Consequence:** messages arriving in the two ops chats have a route pointing at a profile that no +longer exists. Treat the "Ops profile" blocks below as **historical** — the second profile's +config can no longer be read at `~/.hermes/profiles/ops/config.yaml` because that path is gone. + +**Two things need a decision (not actioned here):** +1. Should `profile_routes` be removed, or should the `ops` profile be recreated? +2. Is the `multiplex_profiles: true` + two-route configuration still needed at all? + +Until that is settled, treat the ops chat IDs as **possibly unroutable** and verify the gateway +actually resolves them before relying on this runbook. + ## The Silent-Drop Pitfall The `allowed_chats` gate runs on the **default profile's adapter** BEFORE `profile_routes` stamps `source.profile`. If a chat is only authorized in the secondary profile's config but not the default's, messages from that chat are silently dropped. diff --git a/runbooks/recover-docker-service.md b/runbooks/recover-docker-service.md index f6da697..207f456 100644 --- a/runbooks/recover-docker-service.md +++ b/runbooks/recover-docker-service.md @@ -3,7 +3,8 @@ title: Recover a Docker service type: runbook status: active created: 2026-07-22 -updated: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 tags: [runbook, docker, recovery] sources: [] confidence: high @@ -18,3 +19,16 @@ confidence: high Container shows healthy after recovery probe reboots. ## Last successfully used 2026-07-22 + +> **Verified 2026-10-20 (procedure, not a live recovery).** Before running any `docker` command +> from CT460, note that the **`hermes` user has no access to `/var/run/docker.sock`** — +> `docker ps` returns `permission denied while trying to connect to the docker API`. Use `sudo`, +> or SSH to the host that owns the container. The containers in this estate mostly live on +> **CT201 (runtipi)**, **CT209 (cloudreve)** and **m93p CT500 (LiteLLM)** — not on CT460. +> +> **Two live situations this runbook would have been used for in the last week:** +> - The **CT201 homepage dashboard** is degraded — main container removed, `asset_proxy.py` on 8085 +> proxying to a dead 8082, and the Tailscale `/` route returning 502. See +> [[systems/homepage-dashboard]]. +> - The **Steel browser** service is unreachable (`steel.martinwa.org` → connection failure). +> See [[decisions/rollback-steel-to-camofox]]. diff --git a/runbooks/restart-browser.md b/runbooks/restart-browser.md index ee99561..9c76460 100644 --- a/runbooks/restart-browser.md +++ b/runbooks/restart-browser.md @@ -3,9 +3,9 @@ title: Restart Browser Helper type: runbook status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 -last_tested: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 +last_tested: 2026-10-20 confidence: high tags: [runbook, browser, cua-driver, camofox] sources: [] @@ -22,8 +22,12 @@ Restart the computer-use browser stack that Hermes uses for web automation, scre - VNC/noVNC is unreachable on `5901` / `6081` ## Prerequisites -- Display helper already running: `Xvfb :99`, x11vnc on `127.0.0.1:5901`, noVNC at `127.0.0.1:6081` -- `cua-driver` binary installed at `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` +- Display helper already running: `Xvfb :99`, x11vnc on `0.0.0.0:5901`, noVNC at `127.0.0.1:6081` +- `cua-driver` binary installed — **the release directory has moved on.** The path this runbook + used to hardcode (`.../releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver`) no longer exists; + `/home/hermes/.cua-driver/packages/releases/` now holds `0.19.3`, `0.22.2`, `0.23.2`, `0.24.0` + and **`0.28.2`**, and `/home/hermes/.local/bin/cua-driver` reports **cua-driver 0.28.2**. + Prefer the stable `/home/hermes/.local/bin/cua-driver` path over a versioned release path. - Browser VM/desktop helper is expected but not mandatory ## Procedure @@ -72,7 +76,12 @@ pkill -f "fluxbox" - The browser backend path can work without browser automation; this stack is optional depending on task ## Last tested -2026-07-22 +2026-07-22 — procedure re-checked against live processes 2026-10-20. `Xvfb :99`, `x11vnc -display :99 +-forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw`, `fluxbox` and both +websockify instances (`6080→5900`, `6081→5901`) are running, and the `vnc-watcher.sh` helper +(`/home/hermes/services/camofox-browser/plugins/vnc/vnc-watcher.sh`) is up. Note **5900 has no +listener** — the 6080 pair points at a dead port (Camofox's front door); the live display chain is +`:99 → 5901 → 6081`. See [[systems/headless-server]]. ## Related - [[browser-backend]] diff --git a/runbooks/restart-hermes.md b/runbooks/restart-hermes.md index a0d5d8d..cf5255f 100644 --- a/runbooks/restart-hermes.md +++ b/runbooks/restart-hermes.md @@ -3,9 +3,9 @@ title: Restart Hermes type: runbook status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 -last_tested: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 +last_tested: 2026-10-20 confidence: high tags: [runbook, hermes, restart] sources: [raw/configs/hermes-config-sanitized.txt] @@ -32,17 +32,35 @@ Restart the Hermes agent, gateway, dashboard, and browser helper processes clean ```bash ps aux | grep -E "hermes_cli.main (gateway|dashboard) run|hermes-webui/server.py|cua-driver serve|x11vnc.*5901" | grep -v grep ``` -Expected active pieces on this install: -- gateway: `/home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run` -- dashboard: `/home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main dashboard ...` -- webui: `/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py` +Expected active pieces on this install (live `ps` 2026-10-20): +- gateway: `hermes gateway run` → `/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 … gateway run` + (unit `hermes-gateway.service`) +- dashboard: `hermes dashboard --host 0.0.0.0 --port 9119 --no-open` (unit `hermes-dashboard.service`) +- webui: `/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py` (port 8787) - camofox/cua-driver: `/home/hermes/.local/bin/cua-driver serve` - x11vnc: `-display :99 ... 5901` +> **The `venv/` path in this runbook is wrong for the gateway and dashboard (verified 2026-10-20).** +> They no longer run from `~/.hermes/hermes-agent/venv/`; they run from the **uv-managed runtime** +> `~/.hermes/tools/python-3.14.7+20260901-linux-x64/`, launched through +> `hermes_cli/main.py` with a `runpy` bootstrap. The `hermes` launcher at +> `/home/hermes/.local/bin/hermes` is a 73-byte shim. **Prefer the `hermes` CLI and the systemd +> user units over hand-written `python -m hermes_cli.main …` commands** — the units already carry +> the correct interpreter path and the `_hermes_closure_repair()` bootstrap that protects against a +> half-finished `hermes update`. +> +> **Live user units (2026-10-20):** `hermes-gateway.service`, `hermes-dashboard.service`, +> `hermes-ui.service`, `hermes-ui-prod.service`, `hermes-wiki-serve.service` — all +> `active (running)`. + ### 2. Restart the gateway ```bash hermes gateway restart ``` +Equivalent and preferred (survives a broken CLI shim): +```bash +systemctl --user restart hermes-gateway.service +``` ### 3. Restart the dashboard if needed ```bash diff --git a/runbooks/restore-openrouter.md b/runbooks/restore-openrouter.md index 908932f..b3e984b 100644 --- a/runbooks/restore-openrouter.md +++ b/runbooks/restore-openrouter.md @@ -3,7 +3,8 @@ title: Restore OpenRouter connectivity type: runbook status: active created: 2026-07-22 -updated: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 tags: [runbook, openrouter, recovery] sources: [] confidence: high @@ -19,9 +20,27 @@ confidence: high 3. Restart the Hermes process with the GoOp tool. 4. Send a small test completion request. 5. Confirm tool calling works. + +> **⚠ This runbook is out of step with the current install (flagged 2026-10-20).** +> Steps 2 and 3 reference **WARP** and a **"GoOp tool"** — neither exists on this box. There is no +> WARP client configured and no `GoOp` binary; `hermes` has no such subcommand. The steps are +> carried over from a different install and **were not executed during the 2026-10-20 review**. +> +> **The procedure that actually applies here:** +> 1. Confirm OpenRouter is not a standalone provider in the first place — `custom_providers` in +> `~/.hermes/config.yaml` holds only `litellm`, `guacano`, `Omniroute:20128`, `command-code` and +> `omni-101`. See [[systems/openrouter]]. +> 2. If the failure is a *fallback* failure, use [[runbooks/test-openrouter]] and check the six +> entries of the live chain. +> 3. Restart with `systemctl --user restart hermes-gateway.service` (not "the GoOp tool"). +> 4. Send a small test completion and confirm tool calling works. +> +> **Last successfully used: 2026-06-22** — which predates every configuration referenced on this +> page. Treat the page as unverified until someone runs it end to end. ## Verification A test prompt returns successfully through OpenRouter. ## Escalation If the failure persists, check DNS, certificate expiry and provider status. ## Last successfully used -2026-06-22 +2026-06-22 — **120 days ago.** This is the single oldest verification date still in the wiki +alongside the original 2026-07-22 page set. diff --git a/runbooks/test-openrouter.md b/runbooks/test-openrouter.md index f65106a..a034514 100644 --- a/runbooks/test-openrouter.md +++ b/runbooks/test-openrouter.md @@ -3,9 +3,9 @@ title: Test OpenRouter type: runbook status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 -last_tested: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 +last_tested: 2026-10-20 confidence: high tags: [runbook, openrouter, providers, fallback] sources: [raw/configs/hermes-config-sanitized.txt] @@ -22,7 +22,8 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar - User wants to confirm OpenRouter is usable without relying on catalog names ## Prerequisites -- Hermes CLI available: `/home/hermes/.local/bin/hermes` +- Hermes CLI available: `hermes` on PATH (`/home/hermes/.local/bin/hermes` is a shim to the + uv-managed runtime — see the note under **Last tested**) - Internet egress for opencode.ai, inference-api.nousresearch.com, ollama.com, kilo.ai, litellm:4000 ## Procedure @@ -31,12 +32,22 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar ```bash hermes fallback list ``` -Expected active fallbacks on this install: -1. `deepseek-v4-flash-free` via opencode-zen -2. `stepfun/step-3.7-flash:free` via nous -3. `qwen3.5:397b` via ollama-cloud -4. `poolside/laguna-m.1:free` via kilocode -5. `or-inclusionai_ling-2_6-1t` via custom/litellm +Expected active fallbacks on this install (live `hermes fallback list`, 2026-10-20 — **6** entries): +1. `mimo-v2.5` via opencode-go +2. `deepseek-v4-flash-free` via opencode-zen — `https://opencode.ai/zen/v1` +3. `stepfun/step-3.7-flash:free` via nous — `https://inference-api.nousresearch.com/v1` +4. `qwen3.5:397b` via ollama-cloud — `https://ollama.com/v1` +5. `poolside/laguna-m.1:free` via kilocode — `https://api.kilo.ai/api/gateway` +6. `or-inclusionai_ling-2_6-1t` via custom — `http://litellm:4000/v1` + +**Primary:** `meituan/longcat-2.5-preview:free` (via nous). + +> **Changes since 2026-07-22:** `mimo-v2.5` via **opencode-go** has been **added at position 1** of the +> chain, and the primary changed to `meituan/longcat-2.5-preview:free` (was `freellmapi-auto`). +> The other five entries and their order are unchanged. +> `hermes fallback list` also prints a standing notice on every invocation: +> *"Claude Subscription DirectSDK (Experimental): Claude Code is not installed"* — expected on this +> box, since `claude` is not on PATH. It is a notice, not a failure of the fallback chain. ### 2. Test OpenRouter-style direct access if present ```bash @@ -77,7 +88,12 @@ Verify provider names and URLs in config match what the user wants to use. - This runbook tests confirmed fallback paths, not assumed ones ## Last tested -2026-07-22 +2026-07-22 — re-run 2026-10-20 against live `hermes fallback list` output (6 entries, primary +`meituan/longcat-2.5-preview:free`). Note the CLI reports Hermes Agent **v0.22.2 (2026.9.24), +local `d923a5eb487a`**, and the `/home/hermes/.local/bin/hermes` launcher is a 73-byte shim that +delegates to the uv-managed runtime at +`/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/` — *not* the +`~/.hermes/hermes-agent/venv/` path named in some older runbooks. Use the `hermes` on PATH. ## Related - [[provider-health]] diff --git a/runbooks/troubleshoot-tool-loop.md b/runbooks/troubleshoot-tool-loop.md index 98bd84a..90a8ed9 100644 --- a/runbooks/troubleshoot-tool-loop.md +++ b/runbooks/troubleshoot-tool-loop.md @@ -3,24 +3,99 @@ title: Troubleshoot Tool Loop type: runbook status: unresolved created: 2026-07-22 -updated: 2026-07-22 -review_after: 2026-08-22 -confidence: low +updated: 2026-10-20 +review_after: 2026-11-19 +verified_on: 2026-10-20 +confidence: medium tags: [runbook, hermes] sources: [] --- -> **Stub — structure only.** Headings exist but content has not been verified/written yet. Flagged during 2026-07-22 wiki audit; fill in with verified facts or mark deprecated if no longer relevant. +> **Filled in 2026-10-20** from live configuration, live process state and two recorded incidents. +> The "stub" note below is retained as history. # Troubleshoot Tool Loop + ## Symptoms -Agent loops on one tool or endpoint. -## Procedure -1. Inspect recent Hermes agent logs. -2. Identify the looping tool path. -3. Restart worker or flush stuck request. -4. Re-run limited test scenario. -## Verification -Agent handles the same scenario without repeating. +- The agent repeats the **same tool call** with identical arguments and makes no progress. +- The agent repeats **itself in the response stream** (no tool involved) — the model loops on its + own output and Hermes cuts the stream. + +**These are two different faults with two different detectors.** Do not assume one covers the other. + +## 1. Tool-call loop — detected by `tool_loop_guardrails` + +### Symptoms +Agent loops on one tool or endpoint. Hermes emits a warning naming the tool. + +### Configuration (verified live 2026-10-20, `~/.hermes/config.yaml`) +```yaml +tool_loop_guardrails: + warnings_enabled: true + hard_stop_enabled: false + warn_after: + exact_failure: 2 + same_tool_failure: 3 + idempotent_no_progress: 2 + hard_stop_after: + exact_failure: 5 + same_tool_failure: 8 + idempotent_no_progress: 5 +``` +> **`hard_stop_enabled: false` is the key fact.** Hermes will *warn* about a stuck tool loop but +> will **not** force-stop itself. A genuinely stuck loop only gets a warning — a human (or the +> messaging integration) is the real backstop. See [[concepts/tool-calling]]. + +### Procedure +1. Inspect recent Hermes agent logs: `~/.hermes/logs/agent.log*` (grep for the tool name). +2. Identify the looping tool path. Common culprits in this estate: a browser tool against a dead + Camofox service, a terminal tool against a stale persistent shell, or a webhook that returns + 200 with an empty body. +3. **Check the dependency, not just the agent.** On this estate the most frequent tool-loop cause + was an upstream service being down while the tool still returned a response. Verify with the + relevant runbook: [[runbooks/restart-browser]], [[runbooks/restart-hermes]], + [[runbooks/provider-health]]. +4. If the loop is a stuck worker, restart it: `systemctl --user restart hermes-gateway.service`. +5. Re-run a limited test scenario and confirm the guard no longer fires. + +### Verification +Agent handles the same scenario without repeating, and no `tool_loop_guardrails` warning appears +in `agent.log`. + +## 2. Model-output (degenerate stream) loop — detected by the stream guard + +### Symptoms +The run **delivers a canned non-answer** such as *"Your request was not processed. Send it again +if you still want me to carry it out."*, and then fails with +`RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding +the degenerate response.` + +### This is the 2026-10-18 case — see +[[incidents/2026-10-18-degenerate-stream-canned-substitute]] + +### Procedure +1. Read the run's `error` in `~/.hermes/cron/executions.db` — the phrase `repetition loop` + identifies this fault. +2. **Check `delivery_outcome` on the same row.** In the 2026-10-18 incident the row was + `delivery_outcome: delivered` — the canned non-answer reached Telegram *before* the run + failed. **A delivered message is not evidence of delivered content.** +3. Confirm the provider is healthy: the 2026-10-18 case rode the same `custom:omni-101` alias + that produced the 2026-10-14/15 connection-refused storm and the 2026-10-16/17 and 2026-10-19 + resource-pressure 503 bursts. See [[runbooks/provider-health]] and + [[incidents/2026-10-16-omniroute-503-resource-pressure]]. +4. Do **not** lower `tool_loop_guardrails` thresholds for this fault — they watch tool calls, and + the degenerate-stream guard is a separate mechanism. + +### Verification +The next scheduled run of the job completes with `status: completed` and `error: null`, and the +delivered message contains real content. + ## Last successfully used -2026-07-22 +2026-07-22 — re-verified against live config and live incidents on 2026-10-20; the *procedure* has +not itself been executed end-to-end in a live loop, so `confidence` was raised only to `medium`. + +## Related +- [[concepts/tool-calling]] +- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] +- [[runbooks/provider-health]] +- [[runbooks/restart-hermes]] diff --git a/systems/cloudflare-tunnel.md b/systems/cloudflare-tunnel.md index 15c4f4c..4cead1e 100644 --- a/systems/cloudflare-tunnel.md +++ b/systems/cloudflare-tunnel.md @@ -3,8 +3,8 @@ title: Cloudflare Tunnel type: system status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [system, network, cloudflare] sources: [raw/configs/hermes-config-sanitized.txt] @@ -26,6 +26,12 @@ Exposes internal homelab services, including Hermes, to the public internet with - `originRequest`: `noTLSVerify: true`, `disableChunkedEncoding: true`, `http2Origin: false` (global defaults; some hostnames override per-entry) - Ingress is a flat list of `hostname → service` mappings ending in `service: http_status:404` catch-all — see [[infrastructure/domains-and-tunnels]] for the full list and the entries specific to Hermes. - Two Hermes-relevant entries point at this host's Tailscale IP (`100.118.5.51`): `hermes.martinwa.org` (port 19119) and `hermes-desktop.martinwa.org` (port 5174). +- **Re-verified 2026-10-20:** `cloudflared.service` is still **active** on ct207, the tunnel name/ID + (`ct207-tunnel` / `339a6757-5574-4723-a16c-da4769535caa`) and the credentials-file path are unchanged, + and the global `originRequest` defaults still read `noTLSVerify: true`, `disableChunkedEncoding: true`, + `http2Origin: false`. The Hermes-relevant ingress entries have since **expanded to four**: + `hermes.martinwa.org` (9119), `hermes-chat.martinwa.org` (8787), `tg.hermes.martinwa.org` (19119) + and `hermes-desktop.martinwa.org` (5174). Full live list in [[infrastructure/domains-and-tunnels]]. ## Connected systems - [[systems/hermes-agent]] — the service actually reached through this tunnel @@ -34,7 +40,7 @@ Exposes internal homelab services, including Hermes, to the public internet with ## Known issues - This host cannot restart or inspect the tunnel directly (no `cloudflared` installed here); a tunnel-side problem requires access to ct207. -- `mcp-portal.martinwa.org`, referenced in Hermes's own config as an MCP endpoint, was not found in the ct207 ingress list captured 2026-07-22 — **unresolved**, may be on a different tunnel/host. +- `mcp-portal.martinwa.org`, referenced in Hermes's own config as an MCP endpoint, was not found in the ct207 ingress list captured 2026-07-22 — **unresolved**, may be on a different tunnel/host. Re-checked against the live ct207 `config.yml` on 2026-10-20: **still absent**, still unresolved. ## Relevant runbooks - [[runbooks/diagnose-docker]] (closest existing runbook; no dedicated tunnel-restart runbook exists yet — worth writing one that documents `systemctl restart cloudflared` on ct207, since Hermes has no direct access to perform it) diff --git a/systems/guanaco.md b/systems/guanaco.md index 834d620..f6922de 100644 --- a/systems/guanaco.md +++ b/systems/guanaco.md @@ -3,8 +3,8 @@ title: Guanaco type: system status: active created: 2026-08-14 -updated: 2026-08-14 -verified_on: 2026-08-14 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [llm, proxy, router, guanaco, ollama] sources: [live-probe, session-history] @@ -17,14 +17,18 @@ LLM proxy/router providing model routing and tool-call formatting for the Ollama ## Current Configuration - **Host:** CT205 (`ollama` LXC container) on Proxmox -- **Version:** v0.8.14 (updated from v0.8.11 on 2026-08-14) +- **Version:** v0.8.15 (updated from v0.8.14 on 2026-08-14; verified at tag `v0.8.15`, commit `5df435e`, 2026-08-14 16:06:13 UTC) - **Repo path (on CT205):** `/root/.guanaco/repo` - **API endpoint:** `ollama:11434` (DNS) / `localhost:11434` - **Key component:** `cmdcode_client.py` — Guanaco's tool-call formatting layer (DSML) ## Version History -### v0.8.14 (current, updated 2026-08-14) +### v0.8.15 (current, updated 2026-08-14) +- **Changelog bump + DSML fixes** — HEAD is tagged `v0.8.15` (commit `5df435e`, "chore: bump to v0.8.15 + changelog"), built from `origin/main` / `origin/develop`. +- Live probe 2026-10-20: `/root/.guanaco/repo` on CT205 at tag `v0.8.15`; Ollama API on `localhost:11434` answers **200**. + +### v0.8.14 (previous, 2026-08-14) - **DSML tag leakage fixes** (cumulative across v0.8.12–v0.8.14): - v0.8.12 — incomplete DSML blocks leaking raw tags as content - v0.8.13 — comprehensive DSML tag leakage: double-pipe, bare fragments, solo `invoke` diff --git a/systems/headless-server.md b/systems/headless-server.md index aaebb20..756462d 100644 --- a/systems/headless-server.md +++ b/systems/headless-server.md @@ -3,8 +3,8 @@ title: Headless Server type: system status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [system, browser] sources: [raw/configs/hermes-config-sanitized.txt] @@ -27,7 +27,17 @@ Headless X11 environment backing Hermes's browser automation and any GUI tooling | `x11vnc` | `-display :99 -forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw` — **no VNC password set** | | `websockify` ×2 | `127.0.0.1:6080 → 127.0.0.1:5900` and `127.0.0.1:6081 → 127.0.0.1:5901`, serving noVNC (`/usr/share/novnc`) | -Two VNC/websockify pairs exist (5900/6080 and 5901/6081) — the second (`:99`→5901→6081) is the one confirmed above; the first (5900→6080) predates it and wasn't independently re-verified this pass — **unresolved** whether it's a second live display or a leftover process. +**Re-verified 2026-10-20.** Two websockify instances still run (6080→5900 and 6081→5901), but +**only one Xvfb exists** (`:99`). Port **5900 has no listener at all** — the 6080 websockify points +at a dead port, so it is a **leftover** from an earlier display, not a second live one. The 6080 +pair is the **Camofox** noVNC front door (`/home/hermes/services/camofox-browser/plugins/vnc/vnc-watcher.sh`), +which proxies to 5900; Camofox's REST API is separately live on `:9377` (HTTP 200). +The live chain is therefore **`:99` → x11vnc 5901 → websockify 6081**. + +Also confirmed still true in 2026-10-20: `x11vnc` still runs with `-nopw` (no VNC password) and +still binds **all interfaces** — `0.0.0.0:5901` and `[::]:5901` — while both websockify ports +are correctly loopback-only (`127.0.0.1:6080`, `127.0.0.1:6081`). + ## Connected systems - [[systems/hermes-agent]] (browser backend uses this display — see [[systems/browser-backend]] / [[comparisons/camoufox-vs-chrome]]) diff --git a/systems/hermes-ui.md b/systems/hermes-ui.md index d55f89a..a1e7900 100644 --- a/systems/hermes-ui.md +++ b/systems/hermes-ui.md @@ -3,8 +3,8 @@ title: hermes-ui (Browser Hermes Desktop) type: system status: live created: 2026-07-30 -updated: 2026-07-30 -verified_on: 2026-07-30 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [hermes, webui, desktop, browser] --- @@ -25,6 +25,12 @@ The official Hermes Desktop renderer, extracted from the `hermes-agent` monorepo | CT460 (this box) | `/home/hermes/hermes-ui/` | 5174 | Vite dev server | Active | | CT450 (hermesagent) | `/home/hermes/hermes-ui/` | 5174 | Vite dev server | Active (since 2026-07-30) | +> **Re-verified 2026-10-20 — both deployments are up.** CT460: `0.0.0.0:5174` held by a `node` +> process; CT450: `0.0.0.0:5174` confirmed by `ss -ltn` inside CT450, and `/home/hermes/hermes-ui` +> exists there. Note **5175** on CT460 is a *different* service — a `python3` on `127.0.0.1:5175` +> (the `/ui` Tailscale Serve route) — and **8787** is the Hermes WebUI (`python`, `0.0.0.0:8787`, +> the `/webui` route). Don't conflate 5174 / 5175 / 8787. + ## How It Works - Runs as a Vite dev server, proxying REST/auth/WebSocket to a running Hermes gateway (`http://127.0.0.1:9119` by default). diff --git a/systems/homepage-dashboard.md b/systems/homepage-dashboard.md index 3c8e976..1bda802 100644 --- a/systems/homepage-dashboard.md +++ b/systems/homepage-dashboard.md @@ -1,9 +1,10 @@ --- title: Homepage Dashboard type: systems -status: active +status: degraded created: 2026-07-26 -updated: 2026-07-26 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [dashboard, homepage, homelab, proxmox, monitoring] --- @@ -13,18 +14,35 @@ tags: [dashboard, homepage, homelab, proxmox, monitoring] ## Purpose Central dashboard for all homelab services, with Proxmox integration, per-VM stats, and a management UI. -## URLs -- **Tailscale:** `https://runtipi.kangaroo-eel.ts.net:8443/` -- **Local:** `http://runtipi.kangaroo-eel.ts.net:8082/` -- **Manager UI:** `http://runtipi.kangaroo-eel.ts.net:8084/` +## ⚠ Status: DEGRADED (verified 2026-10-20) +The stock **`homepage` container is not running** on CT201. Only `homepage-manager` (port 8084) +and `homebox` are up; `homebridge` exited two months ago. **Port 8082 has no listener**, and the +`asset_proxy.py` front door on port 8085 proxies to 8082 — so it answers **502 Connection refused**. -## Architecture +The dashboard that *is* live and serving **200** is **Homarr** on port **8102** +(`homar.martinwa.org` in the Cloudflare tunnel ingress, and `homarr` in the Runtipi app list). +The dashboard content itself is now **custom Python tooling** under `/opt/homepage/` +(`manager.py`, `asset_proxy.py`, `dashboard.html/js/css`), not the upstream Homepage image. + +**Two independent dashboards are therefore in play — do not assume "the dashboard" means Homepage.** + +## URLs +- **Homarr (live, 200):** `http://100.96.244.39:8102/` — public via `https://homar.martinwa.org` +- **Tailscale:** `https://runtipi.kangaroo-eel.ts.net:8443/` → proxies to 8085 → **currently 502** +- **Local:** `http://100.96.244.39:8082/` → **no listener** +- **Manager UI:** `http://runtipi.kangaroo-eel.ts.net:8084/` → `homepage-manager`, up + +## Architecture (verified 2026-10-20) ``` CT201 (runtipi) -├── Homepage container (port 8082) — main dashboard -├── Manager container (port 8084) — web UI for adding/removing apps -└── Tailscale serve (port 8443) — HTTPS proxy to Homepage +├── homarr container (port 8102) — live dashboard, HTTP 200 +├── homepage-manager container (8084) — web UI for adding/removing apps, up +├── homebox container (7745) — up +├── asset_proxy.py (python3, port 8085) — proxies 8085 → 8082, currently 502 +├── manager.py (python3) — dashboard management service, up +└── Tailscale serve (port 8443) — / → 8085, /mgr → 8084, /steel → 5173, /steel-api → 3000 ``` +No container is listening on **8082** — `docker ps` on CT201 lists no `homepage` container at all. ## Tabs | Tab | Content | @@ -44,39 +62,43 @@ CT201 (runtipi) - **proxmox.yaml:** Config at `/opt/homepage/config/proxmox.yaml` ### VM/LXC Mapping -| Service | VMID | Type | -|---|---|---| -| PVE | (node) | — | -| Portainer docker | 200 | lxc | -| Portainer debianVM | 101 | qemu | -| Portainer openweb | 245 | lxc | +> **Two Proxmox hosts exist and guest IDs collide** — always qualify with a host. +> `ct460` = PVE `100.96.100.82`; `m93p` = PVE `100.104.51.98`. The mapping below was re-read from +> `pct list` on **both** hosts on 2026-10-20; several rows previously listed bare IDs now live on +> `m93p`, and one row no longer exists at all. + +| Service | Host | VMID | Type | Live state | +|---|---|---|---|---| +| PVE | both | (node) | — | — | +| Portainer docker | ct460 | 200 | lxc | running (8000, 9443) | +| Portainer debianVM | ct460 | 101 | qemu | running | +| Portainer openweb | **m93p** | 245 | lxc | running | +| Runtipi | ct460 | 201 | lxc | running | +| Gitea | **m93p** | 221 | lxc | running | +| FreshRSS | **m93p** | **1260** | lxc | running | +| NextCloud | ct460 | 270 | lxc | running | +| LiteLLM | **m93p** | 500 | lxc | running | +| OmniRoute | ct460 | 600 | lxc | running (also `Omniroute:20128` on m93p 101) | +| Ntfy | **m93p** | **1241** | lxc | running | +| Home Assistant | ct460 | 100 | qemu | running | +| N8n | ct460 | 215 | lxc | running | +| Mattermost | ct460 | 255 | lxc | **stopped** | +| SFTPGo | ct460 | 299 | lxc | running | +| Cliproxy API | ct460 | 103 | lxc | running | +| OpenClaw | — | — | qemu | **absent — no VM 403 on either host** | +| Zeroclaw | ct460 | 333 | lxc | running | +| Hermes | ct460 | 460 | lxc | running | +| Hindsight | ct460 | 450 | lxc | running | > Portainer on CT245 (`openweb.kangaroo-eel.ts.net:9443`) now uses a Tailscale/Let's Encrypt cert (installed 2026-08-09). Files: `/root/portainer-certs/portainer.{crt,key}` mounted into the container, `--sslcert/--sslkey`. Renewal: `/root/renew-portainer-cert.sh` via crontab (04:00 on the 1st monthly) — `tailscale cert --min-validity=20d` then `docker restart portainer`. Cert validity ~90 days. -| Runtipi | 201 | lxc | -| Gitea | 221 | lxc | -| FreshRSS | 260 | lxc | -| NextCloud | 270 | lxc | -| LiteLLM | 500 | lxc | -| OmniRoute | 600 | lxc | -| Ntfy | 241 | lxc | -| Home Assistant | 100 | qemu | -| N8n | 215 | lxc | -| Mattermost | 255 | lxc | -| SFTPGo | 299 | lxc | -| Cliproxy API | 103 | lxc | -| OpenClaw | 403 | qemu | -| Zeroclaw | 333 | lxc | -| Hermes | 460 | lxc | -| Hindsight | 450 | lxc | ## Configuration Files -All at `/opt/homepage/config/` on CT201: -- `services.yaml` — All apps, groups, and Proxmox mappings -- `settings.yaml` — Theme, layout, tabs -- `custom.css` — Warm amber accent overrides -- `widgets.yaml` — System health, clock -- `proxmox.yaml` — PVE API token -- `docker.yaml` — Docker socket integration +`/opt/homepage/` on CT201 now holds **custom Python tooling**, not the upstream Homepage bundle: +- `manager.py` — dashboard management service (running) +- `asset_proxy.py` — port 8085 → 8082 front door (running, currently 502 upstream) +- `dashboard.html` / `dashboard.js` / `dashboard.css` — the custom dashboard frontend +- `config/` — legacy Homepage config directory (still present, upstream `homepage` container not running) +- `manager.py.bak.*` — a long tail of dated manager backups (brand, chat2, cookie, hp, motion, owtools …) ## Manager UI Web form at port 8084 for adding/removing apps without SSH. @@ -96,7 +118,15 @@ Type anywhere on the dashboard to search services by name/description. Google se Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead. ## Maintenance -- **Restart Homepage:** `docker restart homepage` - **Restart Manager:** `docker restart homepage-manager` -- **Edit apps manually:** Edit `/opt/homepage/config/services.yaml` then restart -- **Docker auto-discovery:** Add `homepage.group`, `homepage.name`, `homepage.href` labels to containers +- **Restart Homarr:** `docker restart homarr` +- **Restart the custom dashboard:** the `manager.py` / `asset_proxy.py` processes run directly on CT201 + (not in Docker) — restart via whatever supervises them; check `ps aux | grep -E 'manager.py|asset_proxy'` +- **Recover the upstream Homepage (8082):** the image `ghcr.io/gethomepage/homepage` is still present on + CT201 (4 months old) but no container uses it — recreate it to restore 8082 and the 8085 proxy path +- **Tailscale serve on CT201:** `/` → 8085, `/mgr` → 8084, `/steel` → 5173, `/steel-api` → 3000 + +## Recent Changes +- **2026-10-20 (live probe):** `homepage` container absent, 8082 dead, 8085 → 502, Homarr on 8102 → 200. + Status changed `active` → `degraded`. VM/LXC table re-read from both PVE hosts; corrected IDs and + host assignments, dropped the non-existent OpenClaw VM 403, marked Mattermost CT255 stopped. diff --git a/systems/nextcloud.md b/systems/nextcloud.md index 15a90f3..d5d4d76 100644 --- a/systems/nextcloud.md +++ b/systems/nextcloud.md @@ -3,8 +3,8 @@ title: Nextcloud type: system status: active created: 2026-08-17 -updated: 2026-08-17 -verified_on: 2026-08-17 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [nextcloud, file-sharing, self-hosted, infrastructure] sources: [] @@ -28,6 +28,14 @@ Self-hosted file sync and sharing platform running on Proxmox LXC container. |---|---|---|---| | memories | 8.1.0 | active | Photo management | | metadata | 0.24.0 | active | | + +> **Re-verified 2026-10-20** — every fact on this page still holds: CT270 `nextcloud` is running, +> `occ status` reports **version 33.0.2.2** (installed, not in maintenance, no DB upgrade needed), +> the container holds `192.168.178.56` (LAN) and `100.90.102.19` (Tailscale), and both +> `memories 8.1.0` and `metadata 0.24.0` are enabled. `https://nextcloud.martinwa.org/` answers +> **302** both publicly and via the Tailscale IP. `occ` lives at `/var/www/nextcloud/occ` +> (not `/var/www/html/nextcloud`). +> Note: `occ` must be run from that path and as `www-data`. | ncdownloader | 1.0.23 | **disabled** | Incompatible with NC33 — calls removed method `OC\Server::getAppConfig()` | | news | 28.7.0 | active | RSS reader | | nextcloud_announcements | 5.0.0 | active | | diff --git a/systems/obsidian.md b/systems/obsidian.md index 9fefa82..90e4f5d 100644 --- a/systems/obsidian.md +++ b/systems/obsidian.md @@ -3,7 +3,8 @@ title: Obsidian Vault type: systems status: active created: 2026-07-26 -updated: 2026-07-26 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [obsidian, notes, vault, homelab] --- @@ -19,6 +20,21 @@ Personal knowledge base — notes, research, projects, and agent documentation. - **Structure:** NUMBERED folders (00-99) - **Sync:** Git-based (manual or cron-triggered) +> **Re-verified 2026-10-20 — the "NUMBERED folders (00-99)" description is no longer accurate.** +> On disk there are now only two vault-shaped directories on CT460: +> - `/home/hermes/obsidian-vault/` — contains `50 - Research/` (with `Restaurants and Cafes/`) and +> `Cron Outputs/` (with `Weekend Adventure/`). **No `00 - Home`, no `10 - Inbox`, no `20 - Agent Notes`.** +> This directory is **not a git repo** (`git -C` → "not a git repository"). +> - `/home/hermes/wiki/` — a **local-only** git repo (no remote configured) whose tracked content is +> the `agent-estate-wiki/` mirror plus `reports/uber-eats/`. Its `.obsidian/` dir is under +> `wiki/agent-estate-wiki/.obsidian`. +> +> So the numbered-folder tree described below is **largely absent**, and the vault that *is* mirrored +> to Gitea is the **agent-estate-wiki** repo (`Anthony/agent-estate-wiki`), not `Anthony/obsidian-vault`. +> Treat the folder table as historical. Separately, an **Obsidian LiveSync** service runs on **CT100 +> (`obsidian-livesync`) on the `m93p` host** (`/opt/obsidian-livesync/`, holds only a `.env`). +> Full folder table, left for reference: + ## Folder Structure | Folder | Purpose | |---|---| diff --git a/systems/opencode-go.md b/systems/opencode-go.md index 7736825..8ca957b 100644 --- a/systems/opencode-go.md +++ b/systems/opencode-go.md @@ -3,8 +3,8 @@ title: OpenCode Go type: system status: active created: 2026-07-29 -updated: 2026-07-29 -verified_on: 2026-07-29 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [opencode, provider, models, subscription, limits] sources: [https://opencode.ai/docs/go/] @@ -22,6 +22,12 @@ Low cost subscription for open coding models — $5 first month, then **$10/mont > Only one member per workspace can subscribe. +> **Note (2026-10-20):** the pricing and dollar-limit tables below are **vendor-published figures +> for the OpenCode Go product**, not a measurement of this estate. Nothing in `~/.hermes/config.yaml` +> or `jobs.json` shows an OpenCode Go provider in use — the estate routes through LiteLLM, +> OmniRoute (`omni-101`) and `Omniroute:20128`. Treat this page as reference material for the +> subscription, not as a description of a configured provider. + ## Usage Limits | Window | Dollar Limit | diff --git a/systems/openrouter.md b/systems/openrouter.md index c8fa4a1..659ce0c 100644 --- a/systems/openrouter.md +++ b/systems/openrouter.md @@ -3,8 +3,8 @@ title: OpenRouter type: system status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: medium tags: [system, openrouter, provider] sources: [raw/configs/hermes-config-sanitized.txt] @@ -21,6 +21,14 @@ Primary key, fallback models, rate limits. OpenRouter is not configured as a standalone provider in `~/.hermes/config.yaml` `custom_providers`. OmniRoute’s catalog includes provider names and model IDs consistent with OpenRouter-style slugs, but live routing via OpenRouter directly is not present in current Hermes config. +> **Re-verified 2026-10-20 — the claim above still holds.** `custom_providers` in +> `/home/hermes/.hermes/config.yaml` is a list of exactly five entries: +> `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101`. There is **no `openrouter` +> entry**. OpenRouter-style slugs (`openrouter/auto`, `freeapi/openrouterfree`, +> `kilo-openrouter-auto`, `or-openrouter-fusion`, and many more) exist only inside the **OmniRoute +> model catalog**, not as directly-routable Hermes providers. Routing to OpenRouter therefore still +> goes through OmniRoute, exactly as recorded here. + ## History - **User-provided:** Previously used; now largely replaced by LiteLLM and OmniRoute paths - **Verified:** Decision page `keep-openrouter-as-fallback` exists, indicating retain intent diff --git a/systems/skills-index.md b/systems/skills-index.md index c49d3f4..95be9e3 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-10-19 -verified_on: 2026-10-19 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [skills, hermes] sources: [] diff --git a/systems/terminal-backend.md b/systems/terminal-backend.md index be15b89..96e3c03 100644 --- a/systems/terminal-backend.md +++ b/systems/terminal-backend.md @@ -3,8 +3,8 @@ title: Terminal Backend type: system status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [terminal, shell, backend, persistence] sources: [raw/configs/hermes-config-sanitized.txt] @@ -17,12 +17,17 @@ Execute shell commands, scripts, and interactive sessions for Hermes. ## Current Configuration - **Backend:** local -- **Working directory:** `/home/hermes/workspace` +- **Working directory:** `/home/hermes/workspace/` - **Timeout:** 180 s - **Home mode:** auto - **Container CPU:** 1 - **Persistent shell:** true +Live `terminal:` block re-read from `/home/hermes/.hermes/config.yaml` on 2026-10-20 also carries: +`lifetime_seconds: 300`, `container_memory: 5120`, `container_disk: 51200`, +`container_persistent: true`, `docker_mount_cwd_to_workspace: false`, `docker_run_as_host_user: true`. +`backend: local` and `persistent_shell: true` are unchanged since 2026-07-22. + ## Persistence - Shell state (venv, env vars, aliases) persists across terminal calls - Working directory fixed to `/home/hermes/workspace` diff --git a/systems/tool-search.md b/systems/tool-search.md index a0ba67e..dfff7a6 100644 --- a/systems/tool-search.md +++ b/systems/tool-search.md @@ -3,9 +3,9 @@ title: Hermes Tool Search type: system status: active created: 2026-07-25 -updated: 2026-07-25 -review_after: 2026-10-23 -verified_on: 2026-07-25 +updated: 2026-10-20 +review_after: 2026-11-20 +verified_on: 2026-10-20 version_applies_to: Hermes Agent on CT460 confidence: high tags: [hermes, automation, system] @@ -32,11 +32,19 @@ File: `/home/hermes/.hermes/config.yaml` tools: tool_search: enabled: "on" - threshold_pct: 10 - search_default_limit: 5 - max_search_limit: 20 + threshold_pct: 20 + search_default_limit: 8 + max_search_limit: 50 + listing_max_tokens: 8 ``` +> **Live config re-read 2026-10-20** — the values above are the current ones in +> `/home/hermes/.hermes/config.yaml`. They have drifted from the original 2026-07-25 rollout: +> `threshold_pct` 10 → **20**, `search_default_limit` 5 → **8**, `max_search_limit` 20 → **50**, +> and `listing_max_tokens: 8` has been added. `enabled: "on"` is unchanged. +> The token-budget figures in "Verified result" below were measured under the *original* settings +> and have not been re-measured since the tuning. + `enabled: "on"` is the important setting: it makes Tool Search active on every request instead of waiting for the tool schemas to cross an automatic token threshold. diff --git a/wiki-maintain.sh b/wiki-maintain.sh index 00a2c98..dedc07f 100755 --- a/wiki-maintain.sh +++ b/wiki-maintain.sh @@ -142,21 +142,65 @@ python3 "$STOCKTAKE_DIR/_freshness_audit.py" "$WIKI_DIR" "$REPORT" 21 echo "" # ============================================================================= -# [3/3] Rebuild Quartz static site +# [3/3] Rebuild Quartz static site and deploy to public/ # ============================================================================= -echo "[3/3] Checking Quartz build..." -# Quartz is already built into public/ — if there's a build script, run it -# For now, the public/ dir is the Quartz output, regenerated when content changes -# If Quartz CLI is available, we'd run: npx quartz build -if command -v npx &> /dev/null && [ -f "$WIKI_DIR/quartz.config.ts" ]; then - echo " Running Quartz build..." - cd "$WIKI_DIR" - npx quartz build 2>&1 | head -20 || echo " Quartz build had issues (non-fatal)" -else - echo " Quartz build skipped (no quartz.config.ts or npx not available)" - echo " public/ directory contains the current static site" +# Quartz v4/v5 lives in a SEPARATE source tree (/home/hermes/workspace/quartz-temp); +# the wiki repo only holds the Markdown. The build reads content/ from that tree +# and emits public/ there, which we then deploy into the wiki repo's public/. +# +# PITFALL (hit 2026-10-20): the previous version of this step tested for +# `quartz.config.ts` and `npx`, and the repo has NEITHER — Quartz v4/v5 uses +# `quartz.config.yaml`. The test always failed, so the site silently stopped +# being rebuilt (published site was 40 days stale). Do not reintroduce a +# config-file existence test that isn't the real one: check quartz.config.yaml. +echo "[3/3] Rebuilding Quartz static site..." + +QUARTZ_DIR="/home/hermes/workspace/quartz-temp" +QUARTZ_CONTENT="$QUARTZ_DIR/content" +QUARTZ_PUBLIC="$QUARTZ_DIR/public" +QUARTZ_DEPLOY_DIR="$WIKI_DIR/public" + +if [ ! -f "$QUARTZ_DIR/quartz.config.yaml" ]; then + echo " ERROR: $QUARTZ_DIR/quartz.config.yaml not found — cannot build the Quartz site." >&2 + echo " The published site will stay stale. Fix the Quartz source tree before continuing." >&2 + exit 1 fi +# --- sync wiki Markdown into the Quartz content tree ------------------------- +# raw/ is excluded on purpose: it holds raw config dumps and the stocktake +# reports, which have never been part of the published site. public/ and .git +# are excluded to avoid copying build output into the input tree. +rm -rf "$QUARTZ_CONTENT" +mkdir -p "$QUARTZ_CONTENT" +tar -C "$WIKI_DIR" \ + --exclude=./public --exclude=./.git --exclude=./node_modules --exclude=./raw \ + -cf - . | tar -C "$QUARTZ_CONTENT" -xf - +echo " Synced wiki content into $QUARTZ_CONTENT" + +# --- build ------------------------------------------------------------------- +cd "$QUARTZ_DIR" +# PIPESTATUS is required: a plain pipeline would report tail's exit code (0) +# and silently swallow a failed build. +npx quartz build 2>&1 | tail -8 +BUILD_RC=${PIPESTATUS[0]} +if [ "$BUILD_RC" -ne 0 ]; then + echo " ERROR: Quartz build failed (exit $BUILD_RC)." >&2 + exit 1 +fi + +# --- deploy ------------------------------------------------------------------ +# Clear the CONTENTS of public/ rather than removing the directory itself: +# hermes-wiki-serve.service runs with its cwd inside public/, so replacing the +# directory inode leaves the running server serving a deleted directory (all +# requests 404) until it is restarted. +find "$QUARTZ_DEPLOY_DIR" -mindepth 1 -delete +cp -a "$QUARTZ_PUBLIC/." "$QUARTZ_DEPLOY_DIR/" +echo " Deployed $(find "$QUARTZ_DEPLOY_DIR" -type f | wc -l) files to $QUARTZ_DEPLOY_DIR" + +systemctl --user restart hermes-wiki-serve.service 2>/dev/null \ + && echo " Restarted hermes-wiki-serve.service" \ + || echo " WARNING: could not restart hermes-wiki-serve.service — verify the site manually" + echo "" echo "[wiki-maintain] Maintenance complete at $(date -Iseconds)" echo " Freshness report: $REPORT" @@ -164,4 +208,4 @@ echo "" # Print summary head -5 "$REPORT" -tail -3 "$REPORT" +tail -3 "$REPORT" \ No newline at end of file