snapshot: preserve central wiki state 2026-08-15 (7 modified + 1 new file)

2026-08-15 22:11:30 +08:00
commit 92e8028bb8
71 changed files with 9897 additions and 0 deletions
@@ -0,0 +1,67 @@
+---
+title: Browser Backend
+type: system
+status: active
+created: 2026-07-22
+updated: 2026-08-02
+verified_on: 2026-08-02
+confidence: high
+tags: [browser, camofox, automation]
+sources: [live-probe, config, session-history]
+---
+
+# Browser Backend
+
+## Purpose
+Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.
+
+## Current Configuration (as of 2026-08-02)
+- **Primary backend:** Camoufox (Firefox-based, anti-fingerprint) on CT450
+- **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox`
+- **REST API:** `localhost:9093` (profile `anthony`, sessionKey `anthony`)
+- **CDP WebSocket:** `localhost:9377` (via Camoufox's Playwright bridge)
+- **VNC viewer:** `https://ct460.kangaroo-eel.ts.net/vnc-camofox` (port 6080)
+- **Steel Browser (CT201):** Still running but Hermes not connected — plugin disabled
+
+## Camoufox (CT450)
+- **Host:** CT450 (same host as Hermes — no network hop)
+- **Deployment:** Systemd service `camofox-browser.service`
+- **Engine:** Camoufox (custom Firefox build with anti-detection fingerprinting)
+- **Display:** Headed via Xvfb `:99` (1600×1200×24) + fluxbox window manager
+- **Auth:** Persistent profile `anthony` — authenticated into Uber Eats, LinkedIn, etc.
+- **Connection:** REST API at `localhost:9093` for session management, CDP WebSocket at `localhost:9377` for browser tools
+- **Last verified:** Jul 30 — `browserConnected: true`, `sessions: 1`, `tabs: 1`, active 200s on scroll/navigate
+
+### Access
+| Endpoint | URL | Used by |
+|---|---|---|
+| REST API | `http://localhost:9093` | Hermes browser tools (navigate, click, snapshot) |
+| CDP WebSocket | `ws://localhost:9377` | Direct CDP control |
+| VNC | `https://ct460.kangaroo-eel.ts.net/vnc-camofox` | Visual session viewer |
+| Camoufox UI | `http://localhost:9378` | Internal management |
+
+### Config
+- `cloud_provider: camofox` in `~/.hermes/config.yaml`
+- `CAMOFOX_URL=http://localhost:9093` in `~/.hermes/.env`
+- `CAMOFOX_API_KEY` in `~/.hermes/.env`
+- Profile: `anthony` (502913bf) — NOT `persist-anthony`
+
+## Steel Browser (CT201) — Disabled
+- **Status:** Still running on CT201, Hermes not connected since 2026-07-30
+- **Plugin:** `browser-steel` at `/home/hermes/.hermes/plugins/browser-steel/` — disabled
+- **CDP:** `http://100.96.244.39:9223`
+- **REST API:** `http://100.96.244.39:3000`
+- **UI:** `https://runtipi.kangaroo-eel.ts.net/steel`
+- **See:** [[comparisons/steel-vs-camoufox]] for the full experiment story
+
+## History
+- **2026-07-22:** Camoufox primary, cua-driver bridge
+- **2026-07-29:** Steel Browser deployed on CT201, became primary
+- **2026-07-30:** Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary.
+- **2026-08-02:** Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API)
+
+## Related
+- [[comparisons/steel-vs-camoufox]]
+- [[concepts/browser-automation]]
+- [[decisions/rollback-steel-to-camofox]]
+- [[runbooks/restart-browser]]
@@ -0,0 +1,40 @@
+---
+title: Cloudflare Tunnel
+type: system
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+verified_on: 2026-07-22
+confidence: high
+tags: [system, network, cloudflare]
+sources: [raw/configs/hermes-config-sanitized.txt]
+---
+
+# Cloudflare Tunnel
+
+## Purpose
+Exposes internal homelab services, including Hermes, to the public internet without opening inbound firewall ports.
+
+## Location
+- **Runs on:** ct207 "webserver" (Tailscale `100.82.40.23`), **not** on this host.
+- **Binary:** `/bin/cloudflared`
+- **Service:** `cloudflared.service` (systemd, active/running on ct207)
+- **Tunnel name / ID:** `ct207-tunnel` / `339a6757-5574-4723-a16c-da4769535caa`
+- **Config:** `/etc/cloudflared/config.yml` on ct207; credentials file `339a6757-5574-4723-a16c-da4769535caa.json` in the same directory (not read/copied here — credential material, not a fact to mirror).
+
+## Current configuration
+- `originRequest`: `noTLSVerify: true`, `disableChunkedEncoding: true`, `http2Origin: false` (global defaults; some hostnames override per-entry)
+- Ingress is a flat list of `hostname → service` mappings ending in `service: http_status:404` catch-all — see [[infrastructure/domains-and-tunnels]] for the full list and the entries specific to Hermes.
+- Two Hermes-relevant entries point at this host's Tailscale IP (`100.118.5.51`): `hermes.martinwa.org` (port 19119) and `hermes-desktop.martinwa.org` (port 5174).
+
+## Connected systems
+- [[systems/hermes-agent]] — the service actually reached through this tunnel
+- [[infrastructure/domains-and-tunnels]] — full hostname table
+- [[infrastructure/network-map]]
+
+## Known issues
+- This host cannot restart or inspect the tunnel directly (no `cloudflared` installed here); a tunnel-side problem requires access to ct207.
+- `mcp-portal.martinwa.org`, referenced in Hermes's own config as an MCP endpoint, was not found in the ct207 ingress list captured 2026-07-22 — **unresolved**, may be on a different tunnel/host.
+
+## Relevant runbooks
+- [[runbooks/diagnose-docker]] (closest existing runbook; no dedicated tunnel-restart runbook exists yet — worth writing one that documents `systemctl restart cloudflared` on ct207, since Hermes has no direct access to perform it)
@@ -0,0 +1,38 @@
+---
+title: Current Hermes Profile
+type: system
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+verified_on: 2026-07-22
+confidence: high
+tags: [system, hermes, profile]
+sources: [raw/configs/hermes-config-sanitized.txt]
+---
+
+# Current Hermes Profile
+
+## Active Profile
+- **Name:** `default`
+- **Location:** `~/.hermes/profiles/default/`
+- **No other profiles confirmed**
+
+## Key Settings
+- **Model default:** `freellmapi-auto`
+- **Default provider:** `custom:litellm` → `http://litellm:4000/v1`
+- **Switch persistence:** `persist_switch_by_default: false`
+- **Timezone:** `Australia/Perth`
+- **Approvals mode:** off
+- **Memory:** Mnemosyne, session-scoped, auto-sleep enabled
+- **Pet slug:** none
+- **Skin:** slate
+
+## Paths
+- Config: `~/.hermes/config.yaml`
+- Working directory: `/home/hermes/workspace`
+- Install: `~/.hermes/hermes-agent/`
+- Bin: `~/.local/bin/hermes`
+
+## Related
+- [[systems/hermes-agent]]
+- [[model-providers]]
@@ -0,0 +1,44 @@
+---
+title: Guanaco
+type: system
+status: active
+created: 2026-08-14
+updated: 2026-08-14
+verified_on: 2026-08-14
+confidence: high
+tags: [llm, proxy, router, guanaco, ollama]
+sources: [live-probe, session-history]
+---
+
+# Guanaco
+
+## Purpose
+LLM proxy/router providing model routing and tool-call formatting for the Ollama-based inference stack on CT205.
+
+## Current Configuration
+- **Host:** CT205 (`ollama` LXC container) on Proxmox
+- **Version:** v0.8.14 (updated from v0.8.11 on 2026-08-14)
+- **Repo path (on CT205):** `/root/.guanaco/repo`
+- **API endpoint:** `ollama:11434` (DNS) / `localhost:11434`
+- **Key component:** `cmdcode_client.py` — Guanaco's tool-call formatting layer (DSML)
+
+## Version History
+
+### v0.8.14 (current, updated 2026-08-14)
+- **DSML tag leakage fixes** (cumulative across v0.8.12–v0.8.14):
+  - v0.8.12 — incomplete DSML blocks leaking raw tags as content
+  - v0.8.13 — comprehensive DSML tag leakage: double-pipe, bare fragments, solo `invoke`
+  - v0.8.14 — `<dsml_ignore>` blocks, bare parameter values, stray `>` cleanup
+- **Dashboard performance fix** (`analytics.py`): fixed wrong column names in optimized history query, cached content size, stopped `SELECT *` on history table
+- No new features, no breaking config changes — low-risk incremental bug-fix release
+- Local `app.py` modification (working-tree change) survived the tag checkout intact
+
+### v0.8.11 (previous)
+- Had known DSML tag leakage issues where malformed tool-call output could leak raw `<dsml_...>` syntax into model context stream
+
+## What DSML Is
+DSML = "Daily Smart Messaging Log" — Guanaco's tool-call formatting layer for the cmdcode client. When a model emits a tool call, Guanaco formats it through DSML. Malformed output from the model could previously leak raw tags (`<dsml_invoke>`, bare `>` characters) into the conversation context, which confused downstream models.
+
+## Related
+- [[infrastructure/proxmox]] — CT205 in LXC container table
+- [[systems/ollama]] — Ollama inference stack (if exists)
@@ -0,0 +1,40 @@
+---
+title: Headless Server
+type: system
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+verified_on: 2026-07-22
+confidence: high
+tags: [system, browser]
+sources: [raw/configs/hermes-config-sanitized.txt]
+---
+
+# Headless Server
+
+## Purpose
+Headless X11 environment backing Hermes's browser automation and any GUI tooling.
+
+## Location
+- **Host:** this container (ct460, `hermes`), no separate VM/LXC — headless stack runs directly alongside the agent.
+- **Display:** `Xvfb :99`, resolution `1600x1200x24`, started with `-ac -nolisten tcp`
+
+## Current configuration (verified via `ps aux`, 2026-07-22)
+| Process | Detail |
+|---|---|
+| `Xvfb` | `:99 -screen 0 1600x1200x24 -ac -nolisten tcp` |
+| `fluxbox` | window manager on `:99` |
+| `x11vnc` | `-display :99 -forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw` — **no VNC password set** |
+| `websockify` ×2 | `127.0.0.1:6080 → 127.0.0.1:5900` and `127.0.0.1:6081 → 127.0.0.1:5901`, serving noVNC (`/usr/share/novnc`) |
+
+Two VNC/websockify pairs exist (5900/6080 and 5901/6081) — the second (`:99`→5901→6081) is the one confirmed above; the first (5900→6080) predates it and wasn't independently re-verified this pass — **unresolved** whether it's a second live display or a leftover process.
+
+## Connected systems
+- [[systems/hermes-agent]] (browser backend uses this display — see [[systems/browser-backend]] / [[comparisons/camoufox-vs-chrome]])
+- [[infrastructure/hosts]]
+
+## Known issues
+- `x11vnc -nopw`: VNC is unauthenticated. It's bound to all interfaces (`0.0.0.0:5901` and `[::]:5901`), not just loopback — reachable from the LAN and, if any tunnel/firewall rule forwards it, potentially beyond. Worth restricting to loopback or adding `-passwd`/`-rfbauth` if this hasn't been a deliberate choice.
+
+## Relevant runbooks
+- [[runbooks/restart-browser]]
@@ -0,0 +1,141 @@
+---
+title: Hermes Agent
+type: system
+status: active
+created: 2026-07-22
+updated: 2026-07-25
+verified_on: 2026-07-25
+confidence: high
+tags: [system, hermes, agent, llm]
+sources: [raw/configs/hermes-config-sanitized.txt]
+---
+
+# Hermes Agent
+
+## Purpose
+Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
+
+## Current Status
+- **Version:** v0.19.0 (2026.7.20), upstream `9ecacd6b`
+- **Install method:** git
+- **Python:** 3.11.15
+- **OpenAI SDK:** 2.24.0
+- **Upstream status:** Up to date
+- **Install directory:** `/home/hermes/.hermes/hermes-agent`
+- **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions
+- **Binary:** `/home/hermes/.local/bin/hermes`
+- **Config:** `~/.hermes/config.yaml` (version 33)
+- **Profile:** `default` (`~/.hermes/profiles/default/`)
+
+## Location
+- **Host:** Local Linux host (no containerized Hermes runtime confirmed)
+- **Filesystem root:** `~/.hermes/`
+- **Working directory:** `/home/hermes/workspace`
+
+## Providers & Routing
+- **Default provider:** `custom:litellm` → `http://litellm:4000/v1`
+- **Fallback provider:** `custom:omniroute` → `http://omniroute:20128/v1`
+- **Default model:** `freellmapi-auto`
+- **Provider model count:** ~3430 (omniroute), ~1512 (litellm)
+- **free-stack combo (OmniRoute):** Nemotron 3 Ultra 550B configured with `max_tokens: 8192`, `reasoning_effort: medium`, `chat_template_kwargs.force_nonempty_content: true` (see [[omniroute]])
+- See [[model-providers]] for full breakdown.
+
+## Memory
+- **Backend:** Mnemosyne (local native)
+- **Memory enabled:** true
+- **Char limit:** 5000
+- **User profile enabled:** true
+- **User char limit:** 3000
+- **Flush every:** 6 turns
+- **Auto sleep:** true
+- **Default scope:** session
+- **Vector type:** int8
+- **Shared surface:** `data/shared/mnemosyne.db`
+- See [[memory-backend]] for details.
+
+## Browser
+- **Cloud provider:** Camoufox
+- **Managed persistence:** true
+- **Inactivity timeout:** 120 s
+- **Allow private URLs:** true
+- **cua-driver:** v0.9.0 at `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver`
+- **DISPLAY:** `:102` (Xvfb/X11VNC/fluxbox confirmed working per session notes)
+- See [[browser-backend]] for details.
+
+## Terminal
+- **Backend:** local
+- **Working directory:** `/home/hermes/workspace`
+- **Timeout:** 180 s
+- **Home mode:** auto
+- **Container CPU:** 1
+- **Persistent shell:** true
+- See [[terminal-backend]] for details.
+
+## Skills
+- **Total installed:** 66
+- Key skills: hermes, hermes-gateway-troubleshooting, hermes-mcp-troubleshooting, hermes-api-vault, omniroute, job-search-automation, weekly-review, fresh-rss-intelligence-filter, etc.
+- See [[skills-index]] for categorized list.
+
+## Multi-Profile Multiplexer
+
+The default gateway runs as a **profile multiplexer** (`gateway.multiplex_profiles: true`), serving both the `default` and `ops` profiles through a single gateway process with one Telegram connection.
+
+### Profiles
+
+| Profile | Chat | Purpose |
+|---------|------|---------|
+| default | `-1004321904721` (Rhino chat) | Main agent |
+| ops | `-1003914987043` (Elephant ops), `-1003932503629` (OPS Tember rescue) | Operations/infra agent |
+
+### How It Works
+
+- **One gateway, one Telegram bot token** — shared across profiles
+- **`profile_routes`** maps chat_id → profile at the gateway level
+- **`allowed_chats` on both profiles** must include ALL chats that either profile should hear, because the default profile's adapter processes all inbound messages before `profile_routes` stamps `source.profile`
+- The ops profile does NOT need `platforms.telegram.enabled` — the default adapter handles it
+
+### Critical Rule: allowed_chats Is Pre-Route
+
+The `allowed_chats` gate runs on the **default profile's adapter** before `profile_routes` matching. If a chat_id is only in the ops profile's `allowed_chats` but not the default's, messages from that chat are silently dropped before routing ever fires.
+
+**Fix applied 2026-07-28:** Added both ops group IDs to the default profile's `allowed_chats` and `group_allowed_chats` (comma-separated). See [[multiplexer-setup]].
+
+## Messaging Integrations
+- **Telegram:** enabled, `allowed_chats: -1004321904721,-1003914987043,-1003932503629`, rich messages
+- **Discord:** enabled, streaming true
+- **Slack:** enabled
+- **Email:** connected
+- **Mattermost:** connected
+- **ntfy:** connected
+- **Google Chat:** available (`hermes-google_chat`)
+- **QQ:** available (`hermes-qqbot`)
+- **Yuanbao:** available
+- See [[messaging-integrations]] for details.
+
+## Scheduled Tasks
+- **Cron jobs:** 25 active jobs
+- Delivery targets: Telegram (primary), Discord, origin, local
+- Notable: Birthday checker, job radar, newsletter digests, GitHub trending, mnemosyne health/watchdog/backup, AgentMail watcher
+- See [[scheduled-tasks]] for full table.
+
+## Dependencies
+- LiteLLM proxy at `litellm:4000`
+- OmniRoute at `omniroute:20128`
+- cua-driver binary at path above
+- Xvfb + X11VNC + fluxbox for headed browser
+- Telegram bot token, Discord token, etc. (stored outside wiki)
+
+## Known Issues
+- Router migration incomplete
+- Browser persistence not yet fully verified
+- Obsidian Git plugin auto-sync caused conflicts during vault reorganization
+
+## Related
+- [[model-providers]]
+- [[memory-backend]]
+- [[browser-backend]]
+- [[terminal-backend]]
+- [[skills-index]]
+- [[messaging-integrations]]
+- [[scheduled-tasks]]
+- [[current-state]]
@@ -0,0 +1,51 @@
+---
+title: hermes-ui (Browser Hermes Desktop)
+type: system
+status: live
+created: 2026-07-30
+updated: 2026-07-30
+verified_on: 2026-07-30
+confidence: high
+tags: [hermes, webui, desktop, browser]
+---
+
+# hermes-ui
+
+The official Hermes Desktop renderer, extracted from the `hermes-agent` monorepo (`apps/desktop`) and repackaged as a plain Vite web app — runs in a browser instead of Electron.
+
+- **Repo:** https://github.com/przbadu/hermes-ui
+- **Upstream:** NousResearch/hermes-agent monorepo, commit `56a8e81d33a524f0ba0d68b6d54c8786ed283fb8` (2026-07-08)
+- **Extraction date:** 2026-07-11
+- **License:** MIT (Copyright (c) 2025 Nous Research)
+
+## Deployments
+
+| Host | Location | Port | Status | Started |
+|------|----------|------|--------|---------|
+| CT460 (this box) | `/home/hermes/hermes-ui/` | 5174 | Vite dev server | Active |
+| CT450 (hermesagent) | `/home/hermes/hermes-ui/` | 5174 | Vite dev server | Active (since 2026-07-30) |
+
+## How It Works
+
+- Runs as a Vite dev server, proxying REST/auth/WebSocket to a running Hermes gateway (`http://127.0.0.1:9119` by default).
+- Same-origin proxy via Vite so cookies and the WebSocket work without CORS issues.
+- Electron-only methods are stubbed behind a capability flag in `src/web-bridge/`.
+
+## Quick Start
+
+```bash
+# Install deps (uses bun)
+cd ~/hermes-ui/app && bun install
+
+# Start dev server
+cd ~/hermes-ui && bin/dev
+
+# Or skip bin/dev and run vite directly:
+bunx vite --port 5174 --host 0.0.0.0
+```
+
+## Files
+
+- `PLAN.md` — extraction plan
+- `UPSTREAM.md` — upstream commit diff guide
+- `scripts/serve-on-gateway.sh` — serve built UI same-origin via `hermes serve`
@@ -0,0 +1,36 @@
+---
+title: Home Assistant
+type: system
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+verified_on: 2026-07-22
+confidence: low
+tags: [system, automation]
+sources: [raw/configs/hermes-config-sanitized.txt]
+---
+
+# Home Assistant
+
+## Purpose
+Home automation and device control; one of Hermes's messaging/integration platforms.
+
+## Location
+- **Verified (homelab infra docs, not this host):** VM 100 on `pve`, hostname `haos14.0` (Home Assistant OS), internal IP `172.30.232.1`.
+- **Verified (Hermes config):** `homeassistant` and `hermes-homeassistant` appear in Hermes's platform/display integration lists (`~/.hermes/config.yaml`), alongside `qqbot`, `signal`, `slack`, `teams`, `telegram`.
+- **Unknown:** the actual connection details (HA base URL, long-lived access token, which entities/automations Hermes can reach) are not present in the sanitized config snapshot — likely held in a plugin-specific credentials file not captured there, or not yet wired up beyond being enabled as a platform.
+
+## Current configuration
+Not verified beyond the platform being listed as available/connected in `current-state.md`'s "WebUI connected platforms" line. No automations, entity lists, or voice-assistant config confirmed from this host.
+
+## Connected systems
+- [[systems/hermes-agent]]
+- [[infrastructure/proxmox]] — Runs as VM 100 on PVE
+- [[systems/messaging-integrations]] — Integration platform
+- [[systems/homepage-dashboard]] — Shown in dashboard More tab
+
+## Known issues
+- Genuine gap, not a "known issue" per se: this page cannot confirm HA is *actively* used by Hermes vs. merely enabled — worth verifying directly (e.g. a live call to a known HA entity) next time someone is in this area.
+
+## Relevant runbooks
+None yet — would be a good candidate once the connection is verified (e.g. `runbooks/restart-home-assistant-bridge`).
@@ -0,0 +1,102 @@
+---
+title: Homepage Dashboard
+type: systems
+status: active
+created: 2026-07-26
+updated: 2026-07-26
+confidence: high
+tags: [dashboard, homepage, homelab, proxmox, monitoring]
+---
+
+# Homepage Dashboard
+
+## Purpose
+Central dashboard for all homelab services, with Proxmox integration, per-VM stats, and a management UI.
+
+## URLs
+- **Tailscale:** `https://runtipi.kangaroo-eel.ts.net:8443/`
+- **Local:** `http://runtipi.kangaroo-eel.ts.net:8082/`
+- **Manager UI:** `http://runtipi.kangaroo-eel.ts.net:8084/`
+
+## Architecture
+```
+CT201 (runtipi)
+├── Homepage container (port 8082) — main dashboard
+├── Manager container (port 8084) — web UI for adding/removing apps
+└── Tailscale serve (port 8443) — HTTPS proxy to Homepage
+```
+
+## Tabs
+| Tab | Content |
+|---|---|
+| Overview | Infrastructure (2 cols) + Everyday (4 cols) + Resources |
+| Media | Jellyfin, arr stack, RSS |
+| AI | OmniRoute, LiteLLM, Lobe Chat, SearXNG |
+| Agents | Hermes, OpenClaw, Nanobot, Camofox |
+| Tools | IT tools, Cyber Chef, etc. + Storage |
+| More | Home Assistant, N8n, Jupyter |
+| All Apps | 114 apps in 5-column grid (alphabetical) |
+
+## Proxmox Integration
+- **Node widget:** Shows live VM/LXC counts, CPU, memory for PVE node
+- **Per-VM stats:** Click "View Proxmox stats" on mapped services
+- **API token:** `homepage@pve!homepage` (privsep=0)
+- **proxmox.yaml:** Config at `/opt/homepage/config/proxmox.yaml`
+
+### VM/LXC Mapping
+| Service | VMID | Type |
+|---|---|---|
+| PVE | (node) | — |
+| Portainer docker | 200 | lxc |
+| Portainer debianVM | 101 | qemu |
+| Portainer openweb | 245 | lxc |
+
+> Portainer on CT245 (`openweb.kangaroo-eel.ts.net:9443`) now uses a Tailscale/Let's Encrypt cert (installed 2026-08-09). Files: `/root/portainer-certs/portainer.{crt,key}` mounted into the container, `--sslcert/--sslkey`. Renewal: `/root/renew-portainer-cert.sh` via crontab (04:00 on the 1st monthly) — `tailscale cert --min-validity=20d` then `docker restart portainer`. Cert validity ~90 days.
+| Runtipi | 201 | lxc |
+| Gitea | 221 | lxc |
+| FreshRSS | 260 | lxc |
+| NextCloud | 270 | lxc |
+| LiteLLM | 500 | lxc |
+| OmniRoute | 600 | lxc |
+| Ntfy | 241 | lxc |
+| Home Assistant | 100 | qemu |
+| N8n | 215 | lxc |
+| Mattermost | 255 | lxc |
+| SFTPGo | 299 | lxc |
+| Cliproxy API | 103 | lxc |
+| OpenClaw | 403 | qemu |
+| Zeroclaw | 333 | lxc |
+| Hermes | 460 | lxc |
+| Hindsight | 450 | lxc |
+
+## Configuration Files
+All at `/opt/homepage/config/` on CT201:
+- `services.yaml` — All apps, groups, and Proxmox mappings
+- `settings.yaml` — Theme, layout, tabs
+- `custom.css` — Warm amber accent overrides
+- `widgets.yaml` — System health, clock
+- `proxmox.yaml` — PVE API token
+- `docker.yaml` — Docker socket integration
+
+## Manager UI
+Web form at port 8084 for adding/removing apps without SSH.
+- **Add:** Name, Category (dropdown), URL, optional Icon
+- **Delete:** Search apps, click Delete
+- **Auto-restart:** Homepage restarts after each change
+
+## Quick Launch Search
+Type anywhere on the dashboard to search services by name/description. Google search disabled — only searches your apps.
+
+## Theme
+- Dark base with warm amber (#d4a04a) accents
+- Bigger tiles (68px min-height, 40px icons)
+- Custom CSS in `custom.css`
+
+## Tailscale Serve
+Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead.
+
+## Maintenance
+- **Restart Homepage:** `docker restart homepage`
+- **Restart Manager:** `docker restart homepage-manager`
+- **Edit apps manually:** Edit `/opt/homepage/config/services.yaml` then restart
+- **Docker auto-discovery:** Add `homepage.group`, `homepage.name`, `homepage.href` labels to containers
@@ -0,0 +1,52 @@
+---
+title: Memory Backend
+type: system
+status: active
+created: 2026-07-22
+updated: 2026-08-01
+verified_on: 2026-08-01
+confidence: high
+tags: [memory, mnemosyne, agent]
+sources: [raw/configs/hermes-config-sanitized.txt]
+---
+
+# Memory Backend
+
+## Purpose
+Persist and recall agent memory across sessions.
+
+## Current Configuration
+- **Primary backend:** Mnemosyne (local native)
+- **Memory enabled:** true
+- **User profile enabled:** true
+- **Memory char limit:** 5000
+- **User char limit:** 3000
+- **Flush every:** 6 turns
+- **Auto sleep:** true
+- **Default scope:** session
+- **Vector type:** int8
+- **Profile isolation:** true (configured; requires gateway restart for the current messaging runtime)
+- **Shared surface:** `data/shared/mnemosyne.db`
+- **Shared surface read:** true
+- **Sync roles:** user
+- **Skip contexts:** cron, flush, subagent, background, skill_loop
+- **Reflect max calls per session:** 3
+
+## Storage Location
+- Mnemosyne home: `~/.hermes/mnemosyne/`
+- DB: `~/.hermes/mnemosyne/data/`
+- Backups: `~/.hermes/mnemosyne/backups/`
+- Logs: `~/.hermes/mnemosyne/logs/`
+- Models: `~/.hermes/mnemosyne/models/`
+- Config: `~/.hermes/mnemosyne/config.yaml`
+
+## Known Issues
+- Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods.
+- The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]].
+
+**Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]].
+
+## Related
+- [[memory-backend]]
+- [[systems/mnemosyne]]
+- [[experiments/index]]
@@ -0,0 +1,52 @@
+---
+title: Messaging Integrations
+type: system
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+verified_on: 2026-07-22
+confidence: high
+tags: [messaging, telegram, discord, email, integrations]
+sources: [raw/configs/hermes-config-sanitized.txt]
+---
+
+# Messaging Integrations
+
+## Telegram
+- **Status:** enabled, active
+- **allowed_chats:** `-1004321904721` (primary)
+- **Rich messages:** true
+- **Platform toolsets:** cli, telegram (full suite)
+- **Delivery targets seen in cron:** telegram, `telegram:1793951355`, `telegram:-1004321904721:1030`, `telegram:-1004321907D2:346`, `telegram:-1004321904721:273`
+- **Notes:** Primary delivery target for scheduled jobs
+
+## Discord
+- **Status:** enabled, streaming true
+- **Platform:** `hermes-discord`
+- **Platform toolsets:** cli, discord
+
+## Email
+- **Status:** connected
+- **Libraries:** himalaya CLI available
+- **Platform toolsets:** cli
+
+## Slack
+- **Status:** available (toolsets: cli, slack-platform)
+- **Config noted:** slack entries in config but Slack enabled status unclear from config grep
+
+## Other Platforms Available
+- **Mattermost:** toolsets present (`hermes-mattermost`)
+- **Google Chat:** toolsets present (`hermes-google_chat`)
+- **QQ:** toolsets present (`hermes-qqbot`)
+- **Signal:** toolsets present (`hermes-signal`)
+- **WhatsApp:** toolsets present (`hermes-whatsapp`)
+- **ntfy:** noted in memory config outputs but runtime status unverified
+- **Yuanbao:** toolsets present (`hermes-yuanbao`)
+- **Home Assistant:** homeassistant-platform enabled
+
+## Known Issues
+- Telegram thread_id `34795` for `telegram:1793951355` not found — one cron job delivered without thread_id
+
+## Related
+- [[messaging-integrations]]
+- [[scheduled-tasks]]
@@ -0,0 +1,84 @@
+---
+title: Mnemosyne
+type: system
+status: active
+created: 2026-07-22
+updated: 2026-08-02
+verified_on: 2026-08-02
+confidence: high
+tags: [memory, mnemosyne, backend]
+sources: [live-probe, mnemosyne-db, config, ops-profile-config]
+---
+
+# Mnemosyne
+
+## Purpose
+Primary agent memory store for Hermes.
+
+## Current Configuration
+- **Provider:** `mnemosyne`
+- **Enabled:** true
+- **Auto sleep:** true
+- **Default scope:** `session`
+- **Memory char limit:** 5000
+- **User char limit:** 3000
+- **Vector type:** int8
+- **Profile isolation:** true
+- **Shared surface read:** true
+- **Shared surface path:** `data/shared/mnemosyne.db`
+- **Skip contexts:** cron, flush, subagent, background, skill_loop
+- **Reflect max calls per session:** 3
+- **Sync roles:** user
+- **Flush every:** 6 turns
+- **Embedding model:** `BAAI/bge-small-en-v1.5` (384 dims)
+- **Embeddings via API:** false (local fastembed)
+
+## Database Stats (2026-08-02)
+| Table | Rows | Notes |
+|---|---|---|
+| `working_memory` | 1,476 | Active memories |
+| `episodic_memory` | 458 | Consolidated summaries |
+| `memory_embeddings` | 158 | Only working memories — **0 episodic** |
+| `memory` (legacy) | 2,181 | Old table, not migrated |
+| `vec_working` | 159 | Vector index rows |
+| Shared surface | 0 | Empty — cross-agent sharing not functional |
+
+## Known Issues
+
+### Vector embedding coverage gap
+- **Only 159/1,476 working memories (10.8%)** have vector embeddings
+- **0/458 episodic memories** have embeddings
+- Semantic search (`mnemosyne_recall`) falls back to FTS5 keyword matching for ~90% of working and 100% of episodic memories
+- This means recall quality is degraded — vector similarity can only search a tiny fraction of the memory store
+- **Root cause:** The embedding pipeline (fastembed) never processed the bulk of memories. The `vec_working` table was rebuilt once (Jul 18) but only captured 159 rows.
+- **Fix needed:** Backfill embeddings for all working and episodic memories. This requires running the embedding pipeline across the full dataset.
+
+### Profile isolation chosen over shared surface
+|- The ops profile was enabled with mnemosyne on 2026-08-02
+|- **Decision:** `profile_isolation: true` was set on ops, giving each profile its own independent memory bank — zero cross-contamination
+|- `data/shared/mnemosyne.db` exists with 0 rows; cross-profile sharing is intentionally unused
+|- This means Tusk (ops) and Rhino (default) do not share memories, which is the desired behaviour
+|- Config in `~/.hermes/profiles/ops/config.yaml`: `memory.mnemosyne.profile_isolation: true`, `allow_tool_override: false`, `shared_surface_read: false`
+|- Verified end-to-end on 2026-08-02: memory writes land correctly in both profiles independently
+
+### Legacy table not migrated
+- 2,181 rows in the old `memory` table are not in the new `working_memory`/`episodic_memory` system
+- These are only accessible via the legacy `memory` tool, not via `mnemosyne_recall`
+
+## Known Management Jobs
+- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
+- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
+- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception
+- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
+
+## History and Evidence
+- **2026-08-01 incident:** Automatic conversational capture was broken by stringified structured YAML, especially `sync_roles: "['user']"`. Fixed by rewriting with native YAML types. See [[incidents/2026-08-01-mnemosyne-capture-failure]].
+- **2026-08-01 cleanup:** Legacy always-injected memory reduced from 8,013 bytes to 4,998 bytes (memory) and 2,958 bytes (user).
+- **2026-07-29 incident:** `memory.mnemosyne.tools: None` caused schema loading warnings. Fixed by removing the line. See [[incidents/2026-07-29-mnemosyne-relay]].
+- **2026-07-29 relay scope corruption:** `RuntimeError: scope handle is not at the top of the stack` — 3 occurrences, self-resolved.
+
+## Related
+- [[concepts/agent-memory]]
+- [[comparisons/memory-backends]]
+- [[incidents/2026-08-01-mnemosyne-capture-failure]]
+- [[incidents/2026-07-29-mnemosyne-relay]]
@@ -0,0 +1,77 @@
+---
+title: Model Providers
+type: system
+status: active
+created: 2026-07-22
+updated: 2026-08-14
+verified_on: 2026-08-14
+confidence: high
+tags: [model, provider, inference, routing]
+sources: [live-probe, config, web-search]
+---
+
+# Model Providers
+
+## Purpose
+Inference providers available to Hermes, their routing paths, and notable model updates.
+
+## Current Provider Stack
+| Provider | URL | Role |
+|---|---|---|
+| `custom:litellm` | `http://litellm:4000/v1` | Main inference (slower path, 6–26s/call) |
+| `custom:omni` | `http://omni:20128/v1` | Auxiliary roles (vision, skills, approval, mcp, web_extract, compression) — factory-reset 2026-08-14 |
+| `opencode-zen` | (internal) | Free-tier model access for cron jobs |
+| Guanaco | `ollama:11434` | LLM proxy/router on CT205 (Ollama stack) |
+
+## Default Model
+- `resilient-free` — resolves through OmniRoute (changed from `auto/best-free` on 2026-08-14 after factory reset)
+- Previous default `auto/best-free` was misconfigured — had no combo definition and routed to paid `gpt-5.6-sol`
+- `freellmapi-auto` (LiteLLM path) still available but slower
+
+## Notable Model Updates
+
+### GPT-5.6 Luna — Model Research (2026-08-14)
+- **Community consensus:** Luna is cleaner/more polished; DeepSeek V4 Flash is more persistent/cheaper
+- **Luna strengths:** visual/UI work, final code review, bounded high-stakes tasks, cleaner structured output
+- **DeepSeek strengths:** automation, persistent executor work, high-volume subtasks, long-context economics
+- **Key caveat:** Provider route matters more than model name — `openai-codex/gpt-5.6-luna`, `openrouter/openai/gpt-5.6-luna`, `deepseek/deepseek-v4-flash`, and `openrouter/deepseek/deepseek-v4-flash` are four different tests
+- **Pricing:** Direct OpenAI $0.20/M input, $1.20/M output; OpenRouter $0.10/M input, $0.60/M output
+- **Context:** 1.05M tokens, 128K output, function calling, structured outputs, image input, configurable reasoning effort
+- **Recommendation:** Luna plans and reviews; DeepSeek executes — but test on a clean direct route before judging
+
+### Guanaco v0.8.14 (2026-08-14)
+- Updated from v0.8.11 on CT205 — three incremental bug-fix releases focused on DSML/tool-call output hygiene
+- DSML tag leakage fixes: malformed tool-call output no longer leaks raw syntax junk into model context
+- Dashboard performance fix: cached content size, fixed wrong column names in history query
+- No new features, no breaking changes — low-risk update
+- See [[systems/guanaco]]
+
+### DeepSeek V4 Flash 0731 (2026-07-31)
+- **Same model ID:** `deepseek-v4-flash` — no ID change, just a post-training bump
+- **Same architecture:** 284B total params, 13B active per token, 1M context window
+- **Same pricing:** $0.14/M input, $0.28/M output
+- **Benchmark jump:** Terminal Bench 2.1: 82.7 (up from 61.8 preview — 34% relative gain)
+- **Now surpasses V4-Pro preview** (72.1) on agent/coding benchmarks
+- **Stricter tool name validation:** Rejects tool names not matching `^[a-zA-Z0-9_-]+$` — may cause issues with OmniRoute-transformed tool names
+- **Impact:** Your existing cron jobs using `deepseek-v4-flash` get the upgrade automatically. The empty-stream fallback issue (`opencode-go/deepseek-v4-flash` returning empty 200s) is a provider plumbing problem, not a model quality problem.
+
+### OmniRoute Free-DeepSeek Silent-Stop Mitigation (2026-08-04)
+- Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances
+- Reworked both combos: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort
+- See [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]]
+
+## Context Enforcement
+- Three layers: model native → combo `data.config.context_length` → Hermes client cap
+- Effective context is the minimum of the three
+
+## Combo Behavior
+- `priority` retries same provider up to `maxRetries`, then errors; it does NOT fall through
+- `fill-first` walks the list on failure
+- `auto`/`lkgp` recalculates per request using health scoring
+- Same-provider entries in a combo share rate-limit buckets
+- Session affinity caches provider in `session_model_history`
+
+## Related
+- [[concepts/model-routing]]
+- [[systems/omniroute]]
+- [[systems/opencode-go]]
@@ -0,0 +1,52 @@
+---
+title: Nanobot
+type: system
+status: degraded
+created: 2026-07-22
+updated: 2026-08-07
+verified_on: 2026-08-07
+confidence: high
+tags: [system, nanobot, network]
+sources: [live-probe, omniroute-investigation]
+---
+
+# Nanobot
+## Purpose
+Role in Anthony's setup.
+## Location
+- **Verified 2026-07-22:** LXC 333, hostname `clawtest`, Tailscale IP `100.122.189.81`.
+## Current state
+- **Status: DOWN (as of 2026-08-07).** Connection refused on port 8900 — the OmniRoute provider connection to nanobot (`58debd59`, prefix `nanobot`) was causing event-loop stalls because every model sync + credential health check retried it. Connection disabled in OmniRoute. Nanobot on CT333 is not responding.
+- **Prior state:** `nanobot` process listening on port `8766` (verified via `ss -tlnp` on ct333, 2026-07-22).
+- `whatsapp-bridge.service` (systemd, active) — "WhatsApp Bridge for Nanobot" — also runs on ct333. Status unknown as of 2026-08-07.
+- Publicly reachable via the same Cloudflare Tunnel as Hermes: `nanobot.martinwa.org` → `100.122.189.81:8766`, `nano-ssh.martinwa.org` → SSH on the same host. See [[infrastructure/domains-and-tunnels]].
+
+## Current state as of 2026-07-22 (superseded — see below)
+~~- No nanobot process running on this host~~
+~~- `nanobot` systemd user service: inactive~~
+~~- Not present in `pgrep` process list~~
+
+**Correction (this audit, same day):** the above was true only for *this* host (ct460) — Nanobot was never expected to run here. It **is** actively running, on LXC 333 ("clawtest"), confirmed via direct process/port check. Left the original lines struck through rather than deleted, per SCHEMA rule 9 (mark, don't silently delete).
+
+## History
+- **User-provided:** Nanobox on the PVE host at `192.168.178.39`
+- **Correction:** `192.168.178.39` is `pve`'s own LAN IP (matches the Cloudflare Tunnel's `pve.martinwa.org` entry, which proxies to Proxmox's web UI on that address) — this was a conflation, not a second Nanobot instance. The real Nanobot host is ct333 (see Location above).
+- **Verified:** Lan/Wi-Fi node at `192.168.178.46` running ARM/arm64 firmware; `ubnt` login at `192.168.1.1` does not apply to this node
+- **Verified:** DWARF01 identified as a UniFi device; ONT01 status unresolved; DNS provided by `192.168.178.39`, not the router
+- **User-provided:** Avoid `heroku.com/archive` except for CAPI; use official CAPI course downloads instead
+- **Superseded:** "Nanobot may have been used previously as Hermes gateway or tunnel front on another host" / "Unknown: whether nanobot should be redeployed" — no longer applicable now that it's confirmed running; the open question is now *why* it's running alongside Hermes and whether that's intentional (see [[comparisons/hermes-vs-nanobot]]).
+
+## Connected systems
+- [[systems/hermes-agent]]
+- [[systems/headless-server]]
+
+## Known issues
+- Overlap/purpose vs. Hermes not documented anywhere — see [[comparisons/hermes-vs-nanobot]] for the open question this raises.
+## Change history
+- 2026-07-22: audit corrected "not running" claim after direct verification on ct333.
+- 2026-08-07: status changed to DEGRADED — nanobot confirmed DOWN (connection refused on 8900). OmniRoute provider connection disabled. CT333 service status unknown.
+## Relevant runbooks
+- [[runbooks/restart-hermes]]
+
+## Related
+- [[comparisons/hermes-vs-nanobot]]
@@ -0,0 +1,42 @@
+---
+title: Obsidian Vault
+type: systems
+status: active
+created: 2026-07-26
+updated: 2026-07-26
+confidence: high
+tags: [obsidian, notes, vault, homelab]
+---
+
+# Obsidian Vault
+
+## Purpose
+Personal knowledge base — notes, research, projects, and agent documentation.
+
+## Details
+- **Location:** `/home/hermes/wiki/` (on CT460)
+- **Git repo:** Gitea (`Anthony/obsidian-vault`)
+- **Structure:** NUMBERED folders (00-99)
+- **Sync:** Git-based (manual or cron-triggered)
+
+## Folder Structure
+| Folder | Purpose |
+|---|---|
+| `00 - Home` | Dashboard and entry points |
+| `10 - Inbox` | New/unprocessed notes |
+| `20 - Agent Notes` | Notes from AI agents |
+| `30 - Projects` | Active project documentation |
+| `50 - Research` | Research and investigations |
+| `60 - Resources` | Reference materials |
+| `70 - Reference` | Quick lookup |
+| `90 - Archive` | Completed/old material |
+| `99 - Templates` | Note templates |
+
+## Conventions
+- Numbered folders preferred (Anthony corrected twice)
+- Check Mnemosyne before vault work
+- Don't impose structure changes without understanding usage
+
+## Related
+- [[infrastructure/hosts]] — CT460 host details
+- [[systems/memory-backend]] — Agent memory integration
@@ -0,0 +1,92 @@
+---
+title: OmniRoute
+type: system
+status: active
+created: 2026-07-22
+updated: 2026-08-14
+verified_on: 2026-08-14
+confidence: high
+tags: [routing, provider, model, inference]
+sources: [raw/configs/hermes-config-sanitized.txt]
+---
+
+# OmniRoute
+
+## Purpose
+Secondary provider/router for Hermes auxiliary roles and some agent routing.
+
+## Current Configuration
+- **Hermes provider name:** `custom:omni`
+- **Base URL:** `http://omni:20128/v1`
+- **Host reachability:** `omni.kangaroo-eel.ts.net` resolves to `100.93.204.4`; CT601 running, direct HTTP reachable
+- **Runs on:** Proxmox CT601 (`omni`) — migrated from CT600 (`omniroute`, now stopped) on 2026-08-13
+- **DB path:** `/opt/omniroute/storage.sqlite` (was `/var/lib/omniroute/storage.sqlite`)
+- **Backup:** pre-migration config saved at `~/.hermes/config.yaml.bak.omniroute-removal`
+- **Miss:** Hermes default provider path does not use OmniRoute for main chat; used for auxiliary roles instead
+- **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp`
+- **Key prefix visible in config:** `sk-5c67fbdc0...` — full value not stored in wiki
+
+## Catalog and Routing Facts
+- Catalog reachable: yes, 47 combos returned at `/api/combos`
+- Combo schema observed: `name`, `strategy`, `models`, `id`; some combos have `config`, `isHidden`, `uuid`
+- Combo-level context can override individual models
+- `auto`/`lkgp` strategies may silently change chosen models
+- Model mappings can substitute models silently
+- Session affinity caches last successful provider in `session_model_history`
+
+## Catalog Evidence (2026-07-22)
+- Advertised context for `gemini/gemini-3.6-flash`: `1048576`
+- Related models in catalog: `kc/google/gemini-3.6-flash`, `kilocode/google/gemini-3.6-flash`, `nous/google/gemini-3.6-flash`, `opencode-zen/gemini-3.6-flash`, `vertex/gemini-3.6-flash`
+- No live combo in the returned 47 combos referenced `gemini-3.6-flash`
+
+## Known Issues
+- opencode-zen proxy pitfall documented in omniroute-ops skill
+- Web search proxy failures documented in omniroute-ops skill
+- Combo discovery can differ from resolution;
+
+## AutoRoute variants (2026-08-12)
+- `auto/best-free` (Auto Cheap) and `auto/coding-free` are **virtual** AutoRoute variants — they do NOT appear in the persisted combo list (`/api/combos` returns 54 combos, none named these).
+- The LKGP engine resolves them dynamically: it scores the operator's **live connected provider accounts** (e.g. `antigravity`, `opencode-zen`, `nvidia`, `openrouter`) rather than a hardcoded candidate list.
+- Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
+- Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
+
+## Recent Changes (2026-08-14)
+- **OmniRoute factory reset on CT601.** Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (`auto/best-fast`, `auto/best-coding`, etc.). Old state backed up at `/opt/omniroute/factory-reset-backup-20260814-225524`.
+- **Providers re-added from Notion API vault:** Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry.
+- **Hermes config default model changed:** `auto/best-free` → `resilient-free` (via `~/.hermes/config.yaml`). `auto/best-free` was misconfigured — had no combo definition and routed to `gpt-5.6-sol` (paid) instead of free models. Switched to `resilient-free` which is a working free combo.
+- **Multiple free combos broken by upstream model ID changes:**
+  - OpenRouter: `poolside/laguna-s-2-1:free` (dash) → `poolside/laguna-s-2.1:free` (dot) — model ID format changed
+  - Groq: `meta-llama/llama-4-scout-17b-16e-instruct` — **retired**, only llama-3.3/3.1 remain
+  - Affected combos: `resilient-free`, `free-IA`, `free-stack`, `good-free`, `deepseek-flash-free`, `mimo-free`
+  - All route through OpenRouter → fail with 400/404 when trying these model IDs
+- **Working free combos as of 2026-08-14:** `free-lkgp` (DeepSeek V4 Flash), `mega-free` (StepFun 3.7 Flash), `smart-route` / `smart-route-ia` (nested combos)
+- **LiteLLM routing issue identified:** Hermes was previously pointing at `http://litellm:4000/v1` with `free-auto` model, causing 6–26s per call. OmniRoute's `auto/best-free` tested at 265ms. Config now points at OmniRoute.
+- **GPT-5.6 Luna research:** Community consensus — Luna is cleaner/more polished; DeepSeek V4 Flash is more persistent/cheaper. Luna better for visual/UI work and final review; DeepSeek better for automation and high-volume executor work. Provider route matters more than model name.
+
+## Recent Changes (2026-08-13)
+- **OmniRoute migrated CT600 → CT601.** Provider renamed `custom:omniroute` → `custom:omni`; base URL `http://omniroute:20128/v1` → `http://omni:20128/v1`. Host `omni.kangaroo-eel.ts.net` (100.93.204.4, active) replaces `omniroute.kangaroo-eel.ts.net` (100.88.81.19, offline). CT600 stopped, CT601 running. Database path `/var/lib/omniroute/storage.sqlite` → `/opt/omniroute/storage.sqlite`. API key unchanged. All 14 config + 22 cron-job `custom:omniroute` references updated to `custom:omni`. Pre-migration backup at `~/.hermes/config.yaml.bak.omniroute-removal`. Verified live 2026-08-13 (`pct list`: 600 stopped, 601 running; `grep`: 0 remaining `custom:omniroute` refs).
+
+## Recent Changes (2026-07-25)
+- **free-stack combo updated:** Nemotron 3 Ultra 550B model at index 4 now configured with NVIDIA-specific parameters:
+  - `max_tokens`: 8192 (was default 65536, exceeding NVIDIA NIM limit of 16384)
+  - `reasoning_effort`: medium (was high, reducing long internal monologues)
+  - `chat_template_kwargs.force_nonempty_content`: true (prevents empty responses when tool calling + reasoning)
+- **Combo ID:** `23d25f8f-5eb0-4b6f-b966-e260ca3905b3`
+- **Hermes config.yaml updated:** `agent.reasoning_effort: medium`, `agent.max_tokens: 8192`, `agent.chat_template_kwargs.force_nonempty_content: true`
+- **Rationale:** NVIDIA NIM API documents max output of 16384 tokens; medium reasoning uses more efficient chat-template mode; force_nonempty_content resolves empty responses with tool calls combo in picker may still route
+
+## Recent Changes (2026-08-07)
+- **Three-layer root cause for slowness identified and fixed:**
+  1. **npm package .env override:** OmniRoute loads three .env files (`/var/lib/omniroute/.env`, `/root/.omniroute/.env`, `/usr/lib/node_modules/omniroute/.env`). The npm package's `.env` ships with `ENABLE_SOCKS5_PROXY=true` and `npm update` re-installs it with that default, silently overriding the `/var/lib` config. Fix: flip all three to `false`, then restart. After ANY upgrade, check all three.
+  2. **Dead provider connection:** Connection `58debd59` (prefix `nanobot`) points at `clawtest:8900/v1` — nanobot on clawtest is DOWN. Every model sync + credential health check retried it, burning event-loop time. Disabled it.
+  3. **PVE host load leaking into CT600:** CT600 has `nesting=1`, so `uptime`/`top` inside the container mirror PVE's host load exactly (verified: identical 10.30/9.23/10.37 on both). Same trap as CT241. Judge CT600 CPU from `pct exec 600 -- ps -o %cpu` or host-side `ps`, never from in-container load average.
+- **First-request-after-cache-expiry stall:** `/v1/models` hangs 10-15s+ on the FIRST hit after the model catalog cache expires (rebuild enumerates all connections, including slow dead ones), then answers in ~0.02s from cache. Don't judge health on one curl — run a burst of 5 and check if later tries are sub-second.
+- **`is_active=0` does NOT stop CredentialHealth probes:** Disabling a dead connection stops it from ModelSync but the health checker still probes it and logs `[ProxyFetch] ECONNREFUSED` noise. That noise is benign once the endpoint is unreachable.
+- **Lessons banked in `omniroute-ops` skill:** `references/omniroute-upgrade-procedure.md` updated with all three pitfalls.
+
+## Recent Changes (2026-08-04)
+- **OmniRoute free-DeepSeek silent-stop mitigation:** Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances. Reworked both combos: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort. See [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]].
+
+## Related
+- [[model-providers]]
+- [[model-routing]]
@@ -0,0 +1,147 @@
+---
+title: OpenCode Go
+type: system
+status: active
+created: 2026-07-29
+updated: 2026-07-29
+verified_on: 2026-07-29
+confidence: high
+tags: [opencode, provider, models, subscription, limits]
+sources: [https://opencode.ai/docs/go/]
+---
+
+# OpenCode Go
+
+Low cost subscription for open coding models — $5 first month, then **$10/month**. Designed for international users, models hosted in US, EU, and Singapore.
+
+## How It Works
+
+1. Sign in to [OpenCode Zen](https://opencode.ai/zen), subscribe to Go, copy your API key
+2. Run `/connect` in the TUI, select `OpenCode Go`, paste your API key
+3. Run `/models` to see available models
+
+> Only one member per workspace can subscribe.
+
+## Usage Limits
+
+| Window | Dollar Limit |
+|--------|-------------|
+| 5 hours | $12 |
+| Weekly | $30 |
+| Monthly | $60 |
+
+Limits are in **dollar value**, so your actual request count depends on the model. Cheaper models (DeepSeek V4 Flash, MiMo-V2.5) get far more requests than expensive ones (Kimi K3, Grok 4.5).
+
+### Estimated Requests Per Window
+
+Based on typical Go usage patterns:
+
+| Model | Per 5h | Per Week | Per Month |
+|-------|--------|----------|-----------|
+| Grok 4.5 | 120 | 300 | 600 |
+| GLM-5.2 | 880 | 2,150 | 4,300 |
+| GLM-5.1 | 880 | 2,150 | 4,300 |
+| Kimi K3 | 110 | 250 | 490 |
+| Kimi K2.7 Code | 1,350 | 3,380 | 6,750 |
+| Kimi K2.6 | 1,150 | 2,880 | 5,750 |
+| MiMo-V2.5 | 30,100 | 75,200 | 150,400 |
+| MiMo-V2.5-Pro | 3,250 | 8,150 | 16,300 |
+| MiniMax M3 | 3,200 | 8,000 | 16,000 |
+| MiniMax M2.7 | 3,400 | 8,500 | 17,000 |
+| Qwen3.7 Max | 950 | 2,390 | 4,770 |
+| Qwen3.7 Plus | 4,300 | 10,800 | 21,600 |
+| Qwen3.6 Plus | 3,300 | 8,200 | 16,300 |
+| DeepSeek V4 Pro | 3,450 | 8,550 | 17,150 |
+| DeepSeek V4 Flash | 31,650 | 79,050 | 158,150 |
+| Hy3 | 4,300 | 10,750 | 21,500 |
+
+### Token Estimates Per Request
+
+| Model | Input Tokens | Cached Tokens | Output Tokens |
+|-------|-------------|---------------|---------------|
+| Grok 4.5 | 1,100 | 71,500 | 220 |
+| GLM-5.2/5.1 | 700 | 52,000 | 150 |
+| Kimi K3 | 1,050 | 76,500 | 300 |
+| Kimi K2.7/K2.6 | 870 | 55,000 | 200 |
+| DeepSeek V4 Pro | 750 | 82,000 | 290 |
+| DeepSeek V4 Flash | 790 | 68,000 | 280 |
+| MiniMax M3 | 510 | 56,000 | 190 |
+| MiniMax M2.7 | 300 | 55,000 | 125 |
+| MiMo-V2.5 | 830 | 71,500 | 295 |
+| MiMo-V2.5-Pro | 790 | 86,000 | 305 |
+| Qwen3.7 Max | 420 | 66,000 | 200 |
+| Qwen3.7 Plus | 500 | 57,000 | 190 |
+| Qwen3.6 Plus | 500 | 57,000 | 190 |
+| Hy3 | 830 | 71,500 | 295 |
+
+### Per-Model Pricing & Monthly Usage Included
+
+| Model | Input $/1M | Output $/1M | Cached Read $/1M | Monthly Usage |
+|-------|-----------|------------|-----------------|---------------|
+| Grok 4.5 | $2.00 | $6.00 | $0.30 | $15 |
+| GLM-5.2 | $1.40 | $4.40 | $0.26 | $60 |
+| GLM-5.1 | $1.40 | $4.40 | $0.26 | $60 |
+| Kimi K3 | $3.00 | $15.00 | $0.30 | $15 |
+| Kimi K2.7 Code | $0.95 | $4.00 | $0.19 | $60 |
+| Kimi K2.6 | $0.95 | $4.00 | $0.16 | $60 |
+| MiMo V2.5 | $0.14 | $0.28 | $0.0028 | $60 |
+| MiMo V2.5 Pro | $0.435 | $0.87 | $0.0036 | $15 |
+| MiniMax M3 | $0.30 | $1.20 | $0.06 | $60 |
+| MiniMax M2.7 | $0.30 | $1.20 | $0.06 | $60 |
+| Qwen3.7 Max | $2.50 | $7.50 | $0.50 | $60 |
+| Qwen3.7 Plus (≤256K) | $0.40 | $1.60 | $0.04 | $60 |
+| Qwen3.7 Plus (>256K) | $1.20 | $4.80 | $0.12 | $60 |
+| Qwen3.6 Plus (≤256K) | $0.50 | $3.00 | $0.05 | $60 |
+| Qwen3.6 Plus (>256K) | $2.00 | $6.00 | $0.20 | $60 |
+| DeepSeek V4 Pro | $0.435 | $0.87 | $0.0036 | $15 |
+| DeepSeek V4 Flash | $0.14 | $0.28 | $0.0028 | $60 |
+| Hy3 | $0.14 | $0.58 | $0.035 | $60 |
+
+Track usage in the [console](https://opencode.ai/zen).
+
+> If you reach the usage limit, you can continue using the free models.
+
+## Usage Beyond Limits
+
+If you also have credits on your Zen balance, enable the **Use balance** option in the console. Go will fall back to your Zen balance after reaching usage limits instead of blocking requests.
+
+## Why Some Models Have Lower Usage
+
+With Go, you pay $10/month and they aim to give you **6x that in usage** ($60). For most models, this works through bulk discounts and reserved GPU capacity. For some models (Grok 4.5, Kimi K3, MiMo V2.5 Pro, DeepSeek V4 Pro), they haven't negotiated discounts yet — so the usage multiplier is lower (you get a little more than paying directly, just not the full 6x).
+
+## Available Models
+
+| Model | Model ID | Endpoint | SDK Package |
+|-------|----------|----------|-------------|
+| Grok 4.5 | `grok-4.5` | `https://opencode.ai/zen/go/v1/chat/completions` | @ai-sdk/openai-compatible |
+| GLM-5.2 | `glm-5.2` | same | same |
+| GLM-5.1 | `glm-5.1` | same | same |
+| Kimi K3 | `kimi-k3` | same | same |
+| Kimi K2.7 Code | `kimi-k2.7-code` | same | same |
+| Kimi K2.6 | `kimi-k2.6` | same | same |
+| DeepSeek V4 Pro | `deepseek-v4-pro` | same | same |
+| DeepSeek V4 Flash | `deepseek-v4-flash` | same | same |
+| MiMo-V2.5 | `mimo-v2.5` | same | same |
+| MiMo-V2.5-Pro | `mimo-v2.5-pro` | same | same |
+| MiniMax M3 | `minimax-m3` | `https://opencode.ai/zen/go/v1/messages` | @ai-sdk/anthropic |
+| MiniMax M2.7 | `minimax-m2.7` | same | same |
+| MiniMax M2.5 | `minimax-m2.5` | same | same |
+| Qwen3.7 Max | `qwen3.7-max` | same | same |
+| Qwen3.7 Plus | `qwen3.7-plus` | same | same |
+| Qwen3.6 Plus | `qwen3.6-plus` | same | same |
+| Hy3 | `hy3` | `https://opencode.ai/zen/go/v1/chat/completions` | @ai-sdk/openai-compatible |
+
+**Config format:** `opencode-go/<model-id>` (e.g. `opencode-go/kimi-k3`)
+
+**Model list API:** `https://opencode.ai/zen/go/v1/models`
+
+## Privacy
+
+Zero-retention policy from providers — your data is not used for model training.
+
+## Goals
+
+- Make AI coding accessible via low cost subscription
+- Provide reliable access to the best open coding models
+- Curate tested and benchmarked models for coding agent use
+- No lock-in (use any other provider too)
@@ -0,0 +1,42 @@
+---
+title: OpenRouter
+type: system
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+verified_on: 2026-07-22
+confidence: medium
+tags: [system, openrouter, provider]
+sources: [raw/configs/hermes-config-sanitized.txt]
+---
+
+# OpenRouter
+## Purpose
+Fallback model provider.
+## Location
+Remote service; API token store.
+## Current configuration
+Primary key, fallback models, rate limits.
+## Current state as of 2026-07-22
+OpenRouter is not configured as a standalone provider in `~/.hermes/config.yaml` `custom_providers`.
+OmniRoute’s catalog includes provider names and model IDs consistent with OpenRouter-style slugs, but live routing via OpenRouter directly is not present in current Hermes config.
+
+## History
+- **User-provided:** Previously used; now largely replaced by LiteLLM and OmniRoute paths
+- **Verified:** Decision page `keep-openrouter-as-fallback` exists, indicating retain intent
+- **Inferred:** Multiple backups before model/routing changes imply OpenRouter role changed during prior config iterations
+- **Unknown:** Exact date OpenRouter was replaced or partially deprecated; exact models that were available
+
+## Connected systems
+- [[systems/hermes-agent]]
+- [[systems/omniroute]]
+
+## Known issues
+- Replaced: legacy Hermes autos not persistent, prompting Hermes Agent, which blocks attribution.
+
+## Change history
+- Pre-2026-07-20: OpenRouter provisioning active in some form
+- 2026-07-20: Prior model-fix backup suggests routing changes around this time
+
+## Relevant runbooks
+- [[restore-openrouter]]
@@ -0,0 +1,65 @@
+---
+title: Scheduled Tasks
+type: system
+status: active
+created: 2026-07-22
+updated: 2026-08-10
+verified_on: 2026-08-10
+confidence: high
+tags: [cron, scheduled, automation]
+sources: [live-cron-list]
+---
+
+# Scheduled Tasks
+
+## Count
+- **Active jobs:** 28 (verified from live `hermes cron list`, 2026-08-07)
+- **Previously 25** — GitHub Trending Radar Daily removed 2026-08-02; several jobs added since.
+
+## Job Table
+
+| Name | Schedule | Delivery | Skills | Script | Notes |
+|---|---|---|---|---|---|
+| Facebook Birthday Monthly Scrape | `0 9 1 * *` | telegram | — | — | ⚠ thread_id not found |
+| Facebook Birthday Weekly Registry Sync | `0 3 * * 1` | telegram | — | `birthday_check.py` | ⚠ thread_id not found |
+| Daily Birthday Checker | `30 7 * * *` | telegram | — | `birthday_check.py` | no-agent |
+| Daily Marketing Job Radar | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | Primary `gpt-5.6-terra`; see [[runbooks/job-radar]] |
+| Perth New Restaurant & Cafe Openings (Weekly) | `0 18 * * 4` | telegram | — | — | |
+| AI Newsletter Curation for Patrick (Weekly) | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | |
+| Enhanced Weekend Adventure Planner | `0 8 * * 5` | telegram | — | — | |
+| Tech & AI Newsletter Digest (Analytical) | `0 7 * * *` | telegram:-1004321904721:346 | — | — | |
+| Political News Digest (Mon/Thu) | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | |
+| Uber Eats Monthly Spend Report | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | |
+| Gmail Action Digest | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | |
+| FreshRSS Midday News Briefing | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | |
+| Workday start reminder | `0 9 * * 1-5` | telegram | — | — | |
+| Important-mail monitor | `every 30m` | telegram:1793951355 | — | — | Silent when no important mail |
+| Mnemosyne Auto-Consolidate | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent |
+| Mnemosyne DB Backup | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent |
+| Mnemosyne Health Monitor | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
+| Mnemosyne Watchdog | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts |
+| Mnemosyne Embedding Coverage Monitor | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95% |
+| CT 221 Miner Watchdog | `every 15m` | telegram:1793951355 | pve-lxc-intrusion-detection | `ct221-miner-watchdog.sh` | Added 2026-08-07 after crypto miner incident |
+| Mosquito Tracker | `0 7,15,22 * * *` | telegram:-1004321904721:273 | — | `mosquito_tracker.py` | Excludes 23:00–07:00 |
+| Weekly Review | `0 18 * * 0` | origin | weekly-review | — | |
+| Evening Stand-Down Briefing | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
+| Job Tracker Dashboard Refresh | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
+| Wiki Audit — Nightly | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes |
+| GitHub Trending Radar — Weekly Roundup | `0 9 * * 1` | telegram:1793951355 | — | — | |
+| 🦝 Hermes Config Backup to Gitea (Daily) | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
+| Review Agent Estate Phase One | once at 2026-08-08 18:33 | origin | — | — | ✅ Completed 2026-08-08. See [[current-state]]. |
+
+## Removed Jobs
+| Name | Removed | Reason |
+|---|---|---|
+| GitHub Trending Radar — Daily Digest | 2026-08-02 | Stopped by user. Weekly roundup still active. |
+
+## Known Issues
+- **Session storage failures (2026-08-10):** Four cron jobs failed — two with TimeoutError (idle 601–602s while stuck initializing), two with RuntimeError ("session storage could not be written"). Disk at 83% (11GB free), `state.db` at 593MB. Likely WAL checkpoint or disk pressure issue. Affected jobs: Daily Marketing Job Radar, Tech & AI Newsletter Digest, Political News Digest, GitHub Trending Radar — Weekly Roundup. See [[current-state]] for details.
+- **Wiki Audit cron model drift (2026-08-07):** Global inference config changed (`smart-route` → `deepseek-paid-smart`) and the job is unpinned, causing `RuntimeError: Skipped to prevent unintended spend`. Needs pinning: `cronjob action=update job_id=cbfe434a3b5e provider=<provider> model=<model>`.
+- **Facebook Birthday thread_id warnings:** Configured thread_id 34795 for telegram:1793951355 was not found on several birthday-related jobs — delivered without thread.
+- **Model repinning (2026-08-08):** 13 cron jobs repinned from dead/stale provider combos to `smart-route` or `gpt-5.6-terra`. Dead combos included `freeapi/minimax-m2.7`, `freeapi/auto`, `vertex/gemini-2.5-flash`, `deepseek-v4-flash-free`. Jobs now running on valid model pins.
+
+## Related
+- [[systems/mnemosyne]]
+- [[runbooks/job-radar]]
@@ -0,0 +1,90 @@
+---
+title: Installed Skills
+type: system
+status: active
+created: 2026-07-22
+updated: 2026-07-25
+verified_on: 2026-07-25
+confidence: high
+tags: [skills, hermes]
+sources: []
+---
+
+# Installed Skills
+
+## Inventory
+Total installed: 66 skills in `~/.hermes/skills/`.
+
+## Categorized Active Skills
+
+### Core Hermes
+- `hermes` — Core agent skill
+
+### Providers & Routing
+- `omniroute` — OmniRoute routing
+- `omniroute-ops` — OmniRoute diagnostics and combo management
+
+### Memory
+- `chromadb` — ChromaDB vector store
+- `chromadb-preflight` — ChromaDB RAG preflight
+- `chromadb-skills-rag` — Skills RAG via ChromaDB
+
+### Browser / RPA
+- `computer-use` — Desktop GUI automation
+
+### DevOps / Homelab
+- `devops` — Umbrella for service operations
+- `devops-homelab-architecture-live-probe` — Live homelab diagram via probes
+- `devops-proxmox-operations` — Proxmox VE REST/SSH operations
+- `devops-service-management` — Miscellaneous service ops
+- `devops-tailscale-web-routing` — Tailscale web routing
+- `cloudflare-tunnel-ops` — Cloudflare Tunnels
+- `tailscale-serve-ops` — Tailscale Serve
+- `tailscale-ops` — Tailscale configuration
+
+### Messaging
+- `email` — Email from terminal
+- `email-himalaya` — Himalaya CLI IMAP/SMTP
+
+### Job / Career
+- `job-hunting` — Career transition coaching
+- `job-hunting-assistance` — End-to-end job hunting support
+- `job-search-automation` — Autonomous job board scraper
+- `job-tracker-enrichment` — Notion job tracker enrichment
+
+### Content / Research
+- `tech-ai-newsletter-digest` — Newsletter processing
+- `media` — YouTube transcripts, GIF search, music generation
+
+### Productivity
+- `productivity` — Document creation, presentations, spreadsheets
+- `markdown-to-anywhere` — Cross-posting service
+- `weekly-review` — Sunday weekly review cron
+
+### MCP / Skills Management
+- `mcp` — MCP server tooling
+- `mcp-agent-integration` — Local stdio MCP server wiring
+
+### Unverified / Not Categorized
+- 40+ skills not inspected in detail during this inventory
+- Full list at `~/.hermes/skills/`
+
+## How category discovery works
+
+Skill categories are derived from the directory layout, not from Tool Search.
+The first directory below `~/.hermes/skills/` becomes the category:
+
+- `~/.hermes/skills/devops/my-skill/SKILL.md` → category `devops`
+- `~/.hermes/skills/my-skill/SKILL.md` → no category
+
+The skill's display name and discovery description come from the YAML
+frontmatter in `SKILL.md`. Directory additions are detected automatically.
+In-place edits may take up to 30 seconds to appear because the discovery cache
+has a 30-second lifetime.
+
+See [[systems/tool-search]] for the separate external-tool discovery system.
+
+## Related
+- [[systems/hermes-agent]]
+- [[messaging-integrations]]
+- [[systems/tool-search]]
@@ -0,0 +1,36 @@
+---
+title: Terminal Backend
+type: system
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+verified_on: 2026-07-22
+confidence: high
+tags: [terminal, shell, backend, persistence]
+sources: [raw/configs/hermes-config-sanitized.txt]
+---
+
+# Terminal Backend
+
+## Purpose
+Execute shell commands, scripts, and interactive sessions for Hermes.
+
+## Current Configuration
+- **Backend:** local
+- **Working directory:** `/home/hermes/workspace`
+- **Timeout:** 180 s
+- **Home mode:** auto
+- **Container CPU:** 1
+- **Persistent shell:** true
+
+## Persistence
+- Shell state (venv, env vars, aliases) persists across terminal calls
+- Working directory fixed to `/home/hermes/workspace`
+- No containerized terminal confirmed
+
+## Known Issues
+- None observed
+
+## Related
+- [[persistent-shells]]
+- [[headless-server]]
@@ -0,0 +1,90 @@
+---
+title: Hermes Tool Search
+type: system
+status: active
+created: 2026-07-25
+updated: 2026-07-25
+review_after: 2026-10-23
+verified_on: 2026-07-25
+version_applies_to: Hermes Agent on CT460
+confidence: high
+tags: [hermes, automation, system]
+sources: [live CT460 configuration and isolated CUA validation]
+---
+
+# Hermes Tool Search
+
+## Why it is enabled
+
+Hermes previously sent every external tool schema to the model on every turn.
+That used a large part of the context window before the conversation began and
+made compaction happen much sooner.
+
+Native Hermes Tool Search is now always on. Core tools remain immediately
+available. External tools are discovered only when they are relevant, then
+described and called through the Tool Search bridge.
+
+## Active configuration
+
+File: `/home/hermes/.hermes/config.yaml`
+
+```yaml
+tools:
+  tool_search:
+    enabled: "on"
+    threshold_pct: 10
+    search_default_limit: 5
+    max_search_limit: 20
+```
+
+`enabled: "on"` is the important setting: it makes Tool Search active on every
+request instead of waiting for the tool schemas to cross an automatic token
+threshold.
+
+## Verified result
+
+- 80 external tools were deferred in the production gateway.
+- Approximately 24,151 schema tokens were removed from the always-present
+  prompt.
+- 36 core or directly visible tools remained available.
+- Estimated fixed baseline fell from about 52,955 to 29,182 tokens, a reduction
+  of roughly 45%.
+- In an isolated Computer Use test, 53 tools were discovered and the
+  `health_report` search, description, and call all succeeded.
+- Native Tool Search tests passed: 39 tests.
+
+Tool Search can add a small discovery call when an external tool is first
+needed. It should not reduce answer quality: the model retains direct access to
+core tools and can search for specialized tools by name or purpose.
+
+## New tools and new skills
+
+New external tools are automatically included in Tool Search after their tool
+provider is loaded. They do not need to be manually assigned to a category.
+
+Skills use a separate discovery system:
+
+- `~/.hermes/skills/devops/my-skill/SKILL.md` is categorized as `devops`.
+- A skill directly under `~/.hermes/skills/` has no category.
+- The skill `name` and `description` come from YAML frontmatter in `SKILL.md`.
+- Directory additions are noticed automatically. An in-place edit may take up
+  to 30 seconds to appear because of the skill cache.
+
+Tool Search therefore does not decide skill categories. The first directory
+below `~/.hermes/skills/` does.
+
+## Rollback
+
+The pre-change configuration is preserved at:
+
+`/home/hermes/.hermes/backups/config.yaml.20260725-before-tool-search`
+
+Restore that file only if Tool Search causes a confirmed regression, validate
+the YAML, and restart `hermes-gateway.service`.
+
+## Related
+
+- [[systems/hermes-agent]]
+- [[systems/skills-index]]
+- [[concepts/tool-calling]]
+- [[runbooks/gateway-resource-resilience]]