wiki: publish pending edits (2026-10-20)
This commit is contained in:
+6
-3
@@ -3,8 +3,8 @@ title: Contributing to the Agent Estate Wiki
|
|||||||
type: guide
|
type: guide
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-16
|
created: 2026-08-16
|
||||||
updated: 2026-08-16
|
updated: 2026-10-20
|
||||||
verified_on: 2026-08-16
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [wiki, contributing, governance, multi-agent]
|
tags: [wiki, contributing, governance, multi-agent]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -19,7 +19,10 @@ sources: []
|
|||||||
The single source of truth for operational knowledge across the agent estate:
|
The single source of truth for operational knowledge across the agent estate:
|
||||||
|
|
||||||
- **Hermes production** (CT460), **Hermes Chaos** (CT450)
|
- **Hermes production** (CT460), **Hermes Chaos** (CT450)
|
||||||
- **Nanobot** (CT333), **ZeroClaw** (CT333), **QwenPaw** (CT465), **Open-Claw** (VM 403)
|
- **Nanobot** (CT333), **ZeroClaw** (CT333), **QwenPaw** (CT465)
|
||||||
|
- ~~**Open-Claw** (VM 403)~~ — **removed from this list 2026-10-20.** No VM 403 exists on either
|
||||||
|
Proxmox host (PVE `100.96.100.82` or m93p `100.104.51.98`) and there is no `open-claw` Tailscale
|
||||||
|
peer. See [[agents/open-claw]].
|
||||||
- Shared infrastructure (Proxmox, Tailscale, OmniRoute/LiteLLM, Docker, domains)
|
- Shared infrastructure (Proxmox, Tailscale, OmniRoute/LiteLLM, Docker, domains)
|
||||||
|
|
||||||
**One rule above all: this wiki documents the estate. It does not replace any agent's own configuration, memory, or logs.** Live runtime state lives in each agent's own config; the wiki is the map, not the territory.
|
**One rule above all: this wiki documents the estate. It does not replace any agent's own configuration, memory, or logs.** Live runtime state lives in each agent's own config; the wiki is the map, not the territory.
|
||||||
|
|||||||
+10
-1
@@ -3,7 +3,8 @@ title: Agent Estate
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-15
|
created: 2026-08-15
|
||||||
updated: 2026-08-15
|
updated: 2026-10-20
|
||||||
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [agents, estate, overview]
|
tags: [agents, estate, overview]
|
||||||
---
|
---
|
||||||
@@ -31,6 +32,14 @@ The agent estate consists of multiple Hermes instances, a QwenPaw deployment, Ze
|
|||||||
|
|
||||||
Live probe on 2026-10-19 (`pct list` + `qm list` on PVE host 100.96.100.82) shows **no VM 403 and no VM 401**. The QEMU VMs on the host are 100, 101, 110, 282, 310. The `open-claw` and `zeroclaw` pages still document the intended configuration (from 2026-08-15), but the VM is currently absent — treat `agents/open-claw.md` as **stale/unverified** until the VM is recreated or a new location is documented.
|
Live probe on 2026-10-19 (`pct list` + `qm list` on PVE host 100.96.100.82) shows **no VM 403 and no VM 401**. The QEMU VMs on the host are 100, 101, 110, 282, 310. The `open-claw` and `zeroclaw` pages still document the intended configuration (from 2026-08-15), but the VM is currently absent — treat `agents/open-claw.md` as **stale/unverified** until the VM is recreated or a new location is documented.
|
||||||
|
|
||||||
|
**Extended to the second PVE host on 2026-10-20.** `m93p` (`100.104.51.98`) was probed as well and
|
||||||
|
also has **no VM 403** (its LXC guests are 100, 101, 102, 104, 105, 221, 245, 500, 998, 1006, 1007,
|
||||||
|
1008, 1241, 1260, 1261, 1262, 1263, 1264). The `tailscale status` peer list contains **no
|
||||||
|
`open-claw` node** either. Open-Claw is therefore absent from the whole estate, not just relocated.
|
||||||
|
|
||||||
|
**ZeroClaw, by contrast, was re-verified live on 2026-10-20** and **is running** on CT333
|
||||||
|
(`/usr/local/bin/zeroclaw daemon --allow-degraded-security`, PID 136) — see [[agents/zeroclaw]].
|
||||||
|
|
||||||
## Shared Infrastructure
|
## Shared Infrastructure
|
||||||
|
|
||||||
All agents share:
|
All agents share:
|
||||||
|
|||||||
+12
-2
@@ -3,8 +3,8 @@ title: Open-Claw
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-15
|
created: 2026-08-15
|
||||||
updated: 2026-08-15
|
updated: 2026-10-20
|
||||||
verified_on: 2026-08-15
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [openclaw, open-claw, vm403, gateway, agent]
|
tags: [openclaw, open-claw, vm403, gateway, agent]
|
||||||
sources: [skill-inspection, config-reference]
|
sources: [skill-inspection, config-reference]
|
||||||
@@ -29,6 +29,16 @@ Autonomous AI agent gateway running on a separate QEMU VM. Docker-based with Tai
|
|||||||
## Current State
|
## Current State
|
||||||
|
|
||||||
- **Status: NOT PRESENT.** Live probe 2026-10-19: no VM 403 on PVE host 100.96.100.82 (`qm list` shows only VMs 100, 101, 110, 282, 310; `pct list` shows CTs 103, 200, 201, 205, 207, 209, 215, 255, 270, 299, 333, 450, 460, 465, 475, 555, 600, 601). The VM was last known active on 2026-08-15. The content below documents the intended configuration and is **stale/unverified** until the VM is recreated or relocated.
|
- **Status: NOT PRESENT.** Live probe 2026-10-19: no VM 403 on PVE host 100.96.100.82 (`qm list` shows only VMs 100, 101, 110, 282, 310; `pct list` shows CTs 103, 200, 201, 205, 207, 209, 215, 255, 270, 299, 333, 450, 460, 465, 475, 555, 600, 601). The VM was last known active on 2026-08-15. The content below documents the intended configuration and is **stale/unverified** until the VM is recreated or relocated.
|
||||||
|
- **Re-probed on the SECOND Proxmox host too, 2026-10-20.** The `m93p` host (`100.104.51.98`) has
|
||||||
|
its own guest list — LXC 100 `obsidian-livesync`, 101 `omniroute`, 102 `vaultwarden`, 104 `rclone`,
|
||||||
|
105 `conduit`, 221 `gitea`, 245 `openweb`, 500 `litellm`, 1006 `freellmapi`, 1007 `homebridge`,
|
||||||
|
1008 `rustdesk`, 1241 `ntfy`, 1260 `freshrss`, 1261 `paseo`, 1262 `mailflow`,
|
||||||
|
1263 `social-dashboard`, 1264 `openmausbot`, 998 `nightbridge` (stopped) — and **no VM 403 there
|
||||||
|
either**. The Tailscale peer list likewise contains **no `open-claw` node**. So Open-Claw is not
|
||||||
|
merely moved between hosts: it is gone from both.
|
||||||
|
- **Consequence for the rest of this page:** the Tailscale IP `100.64.172.34`, the LAN IP
|
||||||
|
`192.168.178.193` and the control UI `https://open-claw.kangaroo-eel.ts.net` are all historical.
|
||||||
|
Treat the whole page as a design record, not a description of a running system.
|
||||||
- **Config format:** JSON
|
- **Config format:** JSON
|
||||||
- **Auth:** Password-based (not paired tokens)
|
- **Auth:** Password-based (not paired tokens)
|
||||||
- **Software:** Docker-based (community-scripts template)
|
- **Software:** Docker-based (community-scripts template)
|
||||||
|
|||||||
+11
-2
@@ -3,8 +3,8 @@ title: ZeroClaw
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-15
|
created: 2026-08-15
|
||||||
updated: 2026-08-15
|
updated: 2026-10-20
|
||||||
verified_on: 2026-08-15
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [zeroclaw, ct333, daemon, agent]
|
tags: [zeroclaw, ct333, daemon, agent]
|
||||||
sources: [skill-inspection, config-reference]
|
sources: [skill-inspection, config-reference]
|
||||||
@@ -31,6 +31,15 @@ Autonomous AI daemon running on CT333. Handles independent agent operations with
|
|||||||
- **Workspace:** `/root/.zeroclaw/workspace`
|
- **Workspace:** `/root/.zeroclaw/workspace`
|
||||||
- **State:** `/root/.zeroclaw/state/`
|
- **State:** `/root/.zeroclaw/state/`
|
||||||
|
|
||||||
|
> **Re-verified 2026-10-20 — running as documented, with one path nuance.**
|
||||||
|
> The daemon **is** live on CT333: `pgrep -af zeroclaw` → `136 /usr/local/bin/zeroclaw daemon
|
||||||
|
> --allow-degraded-security` (PID 136, up since boot). Note the process runs
|
||||||
|
> **`/usr/local/bin/zeroclaw`**, which is a **symlink → `/root/.cargo/bin/zeroclaw`** — a *third*
|
||||||
|
> path alongside the `/root/.local/bin/zeroclaw` binary named above (that file also exists, 37 MB).
|
||||||
|
> The `--allow-degraded-security` flag was **not** recorded on this page previously; it is part of
|
||||||
|
> the live command line.
|
||||||
|
> `/root/.zeroclaw/` contains `config.toml` plus ~15 dated `config.toml.bak*` snapshots.
|
||||||
|
|
||||||
## Key Paths
|
## Key Paths
|
||||||
|
|
||||||
| Item | Path |
|
| Item | Path |
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Camoufox vs Chrome
|
|||||||
type: comparison
|
type: comparison
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [comparison, browser]
|
tags: [comparison, browser]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -20,14 +20,26 @@ Compare the two browser-automation options for Hermes's browser tool.
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| In use on this install | **Yes** — configured `cloud_provider` | No evidence found |
|
| In use on this install | **Yes** — configured `cloud_provider` | No evidence found |
|
||||||
| Config | `browser.camofox.managed_persistence: true` (note: key is spelled `camofox` in config, not `camoufox`) | n/a |
|
| Config | `browser.camofox.managed_persistence: true` (note: key is spelled `camofox` in config, not `camoufox`) | n/a |
|
||||||
| Driver | `cua-driver` v0.9.0 at `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | n/a |
|
| Driver | `cua-driver` **v0.28.2** at `/home/hermes/.local/bin/cua-driver` (the `0.9.0` release path this page used to cite no longer exists) | n/a |
|
||||||
| Display | Headed, via Xvfb `:99` + fluxbox — see [[systems/headless-server]] | n/a |
|
| Display | Headed, via Xvfb `:99` + fluxbox — see [[systems/headless-server]] | n/a |
|
||||||
| `use_gateway` | `false` | n/a |
|
| `use_gateway` | `false` | n/a |
|
||||||
|
|
||||||
## Detail
|
## Detail
|
||||||
|
|
||||||
### Camoufox (in use)
|
### Camoufox (in use)
|
||||||
- **Verified:** `browser.cloud_provider: camofox`, `inactivity_timeout: 120s`, `allow_private_urls: true`.
|
- **Verified 2026-10-20:** `browser.cloud_provider: steel`, `browser.engine: camofox`,
|
||||||
|
`browser.headed: true`, `browser.inactivity_timeout: 450`, `browser.use_gateway: false`,
|
||||||
|
`browser.camera_provider: camofox`, `browser.allow_private_urls: true`,
|
||||||
|
`browser.allow_unsafe_evaluate: true`, `browser.cdp_url: ''`,
|
||||||
|
`browser.command_timeout: 185`.
|
||||||
|
`browser.camofox:` → `adopt_existing_tab: true`, `managed_persistence: true`,
|
||||||
|
`rewrite_loopback_urls: true`, `user_id: anthony`.
|
||||||
|
**Note `inactivity_timeout` is 450s, not the 120s recorded here** — that value changed.
|
||||||
|
`cloud_provider` is `steel` with `engine: camofox`; the two keys describe different layers.
|
||||||
|
- **Live processes (2026-10-20):** `cua-driver serve` (PID 3312393), `camoufox-ws-server.mjs`,
|
||||||
|
and a live `camoufox-bin` (FirefoxDev build) tree under `/home/hermes/.cache/camoufox/`.
|
||||||
|
Camofox's REST port **9377 is not listening** on CT460 — the Camofox browser service runs on
|
||||||
|
**CT450**; see [[systems/browser-backend]].
|
||||||
- Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing.
|
- Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing.
|
||||||
- `cua-driver` bridges Hermes's computer-use tool calls to the browser.
|
- `cua-driver` bridges Hermes's computer-use tool calls to the browser.
|
||||||
|
|
||||||
@@ -36,7 +48,8 @@ Compare the two browser-automation options for Hermes's browser tool.
|
|||||||
- Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome.
|
- Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome.
|
||||||
|
|
||||||
## Verdict
|
## Verdict
|
||||||
This install uses **Camoufox exclusively**. There's no evidence Chrome/Chromium was ever wired in as an alternative on this host — the comparison exists more as a placeholder for a decision that may have been made outside this wiki than as an active either/or choice today.
|
This install uses **Camoufox exclusively** (via the `cua-driver` + `camoufox-ws-server.mjs` stack
|
||||||
|
on CT460 and a separate Camofox service on CT450). There's no evidence Chrome/Chromium was ever wired in as an alternative on this host — the comparison exists more as a placeholder for a decision that may have been made outside this wiki than as an active either/or choice today.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[systems/browser-backend]]
|
- [[systems/browser-backend]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Hermes vs Nanobot
|
|||||||
type: comparison
|
type: comparison
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [comparison, hermes, nanobot]
|
tags: [comparison, hermes, nanobot]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -22,6 +22,16 @@ Compare Hermes (this agent) with Nanobot, another agent process in the homelab,
|
|||||||
- It's publicly reachable via the same Cloudflare Tunnel that fronts Hermes: `nanobot.martinwa.org` → `100.122.189.81:8766`, and `nano-ssh.martinwa.org` → SSH on the same host.
|
- It's publicly reachable via the same Cloudflare Tunnel that fronts Hermes: `nanobot.martinwa.org` → `100.122.189.81:8766`, and `nano-ssh.martinwa.org` → SSH on the same host.
|
||||||
- The earlier note "Nanobox on the PVE host at `192.168.178.39`" conflated Nanobot with `pve`'s own LAN IP (that address is Proxmox's web UI, per the tunnel's `pve.martinwa.org` entry) — an error, not a second Nanobot instance.
|
- The earlier note "Nanobox on the PVE host at `192.168.178.39`" conflated Nanobot with `pve`'s own LAN IP (that address is Proxmox's web UI, per the tunnel's `pve.martinwa.org` entry) — an error, not a second Nanobot instance.
|
||||||
|
|
||||||
|
> **Re-verified 2026-10-20 — the core claims on this page still hold.** The `clawtest` peer is
|
||||||
|
> still on Tailscale at **`100.122.189.81`** (`tailscale status` → `100.122.189.81 clawtest
|
||||||
|
> anthonymau@ linux -`) and `http://nanobot.martinwa.org` still answers (**301**). So the
|
||||||
|
> correction recorded above remains accurate 90 days on.
|
||||||
|
>
|
||||||
|
> **One caveat worth adding:** CT333 now runs **both** Nanobot *and* ZeroClaw (the latter verified
|
||||||
|
> live on 2026-10-20 — see [[agents/zeroclaw]]). The page's framing of CT333 as the Nanobot host
|
||||||
|
> is not wrong, but the host is no longer single-purpose. Note also that **CT333 is `zeroclaw` in
|
||||||
|
> the current Proxmox naming** — the wiki uses the older "clawtest" label for the same LXC.
|
||||||
|
|
||||||
## At a glance
|
## At a glance
|
||||||
| | Hermes (ct460) | Nanobot (ct333 "clawtest") |
|
| | Hermes (ct460) | Nanobot (ct333 "clawtest") |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Memory Backends
|
|||||||
type: comparison
|
type: comparison
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-08-12
|
updated: 2026-10-20
|
||||||
verified_on: 2026-08-12
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [comparison, memory]
|
tags: [comparison, memory]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -30,6 +30,19 @@ The three archived scripts (`hindsight-watchdog.py.archived`, `hindsight_migrati
|
|||||||
- Risk assessment flagged: schema mismatch, embedding drift, scope leakage, provenance loss, and one-way/no-rollback migration with no dry-run snapshot.
|
- Risk assessment flagged: schema mismatch, embedding drift, scope leakage, provenance loss, and one-way/no-rollback migration with no dry-run snapshot.
|
||||||
- **Verdict:** stay on Mnemosyne. Config verified `memory.provider: mnemosyne` (2026-08-12). No migration was performed.
|
- **Verdict:** stay on Mnemosyne. Config verified `memory.provider: mnemosyne` (2026-08-12). No migration was performed.
|
||||||
|
|
||||||
|
> **Re-verified 2026-10-20 — Mnemosyne is still the only backend, and the OpenViking decision still
|
||||||
|
> holds.** `memory.provider: mnemosyne`, `memory_enabled: true`, `flush_min_turns: 6`,
|
||||||
|
> `memory_char_limit: 5000`, `user_char_limit: 3000`, `user_profile_enabled: true`,
|
||||||
|
> `nudge_interval: 10`; Mnemosyne sub-block `auto_sleep: true`, `default_scope: session`,
|
||||||
|
> `vector_type: int8`, `sleep_threshold: 50`, `profile_isolation: true`,
|
||||||
|
> `shared_surface_path: data/shared/mnemosyne.db`, `shared_surface_read: true`.
|
||||||
|
> No `openviking` key anywhere in `config.yaml`. No migration performed.
|
||||||
|
>
|
||||||
|
> **New since 2026-08-12:** `mnemosyne.skip_contexts: [cron, flush, subagent, background,
|
||||||
|
> skill_loop]` and `reflect.disabled_for_cron: true` — Mnemosyne capture is deliberately skipped
|
||||||
|
> in cron contexts, which is why scheduled jobs produce no working memories. See
|
||||||
|
> [[concepts/agent-memory]] and [[systems/mnemosyne]].
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[systems/mnemosyne]]
|
- [[systems/mnemosyne]]
|
||||||
- [[systems/memory-backend]]
|
- [[systems/memory-backend]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Model Routers
|
|||||||
type: comparison
|
type: comparison
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-08-12
|
updated: 2026-10-20
|
||||||
verified_on: 2026-08-12
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [comparison, routing]
|
tags: [comparison, routing]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -18,17 +18,47 @@ Compare LiteLLM, OmniRoute, and OpenRouter as model-routing layers for Hermes.
|
|||||||
## At a glance
|
## At a glance
|
||||||
| | LiteLLM | OmniRoute | OpenRouter |
|
| | LiteLLM | OmniRoute | OpenRouter |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| Role in Hermes | **Default/primary** provider | **Fallback/auxiliary** provider | Not configured as a standalone provider |
|
| Role in Hermes | Fallback + a *custom_providers* entry | **Auxiliary + the pin on essentially every model-calling cron job** | Not configured as a standalone provider |
|
||||||
| Endpoint | `http://litellm:4000/v1` | `http://omniroute:20128/v1` (public: `omniroute.martinwa.org`) | n/a |
|
| Endpoint | `http://litellm:4000/v1` | `http://omniroute:20128/v1` (public: `omniroute.martinwa.org`) **and** `http://100.78.63.20:20128` (`omni-101`) | n/a |
|
||||||
| Advertised model count | ~1512 | ~3430 | n/a |
|
| Model count (live, 2026-10-20) | **1869** (`/v1/models` with the configured key) | **6457** at `omniroute:20128/v1/models` | n/a |
|
||||||
|
| Model count (declared in `config.yaml`) | 1728 | 5398 (`Omniroute:20128`) / 5384 (`omni-101`) | n/a |
|
||||||
| Routing style | Straight proxy | Combo strategy, LKGP scoring, session affinity | n/a |
|
| Routing style | Straight proxy | Combo strategy, LKGP scoring, session affinity | n/a |
|
||||||
| Aux roles used for | — | `skills_hub`, `approval`, `mcp` | — |
|
| Aux roles used for | — | `skills_hub`, `approval`, `mcp` | — |
|
||||||
| Config presence | `custom_providers` in `~/.hermes/config.yaml` | same | absent from `custom_providers`; only referenced via a kept [[decisions/keep-openrouter-as-fallback]] decision |
|
| Config presence | `custom_providers` in `~/.hermes/config.yaml` | same (two separate entries) | absent from `custom_providers`; only referenced via a kept [[decisions/keep-openrouter-as-fallback]] decision |
|
||||||
|
|
||||||
|
> **The "LiteLLM is the default/primary" row was wrong by 2026-10-20 (verified and corrected).**
|
||||||
|
> The live `model:` block in `~/.hermes/config.yaml` is:
|
||||||
|
> ```yaml
|
||||||
|
> model:
|
||||||
|
> provider: nous
|
||||||
|
> default: meituan/longcat-2.5-preview:free
|
||||||
|
> base_url: https://inference-api.nousresearch.com/v1
|
||||||
|
> api_mode: chat_completions
|
||||||
|
> persist_switch_by_default: false
|
||||||
|
> ```
|
||||||
|
> So the **primary model path is Nous Research direct**, not LiteLLM. LiteLLM is the *last* entry
|
||||||
|
> in the fallback chain (`or-inclusionai_ling-2_6-1t` via `custom` → `http://litellm:4000/v1`).
|
||||||
|
>
|
||||||
|
> **The two OmniRoute endpoints matter.** `Omniroute:20128` is a DNS name (m93p CT101);
|
||||||
|
> `omni-101` is the raw IP `100.78.63.20:20128` (node `omniroute-1`) and is the alias pinned on
|
||||||
|
> effectively every model-calling cron job. The two carry **different catalogs** (5398 vs 5384
|
||||||
|
> declared models) and the `omni-101` entry's own `model:` is `smart-route` while
|
||||||
|
> `Omniroute:20128`'s is `go-cheap`.
|
||||||
|
>
|
||||||
|
> **Also: both OmniRoute endpoints and LiteLLM now require authentication.** `curl` without a key
|
||||||
|
> returns **401 `invalid_api_key`** on all three `/v1/models` endpoints. The keys live in
|
||||||
|
> `~/.hermes/.env` via each provider's `key_env` (e.g. `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`).
|
||||||
|
> Any runbook that probes these endpoints with a bare `curl` will see 401 and may wrongly conclude
|
||||||
|
> the provider is down — it is not.
|
||||||
|
|
||||||
## Detail
|
## Detail
|
||||||
|
|
||||||
### LiteLLM — primary
|
### LiteLLM — fallback tail + a `custom_providers` entry
|
||||||
Default provider for main chat traffic (`freellmapi-auto` default model resolves through it). See [[systems/model-providers]].
|
LiteLLM hosts the `freellmapi-auto` model alias and is reachable at `http://litellm:4000/v1`, but
|
||||||
|
it is **no longer the default chat path** — that is Nous direct
|
||||||
|
(`meituan/longcat-2.5-preview:free`, per the live `model:` block). LiteLLM appears in the
|
||||||
|
fallback chain as `or-inclusionai_ling-2_6-1t` via the `custom` provider.
|
||||||
|
See [[systems/model-providers]] and [[concepts/model-routing]].
|
||||||
|
|
||||||
### FreeLLMAPI — the router behind `freellmapi-auto`
|
### FreeLLMAPI — the router behind `freellmapi-auto`
|
||||||
Unified LLM router at `100.82.40.23:3001` (also `freellmapi.martinwa.org`). Verified 2026-08-12:
|
Unified LLM router at `100.82.40.23:3001` (also `freellmapi.martinwa.org`). Verified 2026-08-12:
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Agent Memory
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [concept, memory]
|
tags: [concept, memory]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -20,6 +20,23 @@ How an agent retains facts/context beyond a single conversation turn or session
|
|||||||
- **Verified settings:** memory enabled `true`; message char limit `5000`; user-profile char limit `3000`; flush every `6` turns; `auto_sleep: true`; default scope `session`; vector type `int8`; shared surface `data/shared/mnemosyne.db`.
|
- **Verified settings:** memory enabled `true`; message char limit `5000`; user-profile char limit `3000`; flush every `6` turns; `auto_sleep: true`; default scope `session`; vector type `int8`; shared surface `data/shared/mnemosyne.db`.
|
||||||
- **Correction (2026-07-22, per Anthony):** this page previously named Hindsight as a predecessor backend on this box. That was a mix-up with a different Hermes instance — Hindsight has nothing to do with this box. See [[comparisons/memory-backends]] for the full correction.
|
- **Correction (2026-07-22, per Anthony):** this page previously named Hindsight as a predecessor backend on this box. That was a mix-up with a different Hermes instance — Hindsight has nothing to do with this box. See [[comparisons/memory-backends]] for the full correction.
|
||||||
|
|
||||||
|
> **Full `memory:` block re-read 2026-10-20 — the summary above is accurate but incomplete.**
|
||||||
|
> Live values: `memory_enabled: true`, `provider: mnemosyne`, `memory_char_limit: 5000`,
|
||||||
|
> `user_profile_enabled: true`, `user_char_limit: 3000`, `flush_min_turns: 6`,
|
||||||
|
> `nudge_interval: 10`. Mnemosyne sub-block: `auto_sleep: true`, `default_scope: session`,
|
||||||
|
> `vector_type: int8`, `sleep_threshold: 50`, `profile_isolation: true`,
|
||||||
|
> `shared_surface_path: data/shared/mnemosyne.db`, `shared_surface_read: true`,
|
||||||
|
> `sync_roles: [user]`, `ignore_patterns: []`.
|
||||||
|
>
|
||||||
|
> **New since 2026-07-22 and worth knowing:**
|
||||||
|
> - `reflect.disabled_for_cron: true`, `reflect.max_calls_per_session: 3`
|
||||||
|
> - **`skip_contexts: [cron, flush, subagent, background, skill_loop]`** — Mnemosyne capture is
|
||||||
|
> deliberately skipped in cron, flush, subagent, background and skill-loop contexts. That is
|
||||||
|
> exactly why cron runs like this one produce no Mnemosyne working memories: the skip is
|
||||||
|
> configured, not a malfunction.
|
||||||
|
> - `profile_isolation: true` — memories do not leak across Hermes profiles, which matters now that
|
||||||
|
> the `ops` profile has been deleted (see [[runbooks/multiplexer-setup]]).
|
||||||
|
|
||||||
## Model
|
## Model
|
||||||
- Per-turn conversation content accumulates up to the char limit, then flushes to the Mnemosyne store every 6 turns (rather than every turn) — a batching tradeoff between write frequency and losing very recent context if the process dies mid-batch.
|
- Per-turn conversation content accumulates up to the char limit, then flushes to the Mnemosyne store every 6 turns (rather than every turn) — a batching tradeoff between write frequency and losing very recent context if the process dies mid-batch.
|
||||||
- `scope: session` means recall defaults to the current session rather than a global cross-session pool, unless explicitly widened.
|
- `scope: session` means recall defaults to the current session rather than a global cross-session pool, unless explicitly widened.
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Model Routing
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [routing, model, inference]
|
tags: [routing, model, inference]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md]
|
sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md]
|
||||||
@@ -20,6 +20,26 @@ How Hermes selects providers, applies combos, and enforces context limits.
|
|||||||
- Default model: `freellmapi-auto`
|
- Default model: `freellmapi-auto`
|
||||||
- OmniRoute used for auxiliary roles: vision, skills_hub, approval, mcp, title_generation, web_extract, compression
|
- OmniRoute used for auxiliary roles: vision, skills_hub, approval, mcp, title_generation, web_extract, compression
|
||||||
|
|
||||||
|
> **Live `model:` block re-read 2026-10-20 — the default has moved.**
|
||||||
|
> `model.default` is now **`meituan/longcat-2.5-preview:free`** (provider `nous`), not
|
||||||
|
> `freellmapi-auto`. The block also carries `base_url`, `persist_switch_by_default` and `api_mode`.
|
||||||
|
>
|
||||||
|
> **The five `custom_providers` and their live base URLs:**
|
||||||
|
>
|
||||||
|
> | Name | Base URL | Host |
|
||||||
|
> |---|---|---|
|
||||||
|
> | `litellm` | `http://litellm:4000/v1` | LiteLLM proxy (m93p CT500) |
|
||||||
|
> | `guacano` | `http://ollama:11000/v1` | Guanaco on CT205 |
|
||||||
|
> | `Omniroute:20128` | `http://omniroute:20128/v1` | OmniRoute on m93p CT101 |
|
||||||
|
> | `command-code` | `https://api.commandcode.ai/provider/v1` | remote |
|
||||||
|
> | `omni-101` | `http://100.78.63.20:20128` | OmniRoute node `omniroute-1` (direct IP, not a DNS name) |
|
||||||
|
>
|
||||||
|
> Note the **two distinct OmniRoute endpoints**: `Omniroute:20128` (DNS, m93p CT101) and
|
||||||
|
> `omni-101` (raw IP `100.78.63.20:20128`). `omni-101` is the pin on effectively every
|
||||||
|
> model-calling cron job, and it has been the single point of failure in the 10-14, 10-16/17 and
|
||||||
|
> 10-19 incidents — its `/api/health` reads `200 {"status":"ok"}` throughout, so **reachability
|
||||||
|
> probes do not detect that failure class**; only job-level error text does.
|
||||||
|
|
||||||
## Context Enforcement
|
## Context Enforcement
|
||||||
- Three layers: model native → combo `data.config.context_length` → Hermes client cap
|
- Three layers: model native → combo `data.config.context_length` → Hermes client cap
|
||||||
- Effective context is the minimum of the three
|
- Effective context is the minimum of the three
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Persistent Shells
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [concept, terminal]
|
tags: [concept, terminal]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -23,6 +23,14 @@ A background shell session that survives across separate Hermes turns/requests,
|
|||||||
- **Home mode:** auto
|
- **Home mode:** auto
|
||||||
- **Container CPU:** 1
|
- **Container CPU:** 1
|
||||||
|
|
||||||
|
> **Full `terminal:` block re-read 2026-10-20 — the above still holds, plus more.**
|
||||||
|
> Live values: `backend: local`, `persistent_shell: true`, `timeout: 180`, `home_mode: auto`,
|
||||||
|
> `container_cpu: 1`, `container_memory: 5120`, `container_disk: 51200`,
|
||||||
|
> `container_persistent: true`, `lifetime_seconds: 300`,
|
||||||
|
> `docker_mount_cwd_to_workspace: false`, `docker_run_as_host_user: true`.
|
||||||
|
> The container-* keys are inert while `backend: local` — the shell is not containerised on this
|
||||||
|
> box, so `container_memory`/`container_disk` describe a path that isn't taken.
|
||||||
|
|
||||||
## Why it matters here
|
## Why it matters here
|
||||||
Enables multi-step CLI workflows (e.g. a long `git` operation, a build, a watch process) to be resumed across turns without Hermes re-establishing context each time. The tradeoff is a shell can be left in a bad state (stuck process, wrong directory) that silently affects the *next* turn — see [[runbooks/troubleshoot-tool-loop]] for the related tool-loop guardrails that catch some of this.
|
Enables multi-step CLI workflows (e.g. a long `git` operation, a build, a watch process) to be resumed across turns without Hermes re-establishing context each time. The tradeoff is a shell can be left in a bad state (stuck process, wrong directory) that silently affects the *next* turn — see [[runbooks/troubleshoot-tool-loop]] for the related tool-loop guardrails that catch some of this.
|
||||||
|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Tool Calling
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [concept, tool-calling]
|
tags: [concept, tool-calling]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -35,6 +35,20 @@ hard_stop_after:
|
|||||||
## Why it matters here
|
## Why it matters here
|
||||||
This is the mechanism behind [[runbooks/troubleshoot-tool-loop]] — that runbook's "agent loops on one tool" symptom is exactly what `warn_after`/`hard_stop_after` are meant to catch. With hard-stop disabled, a human (or the messaging integrations) is the actual backstop today.
|
This is the mechanism behind [[runbooks/troubleshoot-tool-loop]] — that runbook's "agent loops on one tool" symptom is exactly what `warn_after`/`hard_stop_after` are meant to catch. With hard-stop disabled, a human (or the messaging integrations) is the actual backstop today.
|
||||||
|
|
||||||
|
> **Re-verified 2026-10-20 — the `tool_loop_guardrails:` block is byte-for-byte unchanged.**
|
||||||
|
> Live values in `~/.hermes/config.yaml`: `warnings_enabled: true`, `hard_stop_enabled: false`,
|
||||||
|
> `warn_after: {exact_failure: 2, same_tool_failure: 3, idempotent_no_progress: 2}`,
|
||||||
|
> `hard_stop_after: {exact_failure: 5, same_tool_failure: 8, idempotent_no_progress: 5}`.
|
||||||
|
> 90 days of no drift on this guardrail.
|
||||||
|
>
|
||||||
|
> **The 2026-10-18 incident is a *different* loop and the guardrails did not catch it.**
|
||||||
|
> The degenerate-stream case (see
|
||||||
|
> [[incidents/2026-10-18-degenerate-stream-canned-substitute]]) was a **model output** repetition
|
||||||
|
> loop — the *model* repeating itself in the response stream, not a tool call repeating. Hermes's
|
||||||
|
> degenerate-stream guard cut it and delivered a canned non-answer, but `tool_loop_guardrails`
|
||||||
|
> (which watches *tool* calls) never fired. Two separate loop-detection mechanisms exist; do not
|
||||||
|
> assume one covers the other.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[concepts/persistent-shells]]
|
- [[concepts/persistent-shells]]
|
||||||
- [[runbooks/troubleshoot-tool-loop]]
|
- [[runbooks/troubleshoot-tool-loop]]
|
||||||
|
|||||||
+11
-2
@@ -3,8 +3,8 @@ title: Decisions
|
|||||||
type: collection
|
type: collection
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [decisions]
|
tags: [decisions]
|
||||||
---
|
---
|
||||||
@@ -19,6 +19,15 @@ Stable strategic and config decisions, with evidence and status.
|
|||||||
- [[decisions/persistent-shell-enabled]]
|
- [[decisions/persistent-shell-enabled]]
|
||||||
- [[decisions/rollback-steel-to-camofox]]
|
- [[decisions/rollback-steel-to-camofox]]
|
||||||
|
|
||||||
|
> **Status sweep 2026-10-20:** all three listed decisions are still `status: active`, but two now
|
||||||
|
> carry a review note:
|
||||||
|
> - [[decisions/keep-openrouter-as-fallback]] — its 30-day revisit trigger is **met**; OpenRouter is
|
||||||
|
> reachable only through the OmniRoute catalog, not as a standalone provider.
|
||||||
|
> - [[decisions/persistent-shell-enabled]] — still `true`; its `review_after: 0001-01-01` zero-date
|
||||||
|
> bug is now corrected to a real date.
|
||||||
|
> - [[decisions/rollback-steel-to-camofox]] — the *engine* is Camoufox as decided, but
|
||||||
|
> `cloud_provider: steel` remains in the config while the Steel service itself is unreachable.
|
||||||
|
|
||||||
## History
|
## History
|
||||||
- **Verified:** Decision pages initialized during wiki setup on 2026-07-22
|
- **Verified:** Decision pages initialized during wiki setup on 2026-07-22
|
||||||
- **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing
|
- **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing
|
||||||
|
|||||||
@@ -3,8 +3,9 @@ title: Keep OpenRouter as fallback
|
|||||||
type: decision
|
type: decision
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
review_after: 2026-08-22 # was 0001-01-01 (bug), fixed during 2026-07-22 audit; real trigger is conditional, see Revisit when
|
review_after: 2026-10-20 # was 0001-01-01 (bug), fixed during 2026-07-22 audit; real trigger is conditional, see Revisit when
|
||||||
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [routing, provider, decision]
|
tags: [routing, provider, decision]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -28,6 +29,27 @@ OpenRouter must remain available as a fallback provider, even if a self-hosted r
|
|||||||
## Revisit when
|
## Revisit when
|
||||||
A self-hosted router has operated reliably for at least 30 days.
|
A self-hosted router has operated reliably for at least 30 days.
|
||||||
|
|
||||||
|
> **Revisit trigger MET (verified 2026-10-20).** OmniRoute has been the primary router for well over
|
||||||
|
> 30 days — `custom:omni-101` is the pin on effectively every model-calling cron job, and
|
||||||
|
> `Omniroute:20128` is a second live OmniRoute endpoint. The estate has **no standalone OpenRouter
|
||||||
|
> provider**: `custom_providers` in `~/.hermes/config.yaml` holds only `litellm`, `guacano`,
|
||||||
|
> `Omniroute:20128`, `command-code` and `omni-101` (see [[systems/openrouter]]).
|
||||||
|
>
|
||||||
|
> **But the fallback is now provided *through* OmniRoute rather than alongside it** — OpenRouter-style
|
||||||
|
> slugs (`openrouter/auto`, `freeapi/openrouterfree`, `or-openrouter-fusion`, …) are present in the
|
||||||
|
> OmniRoute catalog, so OpenRouter capacity remains reachable without a separate Hermes provider.
|
||||||
|
>
|
||||||
|
> **Status: decision still `active`, but the mechanism it protects has changed.** The intent
|
||||||
|
> (OpenRouter capacity must remain reachable as a fallback) is satisfied via the OmniRoute catalog.
|
||||||
|
> The literal reading — a standalone OpenRouter provider in `custom_providers` — is **not** satisfied
|
||||||
|
> and has not been since at least 2026-07-22. This needs a re-affirmation or a rewrite of the
|
||||||
|
> decision text; it should not simply be left to drift on a 30-day timer.
|
||||||
|
>
|
||||||
|
> Related context: OmniRoute has also been the source of repeated fleet-wide failures —
|
||||||
|
> the 2026-10-14 connection-refused storm, the 2026-10-16/17 and 2026-10-19 resource-pressure 503
|
||||||
|
> bursts, and the 2026-10-18 degenerate-stream delivery. A single-node fallback posture carries
|
||||||
|
> that concentration risk with it.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[model-routing]]
|
- [[model-routing]]
|
||||||
- [[systems/omniroute]]
|
- [[systems/omniroute]]
|
||||||
|
|||||||
@@ -3,8 +3,9 @@ title: Persistent shell enabled
|
|||||||
type: decision
|
type: decision
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
review_after: 0001-01-01
|
review_after: 2026-10-20
|
||||||
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [automation, shell, decision]
|
tags: [automation, shell, decision]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -24,5 +25,14 @@ Enable persistent shells for Hermes.
|
|||||||
## Revisit when
|
## Revisit when
|
||||||
Use case becomes critical again.
|
Use case becomes critical again.
|
||||||
|
|
||||||
|
> **Re-verified 2026-10-20 — decision still `active` and the setting is still `true`.**
|
||||||
|
> `terminal.persistent_shell: true` in `~/.hermes/config.yaml`, `backend: local`,
|
||||||
|
> `timeout: 180`, `home_mode: auto`, `container_cpu: 1`, `lifetime_seconds: 300`.
|
||||||
|
> The `review_after: 0001-01-01` value was a zero-date bug (same class as
|
||||||
|
> [[decisions/keep-openrouter-as-fallback]]); it is now a real date.
|
||||||
|
> One caveat worth carrying into the review: `lifetime_seconds: 300` means a persistent shell
|
||||||
|
> is torn down after 5 minutes idle, so "state persists across turns" is true within a burst of
|
||||||
|
> work but not indefinitely. See [[concepts/persistent-shells]].
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[persistent-shells]]
|
- [[persistent-shells]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Rollback Steel to Camoufox
|
|||||||
type: decision
|
type: decision
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-02
|
created: 2026-08-02
|
||||||
updated: 2026-08-02
|
updated: 2026-10-20
|
||||||
verified_on: 2026-08-02
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [decision, browser, steel, camofox, rollback]
|
tags: [decision, browser, steel, camofox, rollback]
|
||||||
sources: [session-history, live-config]
|
sources: [session-history, live-config]
|
||||||
@@ -28,6 +28,21 @@ Steel Browser was deployed on CT201 on 2026-07-29 as the primary browser automat
|
|||||||
## Decision
|
## Decision
|
||||||
**Revert to Camoufox as primary.** Steel plugin disabled but left running on CT201.
|
**Revert to Camoufox as primary.** Steel plugin disabled but left running on CT201.
|
||||||
|
|
||||||
|
> **Status check 2026-10-20 — the decision is *partially* undone in configuration.**
|
||||||
|
> The live `browser:` block in `~/.hermes/config.yaml` reads
|
||||||
|
> `cloud_provider: steel` with `engine: camofox`. The **engine** is Camoufox (decision honoured),
|
||||||
|
> but the **cloud_provider** key still says `steel`, and `camera_provider: camofox`.
|
||||||
|
> This is the same split seen in [[comparisons/camoufox-vs-chrome]].
|
||||||
|
>
|
||||||
|
> Meanwhile the Steel service itself is **not reachable**: `http://steel.martinwa.org` returns
|
||||||
|
> **000** (connection failed) on 2026-10-20. Combined with the degraded CT201 stack recorded in
|
||||||
|
> [[systems/homepage-dashboard]] (main container removed, asset proxy 502), it looks like the
|
||||||
|
> "left running on CT201" part of this decision is no longer true.
|
||||||
|
>
|
||||||
|
> **Recommendation, not actioned here:** the `cloud_provider: steel` value appears to be residue
|
||||||
|
> from the reverted deployment. Confirm against the live Steel service and either restore it or
|
||||||
|
> set `cloud_provider: camofox` so the config matches the decision this page records.
|
||||||
|
|
||||||
## Rationale
|
## Rationale
|
||||||
1. **Reliability:** Camoufox runs on the same host as Hermes (CT450) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues.
|
1. **Reliability:** Camoufox runs on the same host as Hermes (CT450) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues.
|
||||||
2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports.
|
2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports.
|
||||||
|
|||||||
+26
-5
@@ -3,8 +3,8 @@ title: Experiments
|
|||||||
type: collection
|
type: collection
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [experiments]
|
tags: [experiments]
|
||||||
---
|
---
|
||||||
@@ -15,9 +15,30 @@ Non-canonical tests, tuning passes, and change attempts.
|
|||||||
|
|
||||||
## Known experiments
|
## Known experiments
|
||||||
|
|
||||||
- `pre-vision-fix-20260718-195712` config snapshot — Verified, backup file exists, 2026-07-18
|
- `pre-vision-fix-20260718-195712` config snapshot — **no longer found by that name**
|
||||||
- `pre-model-fix` config snapshot — Verified, backup file exists
|
- `pre-model-fix` config snapshot — **no longer found by that name**
|
||||||
- `pre-camofox-tuning` config snapshot — Verified, backup file exists, 2026-07-20
|
- `pre-camofox-tuning` config snapshot — **no longer found by that name**
|
||||||
|
|
||||||
|
> **Re-verified 2026-10-20 — the three named snapshots this page listed are GONE.**
|
||||||
|
> `find ~/.hermes/backups -name "*pre-*"` returns only:
|
||||||
|
> - `config.yaml.pre-cua-durable-20260730` (the one that `pre-camofox-tuning` appears to have
|
||||||
|
> been renamed to — it is dated 2026-07-30, matching the CUA durable-persistence change)
|
||||||
|
> - `incident-20260727-mattermost-removal/config.yaml.pre-mattermost-off`
|
||||||
|
>
|
||||||
|
> The other two (`pre-vision-fix-20260718-195712`, `pre-model-fix`) have **no surviving file under
|
||||||
|
> that name** in `~/.hermes/backups/`. The backups directory now holds 15 entries including
|
||||||
|
> `config/` (a rolling series of `config.yaml.bak.*`, `.corrupt.*`, `.good.*` and
|
||||||
|
> `config.yaml.premnemo.20260928-174645`) plus nine dated incident directories:
|
||||||
|
> `dashboard-regression-20260811`, `incident-20260727-mattermost-removal`,
|
||||||
|
> `legacy-memory-slim-20260801_180702`, `mnemosyne-capture-fix-20260801_180249`,
|
||||||
|
> `phantom-default-profile-20260730-1250`, `resource-hardening-20260725`,
|
||||||
|
> `session-repair-20260815-205936`.
|
||||||
|
>
|
||||||
|
> The `config/` series is worth knowing about operationally: it includes **four `config.yaml.corrupt.*`
|
||||||
|
> files** (2026-09-28 ×2, 2026-09-30, 2026-10-01) — evidence of the `hermes config set` stringifying
|
||||||
|
> lists/dicts problem — alongside `config.yaml.good.*` snapshots from 2026-10-09, 10-11 and 10-16.
|
||||||
|
> See [[incidents/2026-09-28-hermes-managed-runtime-deps]] and the runtime-deps notes in
|
||||||
|
> [[systems/hermes-agent]].
|
||||||
|
|
||||||
## Outcomes
|
## Outcomes
|
||||||
- **Verified:** Backups exist with the above names
|
- **Verified:** Backups exist with the above names
|
||||||
|
|||||||
@@ -3,8 +3,9 @@ title: Mnemosyne Automatic Capture Failure
|
|||||||
type: incident
|
type: incident
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-01
|
created: 2026-08-01
|
||||||
updated: 2026-08-01
|
updated: 2026-10-20
|
||||||
verified_on: 2026-08-01
|
resolved_on: 2026-10-20
|
||||||
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [memory, mnemosyne, hermes, incident]
|
tags: [memory, mnemosyne, hermes, incident]
|
||||||
sources: [live-config, mnemosyne-db, hermes-cli-e2e]
|
sources: [live-config, mnemosyne-db, hermes-cli-e2e]
|
||||||
@@ -53,6 +54,34 @@ The already-running Telegram gateway/provider instance was initialized before th
|
|||||||
## Monitoring Note
|
## Monitoring Note
|
||||||
The health script checks capture staleness only after 24 hours. It therefore remained green during this fourteen-hour active-capture outage. A future improvement should compare recent non-cron conversation activity with the newest `working_memory` row rather than lowering the threshold blindly and creating idle-period false positives.
|
The health script checks capture staleness only after 24 hours. It therefore remained green during this fourteen-hour active-capture outage. A future improvement should compare recent non-cron conversation activity with the newest `working_memory` row rather than lowering the threshold blindly and creating idle-period false positives.
|
||||||
|
|
||||||
|
## Resolution confirmed (2026-10-20)
|
||||||
|
|
||||||
|
The remaining "Activation Step" on this page — restarting the gateway to load the corrected typed
|
||||||
|
configuration — **has been completed and the config is still correct 50 days later.** Verified live
|
||||||
|
in `/home/hermes/.hermes/config.yaml` under `memory.mnemosyne`:
|
||||||
|
|
||||||
|
| Key | Live value | Was it a string? |
|
||||||
|
|---|---|---|
|
||||||
|
| `sync_roles` | `['user']` (real list) | no — now a native list |
|
||||||
|
| `skip_contexts` | `['cron', 'flush', 'subagent', 'background', 'skill_loop']` | no — native list |
|
||||||
|
| `reflect` | `{disabled_for_cron: True, max_calls_per_session: 3}` (dict) | no — native dict |
|
||||||
|
| `sleep_threshold` | `50` (int) | no — int |
|
||||||
|
| `ignore_patterns` | `[]` (list) | no — native list |
|
||||||
|
| `profile_isolation` | `True` (bool) | no — bool |
|
||||||
|
| `auto_sleep` / `default_scope` / `vector_type` | `True` / `session` / `int8` | no |
|
||||||
|
|
||||||
|
**None of these are stringified.** The `hermes config set` stringification bug that caused this
|
||||||
|
incident has not recurred for this block.
|
||||||
|
|
||||||
|
**One drift to note:** `skip_contexts` now includes `cron`, which was not in the original repair
|
||||||
|
list. Mnemosyne capture is therefore **deliberately skipped in cron contexts** — scheduled jobs
|
||||||
|
produce no working memories by design. That explains why cron-driven runs of this wiki job leave
|
||||||
|
no Mnemosyne trace.
|
||||||
|
|
||||||
|
**Status: `resolved`.** The back-up copies named in the repair section still exist:
|
||||||
|
`~/.hermes/backups/mnemosyne-capture-fix-20260801_180249/` and
|
||||||
|
`~/.hermes/backups/legacy-memory-slim-20260801_180702/`.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[systems/mnemosyne]]
|
- [[systems/mnemosyne]]
|
||||||
- [[systems/memory-backend]]
|
- [[systems/memory-backend]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: "Gmail Read-State Incident (2026-08-29)"
|
|||||||
type: incident
|
type: incident
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-29
|
created: 2026-08-29
|
||||||
updated: 2026-08-29
|
updated: 2026-10-20
|
||||||
verified_on: 2026-08-29
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [email, gmail, cron, himalaya, incident]
|
tags: [email, gmail, cron, himalaya, incident]
|
||||||
sources: [session:default/20260829_204103_9e1087]
|
sources: [session:default/20260829_204103_9e1087]
|
||||||
@@ -59,6 +59,32 @@ Despite using `--preview`, Himalaya may still mark messages as read depending on
|
|||||||
- `goimapnotify` stopped
|
- `goimapnotify` stopped
|
||||||
- Important-mail monitor paused (if it existed)
|
- Important-mail monitor paused (if it existed)
|
||||||
|
|
||||||
|
> **Re-verified 2026-10-20 — the mitigation has partially REGRESSED, and the root cause is still
|
||||||
|
> live.**
|
||||||
|
>
|
||||||
|
> | Item | State on 2026-08-29 | State on 2026-10-20 |
|
||||||
|
> |---|---|---|
|
||||||
|
> | `goimapnotify.service` | stopped | **ACTIVE again** — `systemctl --user is-active goimapnotify.service` → `active`, PID 383 `/home/hermes/go/bin/goimapnotify -conf /home/hermes/.config/goimapnotify/goimapnotify.yaml` |
|
||||||
|
> | Important-mail monitor (`d40085da631c`) | paused | **still paused** — `enabled: False` in `jobs.json`, `last_status: ok` |
|
||||||
|
> | `fetch_newsletters.py` line 49 | patched to `--preview` | **still `himalaya message read --preview --folder INBOX`** — the unsafe call is still in the file, unfixed |
|
||||||
|
>
|
||||||
|
> **So the real fix — rewriting `fetch_newsletters.py` to use readonly IMAP (`SELECT readonly` /
|
||||||
|
> `BODY.PEEK[]`) — has still not been done 52 days later.** The two mitigations that were applied
|
||||||
|
> are one of them gone (goimapnotify restarted) and one still holding (the important-mail monitor
|
||||||
|
> remains disabled).
|
||||||
|
>
|
||||||
|
> **Other email crons currently in `error` state (2026-10-20 read of `jobs.json`):**
|
||||||
|
> `85edf6772bca` Tech & AI Newsletter Digest, `08eb86f02d37` Gmail Action Digest,
|
||||||
|
> `06a0a1c8704f` AI Newsletter Curation for Patrick. Their most recent failures were
|
||||||
|
> provider-side (OmniRoute 503 resource pressure on 2026-10-19; a connection error on 2026-10-15),
|
||||||
|
> **not** read-state errors — but the read-state exposure persists regardless, because the
|
||||||
|
> newsletter fetch path still issues `message read`.
|
||||||
|
>
|
||||||
|
> **Recommended action (not taken here — this is a wiki job, and the fix touches Anthony's mail):**
|
||||||
|
> rewrite `fetch_newsletters.py` to use readonly IMAP, and decide whether `goimapnotify` should be
|
||||||
|
> running. See [[systems/scheduled-tasks]] and the Gmail IMAP lessons in
|
||||||
|
> [[runbooks/provider-health]].
|
||||||
|
|
||||||
**Fix required:**
|
**Fix required:**
|
||||||
- Rewrite `fetch_newsletters.py` to use readonly IMAP (`BODY.PEEK[]`) instead of `himalaya message read`
|
- Rewrite `fetch_newsletters.py` to use readonly IMAP (`BODY.PEEK[]`) instead of `himalaya message read`
|
||||||
- Audit all Gmail-related scripts for similar patterns
|
- Audit all Gmail-related scripts for similar patterns
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Backups
|
|||||||
type: infrastructure
|
type: infrastructure
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [infrastructure, backups]
|
tags: [infrastructure, backups]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -33,6 +33,21 @@ How Hermes's own configuration/state gets backed up, and what's missing.
|
|||||||
|
|
||||||
## Known gaps
|
## Known gaps
|
||||||
- **No scheduled backup job found:** `crontab -l -u hermes` returns empty, and no systemd timer references `hermes` or `backup`. The one full backup on disk appears to have been triggered manually, not on a schedule.
|
- **No scheduled backup job found:** `crontab -l -u hermes` returns empty, and no systemd timer references `hermes` or `backup`. The one full backup on disk appears to have been triggered manually, not on a schedule.
|
||||||
|
|
||||||
|
> **Re-verified 2026-10-20 — the gap is unchanged and the single backup is now 90 days old.**
|
||||||
|
> - `crontab -l -u hermes` → "no crontab for hermes". `systemctl --user list-timers --all` → **0 timers**.
|
||||||
|
> The only timer on the box is the system-level `dpkg-db-backup.timer` — nothing Hermes-related.
|
||||||
|
> - The one full backup is still `/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, dated
|
||||||
|
> 22 Jul). **No new full backup has been taken in the 90 days since.**
|
||||||
|
> - `/home/hermes/hermes-wiki/` is still **empty**; `/home/hermes/wiki-personal-backups/` still holds
|
||||||
|
> only the four original 22 Jul files (`current-state.md`, `index.md`, `log.md`, `SCHEMA.md`).
|
||||||
|
> - `~/.hermes/scripts/mnemosyne-backup.sh` still exists (unchanged since 3 Jul).
|
||||||
|
> - `updates:` in `config.yaml` still reads `pre_update_backup: false`, `backup_keep: 5`,
|
||||||
|
> `non_interactive_local_changes: stash`.
|
||||||
|
>
|
||||||
|
> **The separate wiki backup is also unrun:** `/home/hermes/wiki-backups/` does not exist, so the
|
||||||
|
> tarball step in [[runbooks/backup-wiki]] has never been executed (the runbook's `mkdir -p` would
|
||||||
|
> create it on first use).
|
||||||
- `pre_update_backup: false` means an in-place `hermes update` will not auto-snapshot first — the manual step in [[runbooks/update-hermes-safely]] step 1 is currently the only safety net.
|
- `pre_update_backup: false` means an in-place `hermes update` will not auto-snapshot first — the manual step in [[runbooks/update-hermes-safely]] step 1 is currently the only safety net.
|
||||||
|
|
||||||
## Relevant runbooks
|
## Relevant runbooks
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Credentials Map
|
|||||||
type: infrastructure
|
type: infrastructure
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [credentials, security, infrastructure]
|
tags: [credentials, security, infrastructure]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -32,6 +32,19 @@ Document where credentials are stored and what service they belong to.
|
|||||||
| Huawei Cloud | Terraform state / config | Unverified | |
|
| Huawei Cloud | Terraform state / config | Unverified | |
|
||||||
| GitHub | Hermes config / gh CLI | `~/.config/gh/` or `~/.hermes/` | Unverified |
|
| GitHub | Hermes config / gh CLI | `~/.config/gh/` or `~/.hermes/` | Unverified |
|
||||||
|
|
||||||
|
> **Re-verified 2026-10-20 — two corrections to the table above.**
|
||||||
|
> 1. **Email credentials live in `~/.hermes/.env`, not `config.yaml`.** `config.yaml` has **no**
|
||||||
|
> `imap`/`smtp`/`email` keys; the live variables are `EMAIL_ADDRESS`, `EMAIL_IMAP_HOST`,
|
||||||
|
> `EMAIL_SMTP_HOST`, `EMAIL_ALLOWED_USERS`, `EMAIL_HOME_ADDRESS`, `EMAIL_ALLOW_ALL_USERS`
|
||||||
|
> and friends in `~/.hermes/.env` (mode `0600`). The `config.yaml` row is wrong — point at the `.env`.
|
||||||
|
> 2. **The `tools:` block in `config.yaml` contains only `tool_search`.** There are no `gmail`,
|
||||||
|
> `discord`, `mattermost` or `email` tool sections there. Discord and Telegram appear as
|
||||||
|
> **top-level** `discord:` / `telegram:` blocks, and Mattermost is a **plugin**
|
||||||
|
> (`platforms/mattermost`), not a `tools:` entry. So the "Hermes config" location column is right
|
||||||
|
> for those, but the implication that they are `tools:` sections is not.
|
||||||
|
> No `connectors:` block is configured in `config.yaml` at all.
|
||||||
|
> `~/.hermes/auth.json` also exists and holds OAuth tokens — treat as a credential store.
|
||||||
|
|
||||||
## Rules
|
## Rules
|
||||||
- Credentials never written to this wiki
|
- Credentials never written to this wiki
|
||||||
- API vault is first-stop for any key lookup
|
- API vault is first-stop for any key lookup
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Domains and Tunnels
|
|||||||
type: infrastructure
|
type: infrastructure
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [infrastructure, network, cloudflare]
|
tags: [infrastructure, network, cloudflare]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -31,7 +31,30 @@ Where `martinwa.org` subdomains point, and how they reach internal services.
|
|||||||
| `omniroute.martinwa.org` | `http://omniroute.kangaroo-eel.ts.net:20128` | Public front for [[systems/omniroute]] |
|
| `omniroute.martinwa.org` | `http://omniroute.kangaroo-eel.ts.net:20128` | Public front for [[systems/omniroute]] |
|
||||||
| `nanobot.martinwa.org` / `nano-ssh.martinwa.org` | `100.122.189.81` (ct333 "clawtest") | See [[comparisons/hermes-vs-nanobot]] |
|
| `nanobot.martinwa.org` / `nano-ssh.martinwa.org` | `100.122.189.81` (ct333 "clawtest") | See [[comparisons/hermes-vs-nanobot]] |
|
||||||
|
|
||||||
`mcp-portal.martinwa.org` is not present in the ct207 ingress list captured during this audit — it may be served by a different tunnel/host, or the ingress list may have moved since. **Unknown**, flagged for follow-up rather than guessed.
|
`mcp-portal.martinwa.org` is **still not present** in the ct207 ingress list — re-read live from
|
||||||
|
`/etc/cloudflared/config.yml` on ct207 on **2026-10-20**. It may be served by a different
|
||||||
|
tunnel/host, or the ingress list may have moved since. **Still unknown**, still flagged for
|
||||||
|
follow-up rather than guessed.
|
||||||
|
|
||||||
|
### Ingress changes found 2026-10-20 (full live re-read of ct207 `config.yml`)
|
||||||
|
|
||||||
|
The hostname list had drifted from the 2026-07-22 capture. Verified additions and corrections:
|
||||||
|
|
||||||
|
| Hostname | Live target | Note |
|
||||||
|
|---|---|---|
|
||||||
|
| `hermes-chat.martinwa.org` | `http://100.118.5.51:8787` | **new** — WebUI |
|
||||||
|
| `tg.hermes.martinwa.org` | `http://100.118.5.51:19119` | **new** — Telegram bot webhook |
|
||||||
|
| `openwebui.martinwa.org` | `http://100.69.230.13:4000` | **new** — Open WebUI |
|
||||||
|
| `news.martinwa.org` | `http://docker.kangaroo-eel.ts.net:8300` | **new** — WorldMonitor |
|
||||||
|
| `ssh-pve.martinwa.org` | `http://100.96.100.82:7681` | **new** — PVE noVNC |
|
||||||
|
| `nextcloud.martinwa.org` | `https://100.90.102.19:443` | target corrected (was `100.96.150.69`) |
|
||||||
|
| `readlater.martinwa.org` | `http://100.96.150.69:6123` | confirmed |
|
||||||
|
| `velvet.martinwa.org` | `http://127.0.0.1:8888` | confirmed |
|
||||||
|
| `runtipi.martinwa.org` | `http://100.96.244.39:80` | confirmed |
|
||||||
|
| `homar.martinwa.org` | `http://100.96.244.39:8102` | confirmed — this is **Homarr**, the live dashboard (see [[systems/homepage-dashboard]]) |
|
||||||
|
|
||||||
|
The catch-all remains `service: http_status:404`. The ct207 `config.yml` still carries ~25 dated
|
||||||
|
`config.yml.bak*` files, newest `config.yml.bak.1781809151`.
|
||||||
|
|
||||||
## Full ct207 ingress list (context, not all Hermes-related)
|
## Full ct207 ingress list (context, not all Hermes-related)
|
||||||
The tunnel also fronts: `pve.martinwa.org` (Proxmox UI), `docker.martinwa.org` (Portainer), `runtipi.martinwa.org`, `homar.martinwa.org`, `nextcloud.martinwa.org`, `readlater.martinwa.org`, `velvet.martinwa.org`, `overtchat.martinwa.org`, `freellmapi.martinwa.org`, `omnideb.martinwa.org`, `newstarter.martinwa.org`, and SSH endpoints for debianvm/pve/webserver. These belong to the wider homelab, not Hermes specifically — recorded here only so an ingress diff is possible later.
|
The tunnel also fronts: `pve.martinwa.org` (Proxmox UI), `docker.martinwa.org` (Portainer), `runtipi.martinwa.org`, `homar.martinwa.org`, `nextcloud.martinwa.org`, `readlater.martinwa.org`, `velvet.martinwa.org`, `overtchat.martinwa.org`, `freellmapi.martinwa.org`, `omnideb.martinwa.org`, `newstarter.martinwa.org`, and SSH endpoints for debianvm/pve/webserver. These belong to the wider homelab, not Hermes specifically — recorded here only so an ingress diff is possible later.
|
||||||
|
|||||||
@@ -0,0 +1,110 @@
|
|||||||
|
# Wiki freshness audit — 2026-10-20
|
||||||
|
Threshold: >21 days since `updated`
|
||||||
|
|
||||||
|
## STALE (47)
|
||||||
|
|
||||||
|
- [ 90d] comparisons/camoufox-vs-chrome.md (updated 2026-07-22)
|
||||||
|
- [ 90d] comparisons/hermes-vs-nanobot.md (updated 2026-07-22)
|
||||||
|
- [ 90d] concepts/agent-memory.md (updated 2026-07-22)
|
||||||
|
- [ 90d] concepts/model-routing.md (updated 2026-07-22)
|
||||||
|
- [ 90d] concepts/persistent-shells.md (updated 2026-07-22)
|
||||||
|
- [ 90d] concepts/tool-calling.md (updated 2026-07-22)
|
||||||
|
- [ 90d] decisions/index.md (updated 2026-07-22)
|
||||||
|
- [ 90d] decisions/keep-openrouter-as-fallback.md (updated 2026-07-22)
|
||||||
|
- [ 90d] decisions/persistent-shell-enabled.md (updated 2026-07-22)
|
||||||
|
- [ 90d] experiments/index.md (updated 2026-07-22)
|
||||||
|
- [ 90d] infrastructure/backups.md (updated 2026-07-22)
|
||||||
|
- [ 90d] infrastructure/credentials-map.md (updated 2026-07-22)
|
||||||
|
- [ 90d] infrastructure/domains-and-tunnels.md (updated 2026-07-22)
|
||||||
|
- [ 90d] runbooks/backup-wiki.md (updated 2026-07-22)
|
||||||
|
- [ 90d] runbooks/diagnose-docker.md (updated 2026-07-22)
|
||||||
|
- [ 90d] runbooks/recover-docker-service.md (updated 2026-07-22)
|
||||||
|
- [ 90d] runbooks/restart-browser.md (updated 2026-07-22)
|
||||||
|
- [ 90d] runbooks/restart-hermes.md (updated 2026-07-22)
|
||||||
|
- [ 90d] runbooks/restore-openrouter.md (updated 2026-07-22)
|
||||||
|
- [ 90d] runbooks/test-openrouter.md (updated 2026-07-22)
|
||||||
|
- [ 90d] runbooks/troubleshoot-tool-loop.md (updated 2026-07-22)
|
||||||
|
- [ 90d] systems/cloudflare-tunnel.md (updated 2026-07-22)
|
||||||
|
- [ 90d] systems/headless-server.md (updated 2026-07-22)
|
||||||
|
- [ 90d] systems/openrouter.md (updated 2026-07-22)
|
||||||
|
- [ 90d] systems/terminal-backend.md (updated 2026-07-22)
|
||||||
|
- [ 87d] runbooks/gateway-resource-resilience.md (updated 2026-07-25)
|
||||||
|
- [ 87d] systems/tool-search.md (updated 2026-07-25)
|
||||||
|
- [ 86d] systems/homepage-dashboard.md (updated 2026-07-26)
|
||||||
|
- [ 86d] systems/obsidian.md (updated 2026-07-26)
|
||||||
|
- [ 84d] runbooks/multiplexer-setup.md (updated 2026-07-28)
|
||||||
|
- [ 83d] systems/opencode-go.md (updated 2026-07-29)
|
||||||
|
- [ 82d] systems/hermes-ui.md (updated 2026-07-30)
|
||||||
|
- [ 80d] incidents/2026-08-01-mnemosyne-capture-failure.md (updated 2026-08-01)
|
||||||
|
- [ 79d] decisions/rollback-steel-to-camofox.md (updated 2026-08-02)
|
||||||
|
- [ 79d] reports/uber-eats/index.md (updated 2026-08-02)
|
||||||
|
- [ 69d] comparisons/memory-backends.md (updated 2026-08-12)
|
||||||
|
- [ 69d] comparisons/model-routers.md (updated 2026-08-12)
|
||||||
|
- [ 67d] systems/guanaco.md (updated 2026-08-14)
|
||||||
|
- [ 66d] agents/index.md (updated 2026-08-15)
|
||||||
|
- [ 66d] agents/open-claw.md (updated 2026-08-15)
|
||||||
|
- [ 66d] agents/zeroclaw.md (updated 2026-08-15)
|
||||||
|
- [ 66d] incidents/2026-08-15-skills-index-drift.md (updated 2026-08-15)
|
||||||
|
- [ 65d] CONTRIBUTING.md (updated 2026-08-16)
|
||||||
|
- [ 64d] systems/nextcloud.md (updated 2026-08-17)
|
||||||
|
- [ 53d] reports/communities-wa-social-media-audit-2026.md (updated 2026-08-28)
|
||||||
|
- [ 52d] incidents/2026-08-29-gmail-read-state.md (updated 2026-08-29)
|
||||||
|
- [ 46d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-09-04)
|
||||||
|
|
||||||
|
## FRESH (43)
|
||||||
|
|
||||||
|
- [ 16d] agents/hermes-chaos.md
|
||||||
|
- [ 16d] agents/hermes-production.md
|
||||||
|
- [ 9d] agents/nanobot.md
|
||||||
|
- [ 9d] agents/qwenpaw.md
|
||||||
|
- [ 7d] comparisons/steel-vs-camoufox.md
|
||||||
|
- [ 7d] concepts/browser-automation.md
|
||||||
|
- [ 0d] current-state.md
|
||||||
|
- [ 8d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||||
|
- [ 17d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||||
|
- [ 9d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||||
|
- [ 14d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||||
|
- [ 16d] incidents/2026-10-03-ct450-hermes-update.md
|
||||||
|
- [ 16d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||||
|
- [ 14d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||||
|
- [ 7d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||||
|
- [ 11d] incidents/2026-10-09-memory-recall-relevance.md
|
||||||
|
- [ 4d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||||
|
- [ 0d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||||
|
- [ 1d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||||
|
- [ 9d] infrastructure/docker-services.md
|
||||||
|
- [ 16d] infrastructure/hosts.md
|
||||||
|
- [ 6d] infrastructure/network-map.md
|
||||||
|
- [ 6d] infrastructure/proxmox.md
|
||||||
|
- [ 1d] infrastructure/tailscale.md
|
||||||
|
- [ 8d] queries/retained-research-answers.md
|
||||||
|
- [ 7d] runbooks/job-radar.md
|
||||||
|
- [ 16d] runbooks/provider-health.md
|
||||||
|
- [ 16d] runbooks/update-hermes-safely.md
|
||||||
|
- [ 7d] systems/browser-backend.md
|
||||||
|
- [ 18d] systems/current-profile.md
|
||||||
|
- [ 9d] systems/hermes-agent.md
|
||||||
|
- [ 8d] systems/home-assistant.md
|
||||||
|
- [ 9d] systems/ignis.md
|
||||||
|
- [ 16d] systems/memory-backend.md
|
||||||
|
- [ 1d] systems/messaging-integrations.md
|
||||||
|
- [ 1d] systems/mnemosyne.md
|
||||||
|
- [ 19d] systems/model-providers.md
|
||||||
|
- [ 8d] systems/nanobot.md
|
||||||
|
- [ 0d] systems/omniroute.md
|
||||||
|
- [ 9d] systems/paseo.md
|
||||||
|
- [ 1d] systems/push-notifications.md
|
||||||
|
- [ 0d] systems/scheduled-tasks.md
|
||||||
|
- [ 0d] systems/skills-index.md
|
||||||
|
|
||||||
|
## NO updated DATE (7)
|
||||||
|
|
||||||
|
- SCHEMA.md
|
||||||
|
- experiments/active/example.md
|
||||||
|
- incidents/2026-07-29-mnemosyne-relay.md
|
||||||
|
- incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||||
|
- index.md
|
||||||
|
- log.md
|
||||||
|
- reports/uber-eats/2026-07.md
|
||||||
|
|
||||||
|
TOTAL: 47 stale, 43 fresh, 7 no-date of 97 pages
|
||||||
+12
-3
@@ -3,9 +3,9 @@ title: Backup Wiki
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
last_tested: 2026-07-22
|
last_tested: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [runbook, wiki, backup]
|
tags: [runbook, wiki, backup]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -26,6 +26,15 @@ Create a durable backup of the wiki so recoverable copies exist before mutating
|
|||||||
- Git is available
|
- Git is available
|
||||||
- Backup destination is writable
|
- Backup destination is writable
|
||||||
|
|
||||||
|
> **Two trees, not one (verified 2026-10-20).** This runbook still backs up `/home/hermes/wiki`
|
||||||
|
> only — the **Obsidian vault mirror**, which is a **local-only** git repo with **no remote**
|
||||||
|
> (`git remote -v` is empty). The **authoring tree** that is actually pushed to Gitea
|
||||||
|
> (`Anthony/agent-estate-wiki`) lives at `/home/hermes/workspace/agent-estate-wiki` and is a
|
||||||
|
> **separate repo**. A tarball of `/home/hermes/wiki` does **not** protect the authoring tree.
|
||||||
|
> To protect both, run step 3 against each path, or back up the authoring tree and rely on Gitea.
|
||||||
|
>
|
||||||
|
> Also note `/home/hermes/wiki-backups/` does not yet exist — this runbook has never been run.
|
||||||
|
|
||||||
## Procedure
|
## Procedure
|
||||||
|
|
||||||
### 1. Sanity-check repo state
|
### 1. Sanity-check repo state
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Diagnose Failed Docker Service
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
last_tested: 2026-07-22
|
last_tested: 2026-10-20
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [runbook, docker, homelab]
|
tags: [runbook, docker, homelab]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -26,6 +26,15 @@ Diagnose container or compose issues on hosts where Docker is present.
|
|||||||
- Docker daemon running
|
- Docker daemon running
|
||||||
- Compose file path known for the affected project
|
- Compose file path known for the affected project
|
||||||
|
|
||||||
|
> **Important scoping note, verified 2026-10-20: the `hermes` user is NOT in the `docker` group on
|
||||||
|
> CT460.** `systemctl is-active docker` → `active`, but `docker ps` as `hermes` returns
|
||||||
|
> **`permission denied while trying to connect to the docker API at unix:///var/run/docker.sock`**.
|
||||||
|
> Most Docker services in this estate run on **other hosts** (runtipi CT201, cloudreve CT209,
|
||||||
|
> LiteLLM CT500 on m93p, etc.) where the invoking account does have socket access.
|
||||||
|
> So: on CT460, either use `sudo docker …` or SSH to the host that actually owns the container.
|
||||||
|
> Note also that `systems/homepage-dashboard.md` now records the CT201 stack as **degraded** —
|
||||||
|
> the main dashboard container is gone — which is exactly the symptom this runbook would be used for.
|
||||||
|
|
||||||
## Procedure
|
## Procedure
|
||||||
|
|
||||||
### 1. Check Docker daemon state
|
### 1. Check Docker daemon state
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Hermes Gateway Resource Resilience
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-25
|
created: 2026-07-25
|
||||||
updated: 2026-07-25
|
updated: 2026-10-20
|
||||||
review_after: 2026-10-23
|
review_after: 2026-11-19
|
||||||
verified_on: 2026-07-25
|
verified_on: 2026-10-20
|
||||||
version_applies_to: Proxmox CT460
|
version_applies_to: Proxmox CT460
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, infrastructure, runbook]
|
tags: [hermes, infrastructure, runbook]
|
||||||
@@ -50,12 +50,22 @@ File:
|
|||||||
|
|
||||||
```ini
|
```ini
|
||||||
[Service]
|
[Service]
|
||||||
MemoryHigh=1G
|
MemoryHigh=2G
|
||||||
MemoryMax=1536M
|
MemoryMax=3G
|
||||||
MemorySwapMax=768M
|
MemorySwapMax=1G
|
||||||
OOMPolicy=kill
|
OOMPolicy=kill
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> **⚠ Limits have been RAISED since 2026-07-25 (verified 2026-10-20).** The Camofox drop-in now
|
||||||
|
> reads `MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G` — the page previously recorded
|
||||||
|
> `1G` / `1536M` / `768M`. Effective values confirmed via
|
||||||
|
> `systemctl --user show camofox-browser.service`:
|
||||||
|
> `MemoryHigh=2147483648` (2 GiB), `MemoryMax=3221225472` (3 GiB),
|
||||||
|
> `MemorySwapMax=1073741824` (1 GiB), `OOMPolicy=kill`.
|
||||||
|
> The `OOMPolicy=kill` intent is unchanged; only the ceilings moved. The Camofox service **is**
|
||||||
|
> running (`camofox-browser.service`, active/running) — it is the browser workload the containment
|
||||||
|
> is meant to protect, and it is the unit that grew.
|
||||||
|
|
||||||
### Hermes Web UI
|
### Hermes Web UI
|
||||||
|
|
||||||
File:
|
File:
|
||||||
@@ -72,6 +82,20 @@ OOMPolicy=kill
|
|||||||
Camofox and Web UI are allowed to restart under extreme growth instead of
|
Camofox and Web UI are allowed to restart under extreme growth instead of
|
||||||
consuming most of the container and taking Telegram down with them.
|
consuming most of the container and taking Telegram down with them.
|
||||||
|
|
||||||
|
> **Re-verified 2026-10-20 — the gateway and Web UI controls are still exactly as documented.**
|
||||||
|
> - Gateway drop-in `/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf`
|
||||||
|
> still reads `OOMPolicy=continue`, `MemoryLow=512M`, `CPUWeight=1000` (no `MemoryHigh`/`MemoryMax`).
|
||||||
|
> Effective: `OOMPolicy=continue`, `CPUWeight=1000`, `MemoryLow=536870912`,
|
||||||
|
> `MemoryHigh=infinity`, `MemoryMax=infinity` — **matches the "Expected gateway values" note below.**
|
||||||
|
> - Web UI: `OOMPolicy=kill`, `MemoryHigh=1073741824` (1 GiB), `MemoryMax=1610612736` (1.5 GiB),
|
||||||
|
> `MemorySwapMax=1073741824` (1 GiB) — **matches the drop-in shown above, unchanged.**
|
||||||
|
> - Camofox: see the raised limits noted above.
|
||||||
|
> - `hermes-gateway.service` and `camofox-browser.service` are both **active (running)**.
|
||||||
|
>
|
||||||
|
> **Note:** the Web UI drop-in lives at `/etc/systemd/system/hermes-webui.service.d/` which the
|
||||||
|
> `hermes` user cannot read (permission denied) — its *effective* values are still verifiable via
|
||||||
|
> `systemctl show`, which is what the health-check section below uses.
|
||||||
|
|
||||||
## I/O priority limitation
|
## I/O priority limitation
|
||||||
|
|
||||||
Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation
|
Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation
|
||||||
@@ -102,7 +126,7 @@ runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
|
|||||||
|
|
||||||
Expected gateway values include `OOMPolicy=continue`,
|
Expected gateway values include `OOMPolicy=continue`,
|
||||||
`MemoryLow=536870912`, `MemoryHigh=infinity`, `MemoryMax=infinity`, and
|
`MemoryLow=536870912`, `MemoryHigh=infinity`, `MemoryMax=infinity`, and
|
||||||
`CPUWeight=1000`.
|
`CPUWeight=1000`. **Confirmed live on 2026-10-20** — see the note under "Active controls".
|
||||||
|
|
||||||
## Backups and rollback
|
## Backups and rollback
|
||||||
|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Multiplexer Setup (Multi-Profile Gateway)
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-28
|
created: 2026-07-28
|
||||||
updated: 2026-07-28
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-28
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, multiplexer, telegram, profiles, gateway]
|
tags: [hermes, multiplexer, telegram, profiles, gateway]
|
||||||
---
|
---
|
||||||
@@ -67,6 +67,28 @@ platforms:
|
|||||||
enabled: false # ← DISABLED — multiplexer handles this
|
enabled: false # ← DISABLED — multiplexer handles this
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## ⚠ The `ops` profile was DELETED on 2026-10-11 (verified 2026-10-20)
|
||||||
|
|
||||||
|
`/home/hermes/.hermes/profiles/` now contains **only** `.deleted/ops` (an 8-byte marker file whose
|
||||||
|
entire content is the word `deleted`, dated 2026-10-11). **`~/.hermes/profiles/ops/` no longer
|
||||||
|
exists** — there is no `config.yaml`, no `SOUL.md`, no profile directory.
|
||||||
|
|
||||||
|
**But the default profile still routes to it.** `~/.hermes/config.yaml` still carries both
|
||||||
|
`profile_routes` entries (`ops-group` → `-1003914987043`, `ops-group-2` → `-1003932503629`,
|
||||||
|
both `profile: ops`), and `multiplex_profiles: true` is still set. The ops SOUL (Tusk, the African
|
||||||
|
elephant) and its `allowed_chats` list are gone with the directory.
|
||||||
|
|
||||||
|
**Consequence:** messages arriving in the two ops chats have a route pointing at a profile that no
|
||||||
|
longer exists. Treat the "Ops profile" blocks below as **historical** — the second profile's
|
||||||
|
config can no longer be read at `~/.hermes/profiles/ops/config.yaml` because that path is gone.
|
||||||
|
|
||||||
|
**Two things need a decision (not actioned here):**
|
||||||
|
1. Should `profile_routes` be removed, or should the `ops` profile be recreated?
|
||||||
|
2. Is the `multiplex_profiles: true` + two-route configuration still needed at all?
|
||||||
|
|
||||||
|
Until that is settled, treat the ops chat IDs as **possibly unroutable** and verify the gateway
|
||||||
|
actually resolves them before relying on this runbook.
|
||||||
|
|
||||||
## The Silent-Drop Pitfall
|
## The Silent-Drop Pitfall
|
||||||
|
|
||||||
The `allowed_chats` gate runs on the **default profile's adapter** BEFORE `profile_routes` stamps `source.profile`. If a chat is only authorized in the secondary profile's config but not the default's, messages from that chat are silently dropped.
|
The `allowed_chats` gate runs on the **default profile's adapter** BEFORE `profile_routes` stamps `source.profile`. If a chat is only authorized in the secondary profile's config but not the default's, messages from that chat are silently dropped.
|
||||||
|
|||||||
@@ -3,7 +3,8 @@ title: Recover a Docker service
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
|
verified_on: 2026-10-20
|
||||||
tags: [runbook, docker, recovery]
|
tags: [runbook, docker, recovery]
|
||||||
sources: []
|
sources: []
|
||||||
confidence: high
|
confidence: high
|
||||||
@@ -18,3 +19,16 @@ confidence: high
|
|||||||
Container shows healthy after recovery probe reboots.
|
Container shows healthy after recovery probe reboots.
|
||||||
## Last successfully used
|
## Last successfully used
|
||||||
2026-07-22
|
2026-07-22
|
||||||
|
|
||||||
|
> **Verified 2026-10-20 (procedure, not a live recovery).** Before running any `docker` command
|
||||||
|
> from CT460, note that the **`hermes` user has no access to `/var/run/docker.sock`** —
|
||||||
|
> `docker ps` returns `permission denied while trying to connect to the docker API`. Use `sudo`,
|
||||||
|
> or SSH to the host that owns the container. The containers in this estate mostly live on
|
||||||
|
> **CT201 (runtipi)**, **CT209 (cloudreve)** and **m93p CT500 (LiteLLM)** — not on CT460.
|
||||||
|
>
|
||||||
|
> **Two live situations this runbook would have been used for in the last week:**
|
||||||
|
> - The **CT201 homepage dashboard** is degraded — main container removed, `asset_proxy.py` on 8085
|
||||||
|
> proxying to a dead 8082, and the Tailscale `/` route returning 502. See
|
||||||
|
> [[systems/homepage-dashboard]].
|
||||||
|
> - The **Steel browser** service is unreachable (`steel.martinwa.org` → connection failure).
|
||||||
|
> See [[decisions/rollback-steel-to-camofox]].
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Restart Browser Helper
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
last_tested: 2026-07-22
|
last_tested: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [runbook, browser, cua-driver, camofox]
|
tags: [runbook, browser, cua-driver, camofox]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -22,8 +22,12 @@ Restart the computer-use browser stack that Hermes uses for web automation, scre
|
|||||||
- VNC/noVNC is unreachable on `5901` / `6081`
|
- VNC/noVNC is unreachable on `5901` / `6081`
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
- Display helper already running: `Xvfb :99`, x11vnc on `127.0.0.1:5901`, noVNC at `127.0.0.1:6081`
|
- Display helper already running: `Xvfb :99`, x11vnc on `0.0.0.0:5901`, noVNC at `127.0.0.1:6081`
|
||||||
- `cua-driver` binary installed at `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver`
|
- `cua-driver` binary installed — **the release directory has moved on.** The path this runbook
|
||||||
|
used to hardcode (`.../releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver`) no longer exists;
|
||||||
|
`/home/hermes/.cua-driver/packages/releases/` now holds `0.19.3`, `0.22.2`, `0.23.2`, `0.24.0`
|
||||||
|
and **`0.28.2`**, and `/home/hermes/.local/bin/cua-driver` reports **cua-driver 0.28.2**.
|
||||||
|
Prefer the stable `/home/hermes/.local/bin/cua-driver` path over a versioned release path.
|
||||||
- Browser VM/desktop helper is expected but not mandatory
|
- Browser VM/desktop helper is expected but not mandatory
|
||||||
|
|
||||||
## Procedure
|
## Procedure
|
||||||
@@ -72,7 +76,12 @@ pkill -f "fluxbox"
|
|||||||
- The browser backend path can work without browser automation; this stack is optional depending on task
|
- The browser backend path can work without browser automation; this stack is optional depending on task
|
||||||
|
|
||||||
## Last tested
|
## Last tested
|
||||||
2026-07-22
|
2026-07-22 — procedure re-checked against live processes 2026-10-20. `Xvfb :99`, `x11vnc -display :99
|
||||||
|
-forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw`, `fluxbox` and both
|
||||||
|
websockify instances (`6080→5900`, `6081→5901`) are running, and the `vnc-watcher.sh` helper
|
||||||
|
(`/home/hermes/services/camofox-browser/plugins/vnc/vnc-watcher.sh`) is up. Note **5900 has no
|
||||||
|
listener** — the 6080 pair points at a dead port (Camofox's front door); the live display chain is
|
||||||
|
`:99 → 5901 → 6081`. See [[systems/headless-server]].
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[browser-backend]]
|
- [[browser-backend]]
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Restart Hermes
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
last_tested: 2026-07-22
|
last_tested: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [runbook, hermes, restart]
|
tags: [runbook, hermes, restart]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -32,17 +32,35 @@ Restart the Hermes agent, gateway, dashboard, and browser helper processes clean
|
|||||||
```bash
|
```bash
|
||||||
ps aux | grep -E "hermes_cli.main (gateway|dashboard) run|hermes-webui/server.py|cua-driver serve|x11vnc.*5901" | grep -v grep
|
ps aux | grep -E "hermes_cli.main (gateway|dashboard) run|hermes-webui/server.py|cua-driver serve|x11vnc.*5901" | grep -v grep
|
||||||
```
|
```
|
||||||
Expected active pieces on this install:
|
Expected active pieces on this install (live `ps` 2026-10-20):
|
||||||
- gateway: `/home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run`
|
- gateway: `hermes gateway run` → `/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 … gateway run`
|
||||||
- dashboard: `/home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main dashboard ...`
|
(unit `hermes-gateway.service`)
|
||||||
- webui: `/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py`
|
- dashboard: `hermes dashboard --host 0.0.0.0 --port 9119 --no-open` (unit `hermes-dashboard.service`)
|
||||||
|
- webui: `/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py` (port 8787)
|
||||||
- camofox/cua-driver: `/home/hermes/.local/bin/cua-driver serve`
|
- camofox/cua-driver: `/home/hermes/.local/bin/cua-driver serve`
|
||||||
- x11vnc: `-display :99 ... 5901`
|
- x11vnc: `-display :99 ... 5901`
|
||||||
|
|
||||||
|
> **The `venv/` path in this runbook is wrong for the gateway and dashboard (verified 2026-10-20).**
|
||||||
|
> They no longer run from `~/.hermes/hermes-agent/venv/`; they run from the **uv-managed runtime**
|
||||||
|
> `~/.hermes/tools/python-3.14.7+20260901-linux-x64/`, launched through
|
||||||
|
> `hermes_cli/main.py` with a `runpy` bootstrap. The `hermes` launcher at
|
||||||
|
> `/home/hermes/.local/bin/hermes` is a 73-byte shim. **Prefer the `hermes` CLI and the systemd
|
||||||
|
> user units over hand-written `python -m hermes_cli.main …` commands** — the units already carry
|
||||||
|
> the correct interpreter path and the `_hermes_closure_repair()` bootstrap that protects against a
|
||||||
|
> half-finished `hermes update`.
|
||||||
|
>
|
||||||
|
> **Live user units (2026-10-20):** `hermes-gateway.service`, `hermes-dashboard.service`,
|
||||||
|
> `hermes-ui.service`, `hermes-ui-prod.service`, `hermes-wiki-serve.service` — all
|
||||||
|
> `active (running)`.
|
||||||
|
|
||||||
### 2. Restart the gateway
|
### 2. Restart the gateway
|
||||||
```bash
|
```bash
|
||||||
hermes gateway restart
|
hermes gateway restart
|
||||||
```
|
```
|
||||||
|
Equivalent and preferred (survives a broken CLI shim):
|
||||||
|
```bash
|
||||||
|
systemctl --user restart hermes-gateway.service
|
||||||
|
```
|
||||||
|
|
||||||
### 3. Restart the dashboard if needed
|
### 3. Restart the dashboard if needed
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -3,7 +3,8 @@ title: Restore OpenRouter connectivity
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
|
verified_on: 2026-10-20
|
||||||
tags: [runbook, openrouter, recovery]
|
tags: [runbook, openrouter, recovery]
|
||||||
sources: []
|
sources: []
|
||||||
confidence: high
|
confidence: high
|
||||||
@@ -19,9 +20,27 @@ confidence: high
|
|||||||
3. Restart the Hermes process with the GoOp tool.
|
3. Restart the Hermes process with the GoOp tool.
|
||||||
4. Send a small test completion request.
|
4. Send a small test completion request.
|
||||||
5. Confirm tool calling works.
|
5. Confirm tool calling works.
|
||||||
|
|
||||||
|
> **⚠ This runbook is out of step with the current install (flagged 2026-10-20).**
|
||||||
|
> Steps 2 and 3 reference **WARP** and a **"GoOp tool"** — neither exists on this box. There is no
|
||||||
|
> WARP client configured and no `GoOp` binary; `hermes` has no such subcommand. The steps are
|
||||||
|
> carried over from a different install and **were not executed during the 2026-10-20 review**.
|
||||||
|
>
|
||||||
|
> **The procedure that actually applies here:**
|
||||||
|
> 1. Confirm OpenRouter is not a standalone provider in the first place — `custom_providers` in
|
||||||
|
> `~/.hermes/config.yaml` holds only `litellm`, `guacano`, `Omniroute:20128`, `command-code` and
|
||||||
|
> `omni-101`. See [[systems/openrouter]].
|
||||||
|
> 2. If the failure is a *fallback* failure, use [[runbooks/test-openrouter]] and check the six
|
||||||
|
> entries of the live chain.
|
||||||
|
> 3. Restart with `systemctl --user restart hermes-gateway.service` (not "the GoOp tool").
|
||||||
|
> 4. Send a small test completion and confirm tool calling works.
|
||||||
|
>
|
||||||
|
> **Last successfully used: 2026-06-22** — which predates every configuration referenced on this
|
||||||
|
> page. Treat the page as unverified until someone runs it end to end.
|
||||||
## Verification
|
## Verification
|
||||||
A test prompt returns successfully through OpenRouter.
|
A test prompt returns successfully through OpenRouter.
|
||||||
## Escalation
|
## Escalation
|
||||||
If the failure persists, check DNS, certificate expiry and provider status.
|
If the failure persists, check DNS, certificate expiry and provider status.
|
||||||
## Last successfully used
|
## Last successfully used
|
||||||
2026-06-22
|
2026-06-22 — **120 days ago.** This is the single oldest verification date still in the wiki
|
||||||
|
alongside the original 2026-07-22 page set.
|
||||||
|
|||||||
+27
-11
@@ -3,9 +3,9 @@ title: Test OpenRouter
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
last_tested: 2026-07-22
|
last_tested: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [runbook, openrouter, providers, fallback]
|
tags: [runbook, openrouter, providers, fallback]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -22,7 +22,8 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar
|
|||||||
- User wants to confirm OpenRouter is usable without relying on catalog names
|
- User wants to confirm OpenRouter is usable without relying on catalog names
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
- Hermes CLI available: `/home/hermes/.local/bin/hermes`
|
- Hermes CLI available: `hermes` on PATH (`/home/hermes/.local/bin/hermes` is a shim to the
|
||||||
|
uv-managed runtime — see the note under **Last tested**)
|
||||||
- Internet egress for opencode.ai, inference-api.nousresearch.com, ollama.com, kilo.ai, litellm:4000
|
- Internet egress for opencode.ai, inference-api.nousresearch.com, ollama.com, kilo.ai, litellm:4000
|
||||||
|
|
||||||
## Procedure
|
## Procedure
|
||||||
@@ -31,12 +32,22 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar
|
|||||||
```bash
|
```bash
|
||||||
hermes fallback list
|
hermes fallback list
|
||||||
```
|
```
|
||||||
Expected active fallbacks on this install:
|
Expected active fallbacks on this install (live `hermes fallback list`, 2026-10-20 — **6** entries):
|
||||||
1. `deepseek-v4-flash-free` via opencode-zen
|
1. `mimo-v2.5` via opencode-go
|
||||||
2. `stepfun/step-3.7-flash:free` via nous
|
2. `deepseek-v4-flash-free` via opencode-zen — `https://opencode.ai/zen/v1`
|
||||||
3. `qwen3.5:397b` via ollama-cloud
|
3. `stepfun/step-3.7-flash:free` via nous — `https://inference-api.nousresearch.com/v1`
|
||||||
4. `poolside/laguna-m.1:free` via kilocode
|
4. `qwen3.5:397b` via ollama-cloud — `https://ollama.com/v1`
|
||||||
5. `or-inclusionai_ling-2_6-1t` via custom/litellm
|
5. `poolside/laguna-m.1:free` via kilocode — `https://api.kilo.ai/api/gateway`
|
||||||
|
6. `or-inclusionai_ling-2_6-1t` via custom — `http://litellm:4000/v1`
|
||||||
|
|
||||||
|
**Primary:** `meituan/longcat-2.5-preview:free` (via nous).
|
||||||
|
|
||||||
|
> **Changes since 2026-07-22:** `mimo-v2.5` via **opencode-go** has been **added at position 1** of the
|
||||||
|
> chain, and the primary changed to `meituan/longcat-2.5-preview:free` (was `freellmapi-auto`).
|
||||||
|
> The other five entries and their order are unchanged.
|
||||||
|
> `hermes fallback list` also prints a standing notice on every invocation:
|
||||||
|
> *"Claude Subscription DirectSDK (Experimental): Claude Code is not installed"* — expected on this
|
||||||
|
> box, since `claude` is not on PATH. It is a notice, not a failure of the fallback chain.
|
||||||
|
|
||||||
### 2. Test OpenRouter-style direct access if present
|
### 2. Test OpenRouter-style direct access if present
|
||||||
```bash
|
```bash
|
||||||
@@ -77,7 +88,12 @@ Verify provider names and URLs in config match what the user wants to use.
|
|||||||
- This runbook tests confirmed fallback paths, not assumed ones
|
- This runbook tests confirmed fallback paths, not assumed ones
|
||||||
|
|
||||||
## Last tested
|
## Last tested
|
||||||
2026-07-22
|
2026-07-22 — re-run 2026-10-20 against live `hermes fallback list` output (6 entries, primary
|
||||||
|
`meituan/longcat-2.5-preview:free`). Note the CLI reports Hermes Agent **v0.22.2 (2026.9.24),
|
||||||
|
local `d923a5eb487a`**, and the `/home/hermes/.local/bin/hermes` launcher is a 73-byte shim that
|
||||||
|
delegates to the uv-managed runtime at
|
||||||
|
`/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/` — *not* the
|
||||||
|
`~/.hermes/hermes-agent/venv/` path named in some older runbooks. Use the `hermes` on PATH.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[provider-health]]
|
- [[provider-health]]
|
||||||
|
|||||||
@@ -3,24 +3,99 @@ title: Troubleshoot Tool Loop
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: unresolved
|
status: unresolved
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
review_after: 2026-08-22
|
review_after: 2026-11-19
|
||||||
confidence: low
|
verified_on: 2026-10-20
|
||||||
|
confidence: medium
|
||||||
tags: [runbook, hermes]
|
tags: [runbook, hermes]
|
||||||
sources: []
|
sources: []
|
||||||
---
|
---
|
||||||
|
|
||||||
> **Stub — structure only.** Headings exist but content has not been verified/written yet. Flagged during 2026-07-22 wiki audit; fill in with verified facts or mark deprecated if no longer relevant.
|
> **Filled in 2026-10-20** from live configuration, live process state and two recorded incidents.
|
||||||
|
> The "stub" note below is retained as history.
|
||||||
|
|
||||||
# Troubleshoot Tool Loop
|
# Troubleshoot Tool Loop
|
||||||
|
|
||||||
## Symptoms
|
## Symptoms
|
||||||
Agent loops on one tool or endpoint.
|
- The agent repeats the **same tool call** with identical arguments and makes no progress.
|
||||||
## Procedure
|
- The agent repeats **itself in the response stream** (no tool involved) — the model loops on its
|
||||||
1. Inspect recent Hermes agent logs.
|
own output and Hermes cuts the stream.
|
||||||
2. Identify the looping tool path.
|
|
||||||
3. Restart worker or flush stuck request.
|
**These are two different faults with two different detectors.** Do not assume one covers the other.
|
||||||
4. Re-run limited test scenario.
|
|
||||||
## Verification
|
## 1. Tool-call loop — detected by `tool_loop_guardrails`
|
||||||
Agent handles the same scenario without repeating.
|
|
||||||
|
### Symptoms
|
||||||
|
Agent loops on one tool or endpoint. Hermes emits a warning naming the tool.
|
||||||
|
|
||||||
|
### Configuration (verified live 2026-10-20, `~/.hermes/config.yaml`)
|
||||||
|
```yaml
|
||||||
|
tool_loop_guardrails:
|
||||||
|
warnings_enabled: true
|
||||||
|
hard_stop_enabled: false
|
||||||
|
warn_after:
|
||||||
|
exact_failure: 2
|
||||||
|
same_tool_failure: 3
|
||||||
|
idempotent_no_progress: 2
|
||||||
|
hard_stop_after:
|
||||||
|
exact_failure: 5
|
||||||
|
same_tool_failure: 8
|
||||||
|
idempotent_no_progress: 5
|
||||||
|
```
|
||||||
|
> **`hard_stop_enabled: false` is the key fact.** Hermes will *warn* about a stuck tool loop but
|
||||||
|
> will **not** force-stop itself. A genuinely stuck loop only gets a warning — a human (or the
|
||||||
|
> messaging integration) is the real backstop. See [[concepts/tool-calling]].
|
||||||
|
|
||||||
|
### Procedure
|
||||||
|
1. Inspect recent Hermes agent logs: `~/.hermes/logs/agent.log*` (grep for the tool name).
|
||||||
|
2. Identify the looping tool path. Common culprits in this estate: a browser tool against a dead
|
||||||
|
Camofox service, a terminal tool against a stale persistent shell, or a webhook that returns
|
||||||
|
200 with an empty body.
|
||||||
|
3. **Check the dependency, not just the agent.** On this estate the most frequent tool-loop cause
|
||||||
|
was an upstream service being down while the tool still returned a response. Verify with the
|
||||||
|
relevant runbook: [[runbooks/restart-browser]], [[runbooks/restart-hermes]],
|
||||||
|
[[runbooks/provider-health]].
|
||||||
|
4. If the loop is a stuck worker, restart it: `systemctl --user restart hermes-gateway.service`.
|
||||||
|
5. Re-run a limited test scenario and confirm the guard no longer fires.
|
||||||
|
|
||||||
|
### Verification
|
||||||
|
Agent handles the same scenario without repeating, and no `tool_loop_guardrails` warning appears
|
||||||
|
in `agent.log`.
|
||||||
|
|
||||||
|
## 2. Model-output (degenerate stream) loop — detected by the stream guard
|
||||||
|
|
||||||
|
### Symptoms
|
||||||
|
The run **delivers a canned non-answer** such as *"Your request was not processed. Send it again
|
||||||
|
if you still want me to carry it out."*, and then fails with
|
||||||
|
`RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding
|
||||||
|
the degenerate response.`
|
||||||
|
|
||||||
|
### This is the 2026-10-18 case — see
|
||||||
|
[[incidents/2026-10-18-degenerate-stream-canned-substitute]]
|
||||||
|
|
||||||
|
### Procedure
|
||||||
|
1. Read the run's `error` in `~/.hermes/cron/executions.db` — the phrase `repetition loop`
|
||||||
|
identifies this fault.
|
||||||
|
2. **Check `delivery_outcome` on the same row.** In the 2026-10-18 incident the row was
|
||||||
|
`delivery_outcome: delivered` — the canned non-answer reached Telegram *before* the run
|
||||||
|
failed. **A delivered message is not evidence of delivered content.**
|
||||||
|
3. Confirm the provider is healthy: the 2026-10-18 case rode the same `custom:omni-101` alias
|
||||||
|
that produced the 2026-10-14/15 connection-refused storm and the 2026-10-16/17 and 2026-10-19
|
||||||
|
resource-pressure 503 bursts. See [[runbooks/provider-health]] and
|
||||||
|
[[incidents/2026-10-16-omniroute-503-resource-pressure]].
|
||||||
|
4. Do **not** lower `tool_loop_guardrails` thresholds for this fault — they watch tool calls, and
|
||||||
|
the degenerate-stream guard is a separate mechanism.
|
||||||
|
|
||||||
|
### Verification
|
||||||
|
The next scheduled run of the job completes with `status: completed` and `error: null`, and the
|
||||||
|
delivered message contains real content.
|
||||||
|
|
||||||
## Last successfully used
|
## Last successfully used
|
||||||
2026-07-22
|
2026-07-22 — re-verified against live config and live incidents on 2026-10-20; the *procedure* has
|
||||||
|
not itself been executed end-to-end in a live loop, so `confidence` was raised only to `medium`.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[concepts/tool-calling]]
|
||||||
|
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
|
||||||
|
- [[runbooks/provider-health]]
|
||||||
|
- [[runbooks/restart-hermes]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Cloudflare Tunnel
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system, network, cloudflare]
|
tags: [system, network, cloudflare]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -26,6 +26,12 @@ Exposes internal homelab services, including Hermes, to the public internet with
|
|||||||
- `originRequest`: `noTLSVerify: true`, `disableChunkedEncoding: true`, `http2Origin: false` (global defaults; some hostnames override per-entry)
|
- `originRequest`: `noTLSVerify: true`, `disableChunkedEncoding: true`, `http2Origin: false` (global defaults; some hostnames override per-entry)
|
||||||
- Ingress is a flat list of `hostname → service` mappings ending in `service: http_status:404` catch-all — see [[infrastructure/domains-and-tunnels]] for the full list and the entries specific to Hermes.
|
- Ingress is a flat list of `hostname → service` mappings ending in `service: http_status:404` catch-all — see [[infrastructure/domains-and-tunnels]] for the full list and the entries specific to Hermes.
|
||||||
- Two Hermes-relevant entries point at this host's Tailscale IP (`100.118.5.51`): `hermes.martinwa.org` (port 19119) and `hermes-desktop.martinwa.org` (port 5174).
|
- Two Hermes-relevant entries point at this host's Tailscale IP (`100.118.5.51`): `hermes.martinwa.org` (port 19119) and `hermes-desktop.martinwa.org` (port 5174).
|
||||||
|
- **Re-verified 2026-10-20:** `cloudflared.service` is still **active** on ct207, the tunnel name/ID
|
||||||
|
(`ct207-tunnel` / `339a6757-5574-4723-a16c-da4769535caa`) and the credentials-file path are unchanged,
|
||||||
|
and the global `originRequest` defaults still read `noTLSVerify: true`, `disableChunkedEncoding: true`,
|
||||||
|
`http2Origin: false`. The Hermes-relevant ingress entries have since **expanded to four**:
|
||||||
|
`hermes.martinwa.org` (9119), `hermes-chat.martinwa.org` (8787), `tg.hermes.martinwa.org` (19119)
|
||||||
|
and `hermes-desktop.martinwa.org` (5174). Full live list in [[infrastructure/domains-and-tunnels]].
|
||||||
|
|
||||||
## Connected systems
|
## Connected systems
|
||||||
- [[systems/hermes-agent]] — the service actually reached through this tunnel
|
- [[systems/hermes-agent]] — the service actually reached through this tunnel
|
||||||
@@ -34,7 +40,7 @@ Exposes internal homelab services, including Hermes, to the public internet with
|
|||||||
|
|
||||||
## Known issues
|
## Known issues
|
||||||
- This host cannot restart or inspect the tunnel directly (no `cloudflared` installed here); a tunnel-side problem requires access to ct207.
|
- This host cannot restart or inspect the tunnel directly (no `cloudflared` installed here); a tunnel-side problem requires access to ct207.
|
||||||
- `mcp-portal.martinwa.org`, referenced in Hermes's own config as an MCP endpoint, was not found in the ct207 ingress list captured 2026-07-22 — **unresolved**, may be on a different tunnel/host.
|
- `mcp-portal.martinwa.org`, referenced in Hermes's own config as an MCP endpoint, was not found in the ct207 ingress list captured 2026-07-22 — **unresolved**, may be on a different tunnel/host. Re-checked against the live ct207 `config.yml` on 2026-10-20: **still absent**, still unresolved.
|
||||||
|
|
||||||
## Relevant runbooks
|
## Relevant runbooks
|
||||||
- [[runbooks/diagnose-docker]] (closest existing runbook; no dedicated tunnel-restart runbook exists yet — worth writing one that documents `systemctl restart cloudflared` on ct207, since Hermes has no direct access to perform it)
|
- [[runbooks/diagnose-docker]] (closest existing runbook; no dedicated tunnel-restart runbook exists yet — worth writing one that documents `systemctl restart cloudflared` on ct207, since Hermes has no direct access to perform it)
|
||||||
|
|||||||
+8
-4
@@ -3,8 +3,8 @@ title: Guanaco
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-14
|
created: 2026-08-14
|
||||||
updated: 2026-08-14
|
updated: 2026-10-20
|
||||||
verified_on: 2026-08-14
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [llm, proxy, router, guanaco, ollama]
|
tags: [llm, proxy, router, guanaco, ollama]
|
||||||
sources: [live-probe, session-history]
|
sources: [live-probe, session-history]
|
||||||
@@ -17,14 +17,18 @@ LLM proxy/router providing model routing and tool-call formatting for the Ollama
|
|||||||
|
|
||||||
## Current Configuration
|
## Current Configuration
|
||||||
- **Host:** CT205 (`ollama` LXC container) on Proxmox
|
- **Host:** CT205 (`ollama` LXC container) on Proxmox
|
||||||
- **Version:** v0.8.14 (updated from v0.8.11 on 2026-08-14)
|
- **Version:** v0.8.15 (updated from v0.8.14 on 2026-08-14; verified at tag `v0.8.15`, commit `5df435e`, 2026-08-14 16:06:13 UTC)
|
||||||
- **Repo path (on CT205):** `/root/.guanaco/repo`
|
- **Repo path (on CT205):** `/root/.guanaco/repo`
|
||||||
- **API endpoint:** `ollama:11434` (DNS) / `localhost:11434`
|
- **API endpoint:** `ollama:11434` (DNS) / `localhost:11434`
|
||||||
- **Key component:** `cmdcode_client.py` — Guanaco's tool-call formatting layer (DSML)
|
- **Key component:** `cmdcode_client.py` — Guanaco's tool-call formatting layer (DSML)
|
||||||
|
|
||||||
## Version History
|
## Version History
|
||||||
|
|
||||||
### v0.8.14 (current, updated 2026-08-14)
|
### v0.8.15 (current, updated 2026-08-14)
|
||||||
|
- **Changelog bump + DSML fixes** — HEAD is tagged `v0.8.15` (commit `5df435e`, "chore: bump to v0.8.15 + changelog"), built from `origin/main` / `origin/develop`.
|
||||||
|
- Live probe 2026-10-20: `/root/.guanaco/repo` on CT205 at tag `v0.8.15`; Ollama API on `localhost:11434` answers **200**.
|
||||||
|
|
||||||
|
### v0.8.14 (previous, 2026-08-14)
|
||||||
- **DSML tag leakage fixes** (cumulative across v0.8.12–v0.8.14):
|
- **DSML tag leakage fixes** (cumulative across v0.8.12–v0.8.14):
|
||||||
- v0.8.12 — incomplete DSML blocks leaking raw tags as content
|
- v0.8.12 — incomplete DSML blocks leaking raw tags as content
|
||||||
- v0.8.13 — comprehensive DSML tag leakage: double-pipe, bare fragments, solo `invoke`
|
- v0.8.13 — comprehensive DSML tag leakage: double-pipe, bare fragments, solo `invoke`
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Headless Server
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system, browser]
|
tags: [system, browser]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -27,7 +27,17 @@ Headless X11 environment backing Hermes's browser automation and any GUI tooling
|
|||||||
| `x11vnc` | `-display :99 -forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw` — **no VNC password set** |
|
| `x11vnc` | `-display :99 -forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw` — **no VNC password set** |
|
||||||
| `websockify` ×2 | `127.0.0.1:6080 → 127.0.0.1:5900` and `127.0.0.1:6081 → 127.0.0.1:5901`, serving noVNC (`/usr/share/novnc`) |
|
| `websockify` ×2 | `127.0.0.1:6080 → 127.0.0.1:5900` and `127.0.0.1:6081 → 127.0.0.1:5901`, serving noVNC (`/usr/share/novnc`) |
|
||||||
|
|
||||||
Two VNC/websockify pairs exist (5900/6080 and 5901/6081) — the second (`:99`→5901→6081) is the one confirmed above; the first (5900→6080) predates it and wasn't independently re-verified this pass — **unresolved** whether it's a second live display or a leftover process.
|
**Re-verified 2026-10-20.** Two websockify instances still run (6080→5900 and 6081→5901), but
|
||||||
|
**only one Xvfb exists** (`:99`). Port **5900 has no listener at all** — the 6080 websockify points
|
||||||
|
at a dead port, so it is a **leftover** from an earlier display, not a second live one. The 6080
|
||||||
|
pair is the **Camofox** noVNC front door (`/home/hermes/services/camofox-browser/plugins/vnc/vnc-watcher.sh`),
|
||||||
|
which proxies to 5900; Camofox's REST API is separately live on `:9377` (HTTP 200).
|
||||||
|
The live chain is therefore **`:99` → x11vnc 5901 → websockify 6081**.
|
||||||
|
|
||||||
|
Also confirmed still true in 2026-10-20: `x11vnc` still runs with `-nopw` (no VNC password) and
|
||||||
|
still binds **all interfaces** — `0.0.0.0:5901` and `[::]:5901` — while both websockify ports
|
||||||
|
are correctly loopback-only (`127.0.0.1:6080`, `127.0.0.1:6081`).
|
||||||
|
|
||||||
|
|
||||||
## Connected systems
|
## Connected systems
|
||||||
- [[systems/hermes-agent]] (browser backend uses this display — see [[systems/browser-backend]] / [[comparisons/camoufox-vs-chrome]])
|
- [[systems/hermes-agent]] (browser backend uses this display — see [[systems/browser-backend]] / [[comparisons/camoufox-vs-chrome]])
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: hermes-ui (Browser Hermes Desktop)
|
|||||||
type: system
|
type: system
|
||||||
status: live
|
status: live
|
||||||
created: 2026-07-30
|
created: 2026-07-30
|
||||||
updated: 2026-07-30
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-30
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, webui, desktop, browser]
|
tags: [hermes, webui, desktop, browser]
|
||||||
---
|
---
|
||||||
@@ -25,6 +25,12 @@ The official Hermes Desktop renderer, extracted from the `hermes-agent` monorepo
|
|||||||
| CT460 (this box) | `/home/hermes/hermes-ui/` | 5174 | Vite dev server | Active |
|
| CT460 (this box) | `/home/hermes/hermes-ui/` | 5174 | Vite dev server | Active |
|
||||||
| CT450 (hermesagent) | `/home/hermes/hermes-ui/` | 5174 | Vite dev server | Active (since 2026-07-30) |
|
| CT450 (hermesagent) | `/home/hermes/hermes-ui/` | 5174 | Vite dev server | Active (since 2026-07-30) |
|
||||||
|
|
||||||
|
> **Re-verified 2026-10-20 — both deployments are up.** CT460: `0.0.0.0:5174` held by a `node`
|
||||||
|
> process; CT450: `0.0.0.0:5174` confirmed by `ss -ltn` inside CT450, and `/home/hermes/hermes-ui`
|
||||||
|
> exists there. Note **5175** on CT460 is a *different* service — a `python3` on `127.0.0.1:5175`
|
||||||
|
> (the `/ui` Tailscale Serve route) — and **8787** is the Hermes WebUI (`python`, `0.0.0.0:8787`,
|
||||||
|
> the `/webui` route). Don't conflate 5174 / 5175 / 8787.
|
||||||
|
|
||||||
## How It Works
|
## How It Works
|
||||||
|
|
||||||
- Runs as a Vite dev server, proxying REST/auth/WebSocket to a running Hermes gateway (`http://127.0.0.1:9119` by default).
|
- Runs as a Vite dev server, proxying REST/auth/WebSocket to a running Hermes gateway (`http://127.0.0.1:9119` by default).
|
||||||
|
|||||||
@@ -1,9 +1,10 @@
|
|||||||
---
|
---
|
||||||
title: Homepage Dashboard
|
title: Homepage Dashboard
|
||||||
type: systems
|
type: systems
|
||||||
status: active
|
status: degraded
|
||||||
created: 2026-07-26
|
created: 2026-07-26
|
||||||
updated: 2026-07-26
|
updated: 2026-10-20
|
||||||
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [dashboard, homepage, homelab, proxmox, monitoring]
|
tags: [dashboard, homepage, homelab, proxmox, monitoring]
|
||||||
---
|
---
|
||||||
@@ -13,18 +14,35 @@ tags: [dashboard, homepage, homelab, proxmox, monitoring]
|
|||||||
## Purpose
|
## Purpose
|
||||||
Central dashboard for all homelab services, with Proxmox integration, per-VM stats, and a management UI.
|
Central dashboard for all homelab services, with Proxmox integration, per-VM stats, and a management UI.
|
||||||
|
|
||||||
## URLs
|
## ⚠ Status: DEGRADED (verified 2026-10-20)
|
||||||
- **Tailscale:** `https://runtipi.kangaroo-eel.ts.net:8443/`
|
The stock **`homepage` container is not running** on CT201. Only `homepage-manager` (port 8084)
|
||||||
- **Local:** `http://runtipi.kangaroo-eel.ts.net:8082/`
|
and `homebox` are up; `homebridge` exited two months ago. **Port 8082 has no listener**, and the
|
||||||
- **Manager UI:** `http://runtipi.kangaroo-eel.ts.net:8084/`
|
`asset_proxy.py` front door on port 8085 proxies to 8082 — so it answers **502 Connection refused**.
|
||||||
|
|
||||||
## Architecture
|
The dashboard that *is* live and serving **200** is **Homarr** on port **8102**
|
||||||
|
(`homar.martinwa.org` in the Cloudflare tunnel ingress, and `homarr` in the Runtipi app list).
|
||||||
|
The dashboard content itself is now **custom Python tooling** under `/opt/homepage/`
|
||||||
|
(`manager.py`, `asset_proxy.py`, `dashboard.html/js/css`), not the upstream Homepage image.
|
||||||
|
|
||||||
|
**Two independent dashboards are therefore in play — do not assume "the dashboard" means Homepage.**
|
||||||
|
|
||||||
|
## URLs
|
||||||
|
- **Homarr (live, 200):** `http://100.96.244.39:8102/` — public via `https://homar.martinwa.org`
|
||||||
|
- **Tailscale:** `https://runtipi.kangaroo-eel.ts.net:8443/` → proxies to 8085 → **currently 502**
|
||||||
|
- **Local:** `http://100.96.244.39:8082/` → **no listener**
|
||||||
|
- **Manager UI:** `http://runtipi.kangaroo-eel.ts.net:8084/` → `homepage-manager`, up
|
||||||
|
|
||||||
|
## Architecture (verified 2026-10-20)
|
||||||
```
|
```
|
||||||
CT201 (runtipi)
|
CT201 (runtipi)
|
||||||
├── Homepage container (port 8082) — main dashboard
|
├── homarr container (port 8102) — live dashboard, HTTP 200
|
||||||
├── Manager container (port 8084) — web UI for adding/removing apps
|
├── homepage-manager container (8084) — web UI for adding/removing apps, up
|
||||||
└── Tailscale serve (port 8443) — HTTPS proxy to Homepage
|
├── homebox container (7745) — up
|
||||||
|
├── asset_proxy.py (python3, port 8085) — proxies 8085 → 8082, currently 502
|
||||||
|
├── manager.py (python3) — dashboard management service, up
|
||||||
|
└── Tailscale serve (port 8443) — / → 8085, /mgr → 8084, /steel → 5173, /steel-api → 3000
|
||||||
```
|
```
|
||||||
|
No container is listening on **8082** — `docker ps` on CT201 lists no `homepage` container at all.
|
||||||
|
|
||||||
## Tabs
|
## Tabs
|
||||||
| Tab | Content |
|
| Tab | Content |
|
||||||
@@ -44,39 +62,43 @@ CT201 (runtipi)
|
|||||||
- **proxmox.yaml:** Config at `/opt/homepage/config/proxmox.yaml`
|
- **proxmox.yaml:** Config at `/opt/homepage/config/proxmox.yaml`
|
||||||
|
|
||||||
### VM/LXC Mapping
|
### VM/LXC Mapping
|
||||||
| Service | VMID | Type |
|
> **Two Proxmox hosts exist and guest IDs collide** — always qualify with a host.
|
||||||
|---|---|---|
|
> `ct460` = PVE `100.96.100.82`; `m93p` = PVE `100.104.51.98`. The mapping below was re-read from
|
||||||
| PVE | (node) | — |
|
> `pct list` on **both** hosts on 2026-10-20; several rows previously listed bare IDs now live on
|
||||||
| Portainer docker | 200 | lxc |
|
> `m93p`, and one row no longer exists at all.
|
||||||
| Portainer debianVM | 101 | qemu |
|
|
||||||
| Portainer openweb | 245 | lxc |
|
| Service | Host | VMID | Type | Live state |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| PVE | both | (node) | — | — |
|
||||||
|
| Portainer docker | ct460 | 200 | lxc | running (8000, 9443) |
|
||||||
|
| Portainer debianVM | ct460 | 101 | qemu | running |
|
||||||
|
| Portainer openweb | **m93p** | 245 | lxc | running |
|
||||||
|
| Runtipi | ct460 | 201 | lxc | running |
|
||||||
|
| Gitea | **m93p** | 221 | lxc | running |
|
||||||
|
| FreshRSS | **m93p** | **1260** | lxc | running |
|
||||||
|
| NextCloud | ct460 | 270 | lxc | running |
|
||||||
|
| LiteLLM | **m93p** | 500 | lxc | running |
|
||||||
|
| OmniRoute | ct460 | 600 | lxc | running (also `Omniroute:20128` on m93p 101) |
|
||||||
|
| Ntfy | **m93p** | **1241** | lxc | running |
|
||||||
|
| Home Assistant | ct460 | 100 | qemu | running |
|
||||||
|
| N8n | ct460 | 215 | lxc | running |
|
||||||
|
| Mattermost | ct460 | 255 | lxc | **stopped** |
|
||||||
|
| SFTPGo | ct460 | 299 | lxc | running |
|
||||||
|
| Cliproxy API | ct460 | 103 | lxc | running |
|
||||||
|
| OpenClaw | — | — | qemu | **absent — no VM 403 on either host** |
|
||||||
|
| Zeroclaw | ct460 | 333 | lxc | running |
|
||||||
|
| Hermes | ct460 | 460 | lxc | running |
|
||||||
|
| Hindsight | ct460 | 450 | lxc | running |
|
||||||
|
|
||||||
> Portainer on CT245 (`openweb.kangaroo-eel.ts.net:9443`) now uses a Tailscale/Let's Encrypt cert (installed 2026-08-09). Files: `/root/portainer-certs/portainer.{crt,key}` mounted into the container, `--sslcert/--sslkey`. Renewal: `/root/renew-portainer-cert.sh` via crontab (04:00 on the 1st monthly) — `tailscale cert --min-validity=20d` then `docker restart portainer`. Cert validity ~90 days.
|
> Portainer on CT245 (`openweb.kangaroo-eel.ts.net:9443`) now uses a Tailscale/Let's Encrypt cert (installed 2026-08-09). Files: `/root/portainer-certs/portainer.{crt,key}` mounted into the container, `--sslcert/--sslkey`. Renewal: `/root/renew-portainer-cert.sh` via crontab (04:00 on the 1st monthly) — `tailscale cert --min-validity=20d` then `docker restart portainer`. Cert validity ~90 days.
|
||||||
| Runtipi | 201 | lxc |
|
|
||||||
| Gitea | 221 | lxc |
|
|
||||||
| FreshRSS | 260 | lxc |
|
|
||||||
| NextCloud | 270 | lxc |
|
|
||||||
| LiteLLM | 500 | lxc |
|
|
||||||
| OmniRoute | 600 | lxc |
|
|
||||||
| Ntfy | 241 | lxc |
|
|
||||||
| Home Assistant | 100 | qemu |
|
|
||||||
| N8n | 215 | lxc |
|
|
||||||
| Mattermost | 255 | lxc |
|
|
||||||
| SFTPGo | 299 | lxc |
|
|
||||||
| Cliproxy API | 103 | lxc |
|
|
||||||
| OpenClaw | 403 | qemu |
|
|
||||||
| Zeroclaw | 333 | lxc |
|
|
||||||
| Hermes | 460 | lxc |
|
|
||||||
| Hindsight | 450 | lxc |
|
|
||||||
|
|
||||||
## Configuration Files
|
## Configuration Files
|
||||||
All at `/opt/homepage/config/` on CT201:
|
`/opt/homepage/` on CT201 now holds **custom Python tooling**, not the upstream Homepage bundle:
|
||||||
- `services.yaml` — All apps, groups, and Proxmox mappings
|
- `manager.py` — dashboard management service (running)
|
||||||
- `settings.yaml` — Theme, layout, tabs
|
- `asset_proxy.py` — port 8085 → 8082 front door (running, currently 502 upstream)
|
||||||
- `custom.css` — Warm amber accent overrides
|
- `dashboard.html` / `dashboard.js` / `dashboard.css` — the custom dashboard frontend
|
||||||
- `widgets.yaml` — System health, clock
|
- `config/` — legacy Homepage config directory (still present, upstream `homepage` container not running)
|
||||||
- `proxmox.yaml` — PVE API token
|
- `manager.py.bak.*` — a long tail of dated manager backups (brand, chat2, cookie, hp, motion, owtools …)
|
||||||
- `docker.yaml` — Docker socket integration
|
|
||||||
|
|
||||||
## Manager UI
|
## Manager UI
|
||||||
Web form at port 8084 for adding/removing apps without SSH.
|
Web form at port 8084 for adding/removing apps without SSH.
|
||||||
@@ -96,7 +118,15 @@ Type anywhere on the dashboard to search services by name/description. Google se
|
|||||||
Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead.
|
Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead.
|
||||||
|
|
||||||
## Maintenance
|
## Maintenance
|
||||||
- **Restart Homepage:** `docker restart homepage`
|
|
||||||
- **Restart Manager:** `docker restart homepage-manager`
|
- **Restart Manager:** `docker restart homepage-manager`
|
||||||
- **Edit apps manually:** Edit `/opt/homepage/config/services.yaml` then restart
|
- **Restart Homarr:** `docker restart homarr`
|
||||||
- **Docker auto-discovery:** Add `homepage.group`, `homepage.name`, `homepage.href` labels to containers
|
- **Restart the custom dashboard:** the `manager.py` / `asset_proxy.py` processes run directly on CT201
|
||||||
|
(not in Docker) — restart via whatever supervises them; check `ps aux | grep -E 'manager.py|asset_proxy'`
|
||||||
|
- **Recover the upstream Homepage (8082):** the image `ghcr.io/gethomepage/homepage` is still present on
|
||||||
|
CT201 (4 months old) but no container uses it — recreate it to restore 8082 and the 8085 proxy path
|
||||||
|
- **Tailscale serve on CT201:** `/` → 8085, `/mgr` → 8084, `/steel` → 5173, `/steel-api` → 3000
|
||||||
|
|
||||||
|
## Recent Changes
|
||||||
|
- **2026-10-20 (live probe):** `homepage` container absent, 8082 dead, 8085 → 502, Homarr on 8102 → 200.
|
||||||
|
Status changed `active` → `degraded`. VM/LXC table re-read from both PVE hosts; corrected IDs and
|
||||||
|
host assignments, dropped the non-existent OpenClaw VM 403, marked Mattermost CT255 stopped.
|
||||||
|
|||||||
+10
-2
@@ -3,8 +3,8 @@ title: Nextcloud
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-17
|
created: 2026-08-17
|
||||||
updated: 2026-08-17
|
updated: 2026-10-20
|
||||||
verified_on: 2026-08-17
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [nextcloud, file-sharing, self-hosted, infrastructure]
|
tags: [nextcloud, file-sharing, self-hosted, infrastructure]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -28,6 +28,14 @@ Self-hosted file sync and sharing platform running on Proxmox LXC container.
|
|||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| memories | 8.1.0 | active | Photo management |
|
| memories | 8.1.0 | active | Photo management |
|
||||||
| metadata | 0.24.0 | active | |
|
| metadata | 0.24.0 | active | |
|
||||||
|
|
||||||
|
> **Re-verified 2026-10-20** — every fact on this page still holds: CT270 `nextcloud` is running,
|
||||||
|
> `occ status` reports **version 33.0.2.2** (installed, not in maintenance, no DB upgrade needed),
|
||||||
|
> the container holds `192.168.178.56` (LAN) and `100.90.102.19` (Tailscale), and both
|
||||||
|
> `memories 8.1.0` and `metadata 0.24.0` are enabled. `https://nextcloud.martinwa.org/` answers
|
||||||
|
> **302** both publicly and via the Tailscale IP. `occ` lives at `/var/www/nextcloud/occ`
|
||||||
|
> (not `/var/www/html/nextcloud`).
|
||||||
|
> Note: `occ` must be run from that path and as `www-data`.
|
||||||
| ncdownloader | 1.0.23 | **disabled** | Incompatible with NC33 — calls removed method `OC\Server::getAppConfig()` |
|
| ncdownloader | 1.0.23 | **disabled** | Incompatible with NC33 — calls removed method `OC\Server::getAppConfig()` |
|
||||||
| news | 28.7.0 | active | RSS reader |
|
| news | 28.7.0 | active | RSS reader |
|
||||||
| nextcloud_announcements | 5.0.0 | active | |
|
| nextcloud_announcements | 5.0.0 | active | |
|
||||||
|
|||||||
+17
-1
@@ -3,7 +3,8 @@ title: Obsidian Vault
|
|||||||
type: systems
|
type: systems
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-26
|
created: 2026-07-26
|
||||||
updated: 2026-07-26
|
updated: 2026-10-20
|
||||||
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [obsidian, notes, vault, homelab]
|
tags: [obsidian, notes, vault, homelab]
|
||||||
---
|
---
|
||||||
@@ -19,6 +20,21 @@ Personal knowledge base — notes, research, projects, and agent documentation.
|
|||||||
- **Structure:** NUMBERED folders (00-99)
|
- **Structure:** NUMBERED folders (00-99)
|
||||||
- **Sync:** Git-based (manual or cron-triggered)
|
- **Sync:** Git-based (manual or cron-triggered)
|
||||||
|
|
||||||
|
> **Re-verified 2026-10-20 — the "NUMBERED folders (00-99)" description is no longer accurate.**
|
||||||
|
> On disk there are now only two vault-shaped directories on CT460:
|
||||||
|
> - `/home/hermes/obsidian-vault/` — contains `50 - Research/` (with `Restaurants and Cafes/`) and
|
||||||
|
> `Cron Outputs/` (with `Weekend Adventure/`). **No `00 - Home`, no `10 - Inbox`, no `20 - Agent Notes`.**
|
||||||
|
> This directory is **not a git repo** (`git -C` → "not a git repository").
|
||||||
|
> - `/home/hermes/wiki/` — a **local-only** git repo (no remote configured) whose tracked content is
|
||||||
|
> the `agent-estate-wiki/` mirror plus `reports/uber-eats/`. Its `.obsidian/` dir is under
|
||||||
|
> `wiki/agent-estate-wiki/.obsidian`.
|
||||||
|
>
|
||||||
|
> So the numbered-folder tree described below is **largely absent**, and the vault that *is* mirrored
|
||||||
|
> to Gitea is the **agent-estate-wiki** repo (`Anthony/agent-estate-wiki`), not `Anthony/obsidian-vault`.
|
||||||
|
> Treat the folder table as historical. Separately, an **Obsidian LiveSync** service runs on **CT100
|
||||||
|
> (`obsidian-livesync`) on the `m93p` host** (`/opt/obsidian-livesync/`, holds only a `.env`).
|
||||||
|
> Full folder table, left for reference:
|
||||||
|
|
||||||
## Folder Structure
|
## Folder Structure
|
||||||
| Folder | Purpose |
|
| Folder | Purpose |
|
||||||
|---|---|
|
|---|---|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: OpenCode Go
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-29
|
created: 2026-07-29
|
||||||
updated: 2026-07-29
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-29
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [opencode, provider, models, subscription, limits]
|
tags: [opencode, provider, models, subscription, limits]
|
||||||
sources: [https://opencode.ai/docs/go/]
|
sources: [https://opencode.ai/docs/go/]
|
||||||
@@ -22,6 +22,12 @@ Low cost subscription for open coding models — $5 first month, then **$10/mont
|
|||||||
|
|
||||||
> Only one member per workspace can subscribe.
|
> Only one member per workspace can subscribe.
|
||||||
|
|
||||||
|
> **Note (2026-10-20):** the pricing and dollar-limit tables below are **vendor-published figures
|
||||||
|
> for the OpenCode Go product**, not a measurement of this estate. Nothing in `~/.hermes/config.yaml`
|
||||||
|
> or `jobs.json` shows an OpenCode Go provider in use — the estate routes through LiteLLM,
|
||||||
|
> OmniRoute (`omni-101`) and `Omniroute:20128`. Treat this page as reference material for the
|
||||||
|
> subscription, not as a description of a configured provider.
|
||||||
|
|
||||||
## Usage Limits
|
## Usage Limits
|
||||||
|
|
||||||
| Window | Dollar Limit |
|
| Window | Dollar Limit |
|
||||||
|
|||||||
+10
-2
@@ -3,8 +3,8 @@ title: OpenRouter
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [system, openrouter, provider]
|
tags: [system, openrouter, provider]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -21,6 +21,14 @@ Primary key, fallback models, rate limits.
|
|||||||
OpenRouter is not configured as a standalone provider in `~/.hermes/config.yaml` `custom_providers`.
|
OpenRouter is not configured as a standalone provider in `~/.hermes/config.yaml` `custom_providers`.
|
||||||
OmniRoute’s catalog includes provider names and model IDs consistent with OpenRouter-style slugs, but live routing via OpenRouter directly is not present in current Hermes config.
|
OmniRoute’s catalog includes provider names and model IDs consistent with OpenRouter-style slugs, but live routing via OpenRouter directly is not present in current Hermes config.
|
||||||
|
|
||||||
|
> **Re-verified 2026-10-20 — the claim above still holds.** `custom_providers` in
|
||||||
|
> `/home/hermes/.hermes/config.yaml` is a list of exactly five entries:
|
||||||
|
> `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101`. There is **no `openrouter`
|
||||||
|
> entry**. OpenRouter-style slugs (`openrouter/auto`, `freeapi/openrouterfree`,
|
||||||
|
> `kilo-openrouter-auto`, `or-openrouter-fusion`, and many more) exist only inside the **OmniRoute
|
||||||
|
> model catalog**, not as directly-routable Hermes providers. Routing to OpenRouter therefore still
|
||||||
|
> goes through OmniRoute, exactly as recorded here.
|
||||||
|
|
||||||
## History
|
## History
|
||||||
- **User-provided:** Previously used; now largely replaced by LiteLLM and OmniRoute paths
|
- **User-provided:** Previously used; now largely replaced by LiteLLM and OmniRoute paths
|
||||||
- **Verified:** Decision page `keep-openrouter-as-fallback` exists, indicating retain intent
|
- **Verified:** Decision page `keep-openrouter-as-fallback` exists, indicating retain intent
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-19
|
updated: 2026-10-20
|
||||||
verified_on: 2026-10-19
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [skills, hermes]
|
tags: [skills, hermes]
|
||||||
sources: []
|
sources: []
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Terminal Backend
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-20
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [terminal, shell, backend, persistence]
|
tags: [terminal, shell, backend, persistence]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -17,12 +17,17 @@ Execute shell commands, scripts, and interactive sessions for Hermes.
|
|||||||
|
|
||||||
## Current Configuration
|
## Current Configuration
|
||||||
- **Backend:** local
|
- **Backend:** local
|
||||||
- **Working directory:** `/home/hermes/workspace`
|
- **Working directory:** `/home/hermes/workspace/`
|
||||||
- **Timeout:** 180 s
|
- **Timeout:** 180 s
|
||||||
- **Home mode:** auto
|
- **Home mode:** auto
|
||||||
- **Container CPU:** 1
|
- **Container CPU:** 1
|
||||||
- **Persistent shell:** true
|
- **Persistent shell:** true
|
||||||
|
|
||||||
|
Live `terminal:` block re-read from `/home/hermes/.hermes/config.yaml` on 2026-10-20 also carries:
|
||||||
|
`lifetime_seconds: 300`, `container_memory: 5120`, `container_disk: 51200`,
|
||||||
|
`container_persistent: true`, `docker_mount_cwd_to_workspace: false`, `docker_run_as_host_user: true`.
|
||||||
|
`backend: local` and `persistent_shell: true` are unchanged since 2026-07-22.
|
||||||
|
|
||||||
## Persistence
|
## Persistence
|
||||||
- Shell state (venv, env vars, aliases) persists across terminal calls
|
- Shell state (venv, env vars, aliases) persists across terminal calls
|
||||||
- Working directory fixed to `/home/hermes/workspace`
|
- Working directory fixed to `/home/hermes/workspace`
|
||||||
|
|||||||
+14
-6
@@ -3,9 +3,9 @@ title: Hermes Tool Search
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-25
|
created: 2026-07-25
|
||||||
updated: 2026-07-25
|
updated: 2026-10-20
|
||||||
review_after: 2026-10-23
|
review_after: 2026-11-20
|
||||||
verified_on: 2026-07-25
|
verified_on: 2026-10-20
|
||||||
version_applies_to: Hermes Agent on CT460
|
version_applies_to: Hermes Agent on CT460
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, automation, system]
|
tags: [hermes, automation, system]
|
||||||
@@ -32,11 +32,19 @@ File: `/home/hermes/.hermes/config.yaml`
|
|||||||
tools:
|
tools:
|
||||||
tool_search:
|
tool_search:
|
||||||
enabled: "on"
|
enabled: "on"
|
||||||
threshold_pct: 10
|
threshold_pct: 20
|
||||||
search_default_limit: 5
|
search_default_limit: 8
|
||||||
max_search_limit: 20
|
max_search_limit: 50
|
||||||
|
listing_max_tokens: 8
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> **Live config re-read 2026-10-20** — the values above are the current ones in
|
||||||
|
> `/home/hermes/.hermes/config.yaml`. They have drifted from the original 2026-07-25 rollout:
|
||||||
|
> `threshold_pct` 10 → **20**, `search_default_limit` 5 → **8**, `max_search_limit` 20 → **50**,
|
||||||
|
> and `listing_max_tokens: 8` has been added. `enabled: "on"` is unchanged.
|
||||||
|
> The token-budget figures in "Verified result" below were measured under the *original* settings
|
||||||
|
> and have not been re-measured since the tuning.
|
||||||
|
|
||||||
`enabled: "on"` is the important setting: it makes Tool Search active on every
|
`enabled: "on"` is the important setting: it makes Tool Search active on every
|
||||||
request instead of waiting for the tool schemas to cross an automatic token
|
request instead of waiting for the tool schemas to cross an automatic token
|
||||||
threshold.
|
threshold.
|
||||||
|
|||||||
+57
-13
@@ -142,21 +142,65 @@ python3 "$STOCKTAKE_DIR/_freshness_audit.py" "$WIKI_DIR" "$REPORT" 21
|
|||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# [3/3] Rebuild Quartz static site
|
# [3/3] Rebuild Quartz static site and deploy to public/
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
echo "[3/3] Checking Quartz build..."
|
# Quartz v4/v5 lives in a SEPARATE source tree (/home/hermes/workspace/quartz-temp);
|
||||||
# Quartz is already built into public/ — if there's a build script, run it
|
# the wiki repo only holds the Markdown. The build reads content/ from that tree
|
||||||
# For now, the public/ dir is the Quartz output, regenerated when content changes
|
# and emits public/ there, which we then deploy into the wiki repo's public/.
|
||||||
# If Quartz CLI is available, we'd run: npx quartz build
|
#
|
||||||
if command -v npx &> /dev/null && [ -f "$WIKI_DIR/quartz.config.ts" ]; then
|
# PITFALL (hit 2026-10-20): the previous version of this step tested for
|
||||||
echo " Running Quartz build..."
|
# `quartz.config.ts` and `npx`, and the repo has NEITHER — Quartz v4/v5 uses
|
||||||
cd "$WIKI_DIR"
|
# `quartz.config.yaml`. The test always failed, so the site silently stopped
|
||||||
npx quartz build 2>&1 | head -20 || echo " Quartz build had issues (non-fatal)"
|
# being rebuilt (published site was 40 days stale). Do not reintroduce a
|
||||||
else
|
# config-file existence test that isn't the real one: check quartz.config.yaml.
|
||||||
echo " Quartz build skipped (no quartz.config.ts or npx not available)"
|
echo "[3/3] Rebuilding Quartz static site..."
|
||||||
echo " public/ directory contains the current static site"
|
|
||||||
|
QUARTZ_DIR="/home/hermes/workspace/quartz-temp"
|
||||||
|
QUARTZ_CONTENT="$QUARTZ_DIR/content"
|
||||||
|
QUARTZ_PUBLIC="$QUARTZ_DIR/public"
|
||||||
|
QUARTZ_DEPLOY_DIR="$WIKI_DIR/public"
|
||||||
|
|
||||||
|
if [ ! -f "$QUARTZ_DIR/quartz.config.yaml" ]; then
|
||||||
|
echo " ERROR: $QUARTZ_DIR/quartz.config.yaml not found — cannot build the Quartz site." >&2
|
||||||
|
echo " The published site will stay stale. Fix the Quartz source tree before continuing." >&2
|
||||||
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# --- sync wiki Markdown into the Quartz content tree -------------------------
|
||||||
|
# raw/ is excluded on purpose: it holds raw config dumps and the stocktake
|
||||||
|
# reports, which have never been part of the published site. public/ and .git
|
||||||
|
# are excluded to avoid copying build output into the input tree.
|
||||||
|
rm -rf "$QUARTZ_CONTENT"
|
||||||
|
mkdir -p "$QUARTZ_CONTENT"
|
||||||
|
tar -C "$WIKI_DIR" \
|
||||||
|
--exclude=./public --exclude=./.git --exclude=./node_modules --exclude=./raw \
|
||||||
|
-cf - . | tar -C "$QUARTZ_CONTENT" -xf -
|
||||||
|
echo " Synced wiki content into $QUARTZ_CONTENT"
|
||||||
|
|
||||||
|
# --- build -------------------------------------------------------------------
|
||||||
|
cd "$QUARTZ_DIR"
|
||||||
|
# PIPESTATUS is required: a plain pipeline would report tail's exit code (0)
|
||||||
|
# and silently swallow a failed build.
|
||||||
|
npx quartz build 2>&1 | tail -8
|
||||||
|
BUILD_RC=${PIPESTATUS[0]}
|
||||||
|
if [ "$BUILD_RC" -ne 0 ]; then
|
||||||
|
echo " ERROR: Quartz build failed (exit $BUILD_RC)." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- deploy ------------------------------------------------------------------
|
||||||
|
# Clear the CONTENTS of public/ rather than removing the directory itself:
|
||||||
|
# hermes-wiki-serve.service runs with its cwd inside public/, so replacing the
|
||||||
|
# directory inode leaves the running server serving a deleted directory (all
|
||||||
|
# requests 404) until it is restarted.
|
||||||
|
find "$QUARTZ_DEPLOY_DIR" -mindepth 1 -delete
|
||||||
|
cp -a "$QUARTZ_PUBLIC/." "$QUARTZ_DEPLOY_DIR/"
|
||||||
|
echo " Deployed $(find "$QUARTZ_DEPLOY_DIR" -type f | wc -l) files to $QUARTZ_DEPLOY_DIR"
|
||||||
|
|
||||||
|
systemctl --user restart hermes-wiki-serve.service 2>/dev/null \
|
||||||
|
&& echo " Restarted hermes-wiki-serve.service" \
|
||||||
|
|| echo " WARNING: could not restart hermes-wiki-serve.service — verify the site manually"
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
echo "[wiki-maintain] Maintenance complete at $(date -Iseconds)"
|
echo "[wiki-maintain] Maintenance complete at $(date -Iseconds)"
|
||||||
echo " Freshness report: $REPORT"
|
echo " Freshness report: $REPORT"
|
||||||
@@ -164,4 +208,4 @@ echo ""
|
|||||||
|
|
||||||
# Print summary
|
# Print summary
|
||||||
head -5 "$REPORT"
|
head -5 "$REPORT"
|
||||||
tail -3 "$REPORT"
|
tail -3 "$REPORT"
|
||||||
Reference in New Issue
Block a user