wiki-maintain.sh ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted, service restarted). The freshness report listed 8 stale pages; each was diffed against live state and corrected only where reality could be verified. A second script run rewrote the same-day report with 0 stale / 103 fresh / 4 no-date, so the committed report matches the tree it ships with. - infrastructure/hosts.md: CT600 is LIVE (not retired); CT601 does not exist on either hypervisor; CT333 verified running; both hypervisors verified; config 33->51; skills 66->302; cron 25->28 (25 enabled / 3 disabled); Xvfb :102->:99; providers split into Omniroute:20128 (CT600) and omni-101 (CT101, default). - agents/hermes-production.md: Tailscale IP 100.96.100.82 -> 100.118.5.51 (the old value is the PVE hypervisor); v0.21.5 -> v0.22.3; dead 8082 claim removed; fallback chain rewritten from live config (first entry is the dead nous credential); all 8 auxiliary slots now no-think/freeapi/auto; cron 30->28. - agents/hermes-chaos.md: v0.22.3 (same build as CT460); "3 commits behind" -> 0 behind origin/master; default model smart-route on custom:omni-101; cron "11 paused" -> 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460. - infrastructure/tailscale.md: node table rebuilt from status --json; octoagent and omarchymac-1 were missing entirely (52 -> 54 non-funnel nodes); m93p now online; every offline age refreshed; omni row corrected to "CT601 removed from both hypervisors"; Serve tables re-probed per host. - systems/memory-backend.md: re-verified against live config; coverage 1899/2350 (80.8%), working 96.8%, episodic 99.4% — above the 95% reindex threshold. - runbooks/provider-health.md: the step-3 OmniRoute snippet raised IndexError (provider renamed omniroute -> Omniroute:20128; key moved to key_env) — fixed; new step 3b documents the omni-101 /v1/models false-failure quirk. - runbooks/update-hermes-safely.md: added managed-runtime dependency re-assert step; clarified --branch vs stable channel (v0.22.5 available). - incidents/2026-10-03-ct450-hermes-update.md + incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md: Current-state sections recording current versions and embedding coverage. All 9 corrected pages verified 200 at http://127.0.0.1:9120/<path>. Vault mirror at /home/hermes/wiki synced for all touched files (22 copies, sha256-verified) and committed locally; the vault is local-only and is not pushed. No secrets written — IDs, URLs, key env var names, counts and timestamps only.
82 lines
4.8 KiB
Markdown
82 lines
4.8 KiB
Markdown
---
|
|
title: Hermes Chaos
|
|
type: system
|
|
status: active
|
|
created: 2026-08-15
|
|
updated: 2026-10-26
|
|
verified_on: 2026-10-26
|
|
confidence: high
|
|
tags: [hermes, chaos, ct450, experimental, agent]
|
|
sources: [live-probe, config-inspection]
|
|
---
|
|
|
|
# Hermes Chaos
|
|
|
|
## Purpose
|
|
|
|
Experimental Hermes instance for testing new configurations, skills, themes, model routing, and plugin changes before promoting them to production. Changes here must not affect CT460 production without explicit approval.
|
|
|
|
## Location
|
|
|
|
- **Container:** LXC CT450 (`hermesagent`)
|
|
- **Access:** **no SSH** — go through the PVE host: `ssh root@100.96.100.82`, then `pct exec 450 -- <cmd>` (or `pct enter 450`). `hermes` is not on `pct exec`'s default PATH; use `su - hermes -c "/home/hermes/.local/bin/hermes …"`.
|
|
- **Tailscale IP:** `100.98.205.112`
|
|
- **Hostname:** `hermesagent`
|
|
- **Software:** hermes-ui (Vite web app for browser use, port 5174)
|
|
- **Memory:** CPU-only container (no GPU) — self-hosted VRAM models are off the table
|
|
|
|
## Current State
|
|
|
|
- **Status:** Active (experimental)
|
|
- **Hermes version:** `v0.22.3` (2026.9.24), local `5b147ab036d6` — **the same build as CT460** (updated 2026-10-21). **0 commits behind `origin/main`** (verified 2026-10-26; the old "3 commits behind" claim is stale), though `hermes update --check` reports stable **v0.22.5** available. Python 3.14.7. See [[incidents/2026-10-03-ct450-hermes-update]].
|
|
- **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]].
|
|
- **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01)
|
|
- **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` 2026-10-26). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460)
|
|
- **Production counterparts:** CT460 has **28 jobs / 25 enabled** — the two fleets no longer share a job count, and the CT450 duplicates are paused
|
|
|
|
## Providers & Routing
|
|
|
|
- **Default model:** **`smart-route`** via `custom` → `http://100.78.63.20:20128` (the `omni-101` node) — verified live in CT450's `config.yaml` 2026-10-26. ⚠ The old claim of `deepseek-flash-mix` on `custom:omniroute` (`100.88.81.19`) no longer matches the config
|
|
- **Fallback chain** (fixed 2026-08-15 — OmniRoute fallback entries removed, they were non-functional):
|
|
1. `kilo-stepfun-step-3_7-flash-free` via `litellm_4000`
|
|
2. `nvidia/nemotron-3-ultra:free` via `nous`
|
|
3. `auto` via `custom` at `https://freellmapi.com/v1`
|
|
- **Important:** `fallback_providers` must stay a YAML list — `hermes config set` writes strings, which silently disables the whole chain (`_iter_fallback_entries` returns `[]` for strings). Edit via the `hermes fallback` command or direct YAML edit.
|
|
- See [[systems/model-providers]] for full breakdown.
|
|
|
|
## Browser & Desktop Access
|
|
|
|
- **Browser engine:** `camofox` on port 9377 (config: `engine: camofox`). Does **not** expose raw CDP on 9222 — do not point `cdp_url` at 9222 or browser tools fail with connection refused.
|
|
- **VNC desktop:** `camofox-vnc.service` (user unit) on Xvfb `:127`, RFB 5901, localhost-only no-auth. View at `hermesagent.kangaroo-eel.ts.net/vnc.html`.
|
|
- **CDP endpoint for typed-browser ops:** bind via `cua_browser_prepare`/`cua_browser_state` on the Camofox window; legacy `browser_navigate` works for plain navigation.
|
|
- See [[systems/browser-backend]] for browser architecture.
|
|
|
|
## Web UIs (do not conflate)
|
|
|
|
- **WebUI:** port 8787 (Python/vanilla JS)
|
|
- **hermes-ui:** port 5174 (Vite/React desktop-style renderer, proxies to gateway 9119)
|
|
- These are separate projects. Deployment of hermes-ui is done via CX-dump + tar through the PVE host.
|
|
|
|
## Phase One Review (2026-08-08)
|
|
|
|
Phase One seven-day review completed and independently live-verified:
|
|
- CT450 duplicate jobs remained paused with no post-pause execution records
|
|
- CT460 counterparts remain enabled with successful latest runs
|
|
- CT333 schedules remained enabled and healthy
|
|
- **Phase Two is ready for consideration but is NOT approved**
|
|
|
|
## Key Rules
|
|
|
|
- **Do not promote changes to production without explicit approval.** Chaos is for testing.
|
|
- **Duplicate cron jobs are paused.** Do not re-enable without approval.
|
|
- **Separate state.** Memory, skills, and config are independent from CT460.
|
|
- **Wiki governance:** write only to this section (`agents/hermes-chaos.md`); never hand-edit auto-generated pages like [[systems/skills-index]].
|
|
|
|
## Relationship to Production
|
|
|
|
Hermes Chaos (CT450) and Hermes Production (CT460) are separate Hermes instances with:
|
|
- Separate configs
|
|
- Separate skills
|
|
- Separate cron jobs
|
|
- Separate memory state (Chaos runs Hindsight, bank `hermes-clean`; production runs Mnemosyne)
|
|
- Separate Tailscale |