--- title: CT450 Hermes Update to v0.21.5 type: incident status: resolved created: 2026-10-04 updated: 2026-11-18 verified_on: 2026-11-18 confidence: high tags: [hermes, ct450, chaos, update, incident] sources: [live-probe, pve-100.96.100.82, session-2026-10-03] --- # CT450 Hermes Update to v0.21.5 ## Summary The Chaos instance (LXC CT450, `hermesagent`) was updated **v0.19.0 → v0.21.5** on 2026-10-03 — its first update since July. CT460 (production) was rebuilt the same morning, so both instances now carry the same semver token from separate builds. ## Result (live-verified 2026-10-04) | | CT460 (production) | CT450 (chaos) | |---|---|---| | Version | `v0.21.5+6169.gab86a20` (2026.9.24) | `v0.21.5+6170.ga510d6c` (2026.9.24) | | Upstream | `ab86a20a` | `a510d6c6` — 3 commits behind, `hermes update` available | | Python | managed runtime 3.14 (legacy `venv/` 3.11.15) | 3.14.7 | | Install tree mtime | 2026-10-03 10:11 AWST | 2026-10-03 10:22 AWST | ## Access **CT450 has no SSH.** Reach it through the PVE host: ```bash ssh root@100.96.100.82 pct exec 450 -- # or: pct enter 450 pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version" ``` Two gotchas that cost time: - `pct exec 450 -- hermes ...` **fails** with `Failed to exec "hermes"` — the binary is not on `lxc-attach`'s default PATH. Use the absolute path, or `su - hermes -c` (the dash matters: only a login shell puts `~/.local/bin` on PATH). - A `hermes` login shell prints a `hermes-setup` banner ahead of command output; parse with `| tail -n` rather than expecting clean stdout. ## Pre-flight checks (each cost one failed run on 2026-10-03) 1. **Build toolchain.** `clang` must be present *before* updating — source-compiled Python dependencies will not build without it. Verified present on CT450 (`/usr/bin/clang`, 2026-10-04). 2. **Stale update locks.** A leftover `.install.lock` / `.lazy-refresh-incomplete` in the Hermes home makes the updater refuse or silently skip the refresh step. Neither exists on CT450 now. 3. **Install-tree ownership.** `~/.hermes/hermes-agent` must stay owned by `hermes:hermes` (uid 1000); a root-owned tree left by an earlier root-run update fails the in-place build. 4. **Caddy keyring permissions.** Host-level Caddy/`keyring` file permissions (`0600`) blocked a step mid-update — check them before starting, not during. ## Verification - `hermes --version` on CT450 → `Hermes Agent v0.21.5+6170.ga510d6c (2026.9.24) · upstream a510d6c6`, Python 3.14.7. - Install tree mtime `2026-10-03 10:22` matches the update window. - `clang` present; no stale locks present. - CT460 production untouched — separate config, skills, cron and memory state. ## Current state (re-verified 2026-10-26) Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. ~~The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.~~ **SUPERSEDED 2026-11-18** — CT255 is `running` and its peer online; see the 11-18 state above. ## Current state (re-verified 2026-11-18) Both instances have moved on again and are now on **different builds**: **CT460 runs `v0.23.0` (local `2bd378ed1b8d`)**, **CT450 runs `v0.23.1` (local `3b4215785882`)** — CT450 is one patch ahead and **0 commits behind `origin/main`**. The 10-26 Mattermost condition is **superseded**: **CT255 (`mattermost`) is `running` on `pve`** and its tailnet peer is `online`, so the old "CT255 stopped / peer offline 36 d" note no longer describes reality. ## Follow-ups - ~~CT450 reports `Update available: 3 commits behind`.~~ **SUPERSEDED 2026-10-26** — CT450 is now **0 commits behind `origin/main`** (it tracks the `NousResearch/hermes-agent` `main` branch; there is no `origin/master` on that remote). Re-run `hermes update` if the chaos box should track upstream exactly. - CT450's Mattermost connection is failing (`mattermost connect timed out after 30s`, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See [[agents/hermes-chaos]]. ## Related - [[agents/hermes-chaos]] - [[agents/hermes-production]] - [[runbooks/update-hermes-safely]] - [[infrastructure/hosts]]