Files
agent-estate-wiki/incidents/2026-10-03-ct450-hermes-update.md
T

3.1 KiB

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
CT450 Hermes Update to v0.21.5 incident resolved 2026-10-04 2026-10-04 2026-10-04 high
hermes
ct450
chaos
update
incident
live-probe
pve-100.96.100.82
session-2026-10-03

CT450 Hermes Update to v0.21.5

Summary

The Chaos instance (LXC CT450, hermesagent) was updated v0.19.0 → v0.21.5 on 2026-10-03 — its first update since July. CT460 (production) was rebuilt the same morning, so both instances now carry the same semver token from separate builds.

Result (live-verified 2026-10-04)

CT460 (production) CT450 (chaos)
Version v0.21.5+6169.gab86a20 (2026.9.24) v0.21.5+6170.ga510d6c (2026.9.24)
Upstream ab86a20a a510d6c6 — 3 commits behind, hermes update available
Python managed runtime 3.14 (legacy venv/ 3.11.15) 3.14.7
Install tree mtime 2026-10-03 10:11 AWST 2026-10-03 10:22 AWST

Access

CT450 has no SSH. Reach it through the PVE host:

ssh root@100.96.100.82
pct exec 450 -- <command>            # or: pct enter 450
pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"

Two gotchas that cost time:

  • pct exec 450 -- hermes ... fails with Failed to exec "hermes" — the binary is not on lxc-attach's default PATH. Use the absolute path, or su - hermes -c (the dash matters: only a login shell puts ~/.local/bin on PATH).
  • A hermes login shell prints a hermes-setup banner ahead of command output; parse with | tail -n rather than expecting clean stdout.

Pre-flight checks (each cost one failed run on 2026-10-03)

  1. Build toolchain. clang must be present before updating — source-compiled Python dependencies will not build without it. Verified present on CT450 (/usr/bin/clang, 2026-10-04).
  2. Stale update locks. A leftover .install.lock / .lazy-refresh-incomplete in the Hermes home makes the updater refuse or silently skip the refresh step. Neither exists on CT450 now.
  3. Install-tree ownership. ~/.hermes/hermes-agent must stay owned by hermes:hermes (uid 1000); a root-owned tree left by an earlier root-run update fails the in-place build.
  4. Caddy keyring permissions. Host-level Caddy/keyring file permissions (0600) blocked a step mid-update — check them before starting, not during.

Verification

  • hermes --version on CT450 → Hermes Agent v0.21.5+6170.ga510d6c (2026.9.24) · upstream a510d6c6, Python 3.14.7.
  • Install tree mtime 2026-10-03 10:22 matches the update window.
  • clang present; no stale locks present.
  • CT460 production untouched — separate config, skills, cron and memory state.

Follow-ups

  • CT450 reports Update available: 3 commits behind. Harmless; re-run hermes update if the chaos box should track upstream exactly.
  • CT450's Mattermost connection is failing (mattermost connect timed out after 30s, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See agents/hermes-chaos.