Files
agent-estate-wiki/incidents/2026-10-03-ct450-hermes-update.md
T
Tony0410 3f796dc3b9 stocktake(2026-10-26): strike superseded follow-ups per SCHEMA rule 9
Mark-don't-delete: the CT450 '3 commits behind' follow-up and the two
mnemosyne-vector-recall follow-ups are struck through with their verified
2026-10-26 resolution rather than removed.

- CT450 tracks origin/main (there is no origin/master on that remote) and is
  0 commits behind.
- Coverage monitor b63ed5817212 is enabled, last_status ok, working coverage
  96.8% — correctly has nothing to fire on.
- fastembed 0.8.1 now imports in both managed runtimes plus the legacy venv.
2026-10-26 04:35:33 +08:00

3.9 KiB

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
CT450 Hermes Update to v0.21.5 incident resolved 2026-10-04 2026-10-26 2026-10-26 high
hermes
ct450
chaos
update
incident
live-probe
pve-100.96.100.82
session-2026-10-03

CT450 Hermes Update to v0.21.5

Summary

The Chaos instance (LXC CT450, hermesagent) was updated v0.19.0 → v0.21.5 on 2026-10-03 — its first update since July. CT460 (production) was rebuilt the same morning, so both instances now carry the same semver token from separate builds.

Result (live-verified 2026-10-04)

CT460 (production) CT450 (chaos)
Version v0.21.5+6169.gab86a20 (2026.9.24) v0.21.5+6170.ga510d6c (2026.9.24)
Upstream ab86a20a a510d6c6 — 3 commits behind, hermes update available
Python managed runtime 3.14 (legacy venv/ 3.11.15) 3.14.7
Install tree mtime 2026-10-03 10:11 AWST 2026-10-03 10:22 AWST

Access

CT450 has no SSH. Reach it through the PVE host:

ssh root@100.96.100.82
pct exec 450 -- <command>            # or: pct enter 450
pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"

Two gotchas that cost time:

  • pct exec 450 -- hermes ... fails with Failed to exec "hermes" — the binary is not on lxc-attach's default PATH. Use the absolute path, or su - hermes -c (the dash matters: only a login shell puts ~/.local/bin on PATH).
  • A hermes login shell prints a hermes-setup banner ahead of command output; parse with | tail -n rather than expecting clean stdout.

Pre-flight checks (each cost one failed run on 2026-10-03)

  1. Build toolchain. clang must be present before updating — source-compiled Python dependencies will not build without it. Verified present on CT450 (/usr/bin/clang, 2026-10-04).
  2. Stale update locks. A leftover .install.lock / .lazy-refresh-incomplete in the Hermes home makes the updater refuse or silently skip the refresh step. Neither exists on CT450 now.
  3. Install-tree ownership. ~/.hermes/hermes-agent must stay owned by hermes:hermes (uid 1000); a root-owned tree left by an earlier root-run update fails the in-place build.
  4. Caddy keyring permissions. Host-level Caddy/keyring file permissions (0600) blocked a step mid-update — check them before starting, not during.

Verification

  • hermes --version on CT450 → Hermes Agent v0.21.5+6170.ga510d6c (2026.9.24) · upstream a510d6c6, Python 3.14.7.
  • Install tree mtime 2026-10-03 10:22 matches the update window.
  • clang present; no stale locks present.
  • CT460 production untouched — separate config, skills, cron and memory state.

Current state (re-verified 2026-10-26)

Both instances have since moved on: CT460 and CT450 now both run v0.22.3 (2026.9.24), local 5b147ab036d6 — the same build. CT450 is 0 commits behind origin/main (the "3 commits behind" note below is superseded), though hermes update --check reports stable v0.22.5 available on both. CT450's default model is now smart-route on custom → http://100.78.63.20:20128. The CT450 Mattermost timeout remains — CT255 is stopped on pve and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.

Follow-ups

  • CT450 reports Update available: 3 commits behind. SUPERSEDED 2026-10-26 — CT450 is now 0 commits behind origin/main (it tracks the NousResearch/hermes-agent main branch; there is no origin/master on that remote). Re-run hermes update if the chaos box should track upstream exactly.
  • CT450's Mattermost connection is failing (mattermost connect timed out after 30s, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See agents/hermes-chaos.