Files
agent-estate-wiki/incidents/2026-10-03-ct450-hermes-update.md
T
Tony0410 4126093644 stocktake(2026-10-26): correct 8 stale pages against live state — 0 stale remaining
wiki-maintain.sh ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted,
service restarted). The freshness report listed 8 stale pages; each was diffed
against live state and corrected only where reality could be verified. A second
script run rewrote the same-day report with 0 stale / 103 fresh / 4 no-date,
so the committed report matches the tree it ships with.

- infrastructure/hosts.md: CT600 is LIVE (not retired); CT601 does not exist on
  either hypervisor; CT333 verified running; both hypervisors verified; config
  33->51; skills 66->302; cron 25->28 (25 enabled / 3 disabled); Xvfb :102->:99;
  providers split into Omniroute:20128 (CT600) and omni-101 (CT101, default).
- agents/hermes-production.md: Tailscale IP 100.96.100.82 -> 100.118.5.51 (the
  old value is the PVE hypervisor); v0.21.5 -> v0.22.3; dead 8082 claim removed;
  fallback chain rewritten from live config (first entry is the dead nous
  credential); all 8 auxiliary slots now no-think/freeapi/auto; cron 30->28.
- agents/hermes-chaos.md: v0.22.3 (same build as CT460); "3 commits behind" ->
  0 behind origin/master; default model smart-route on custom:omni-101; cron
  "11 paused" -> 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460.
- infrastructure/tailscale.md: node table rebuilt from status --json; octoagent
  and omarchymac-1 were missing entirely (52 -> 54 non-funnel nodes); m93p now
  online; every offline age refreshed; omni row corrected to "CT601 removed
  from both hypervisors"; Serve tables re-probed per host.
- systems/memory-backend.md: re-verified against live config; coverage 1899/2350
  (80.8%), working 96.8%, episodic 99.4% — above the 95% reindex threshold.
- runbooks/provider-health.md: the step-3 OmniRoute snippet raised IndexError
  (provider renamed omniroute -> Omniroute:20128; key moved to key_env) — fixed;
  new step 3b documents the omni-101 /v1/models false-failure quirk.
- runbooks/update-hermes-safely.md: added managed-runtime dependency re-assert
  step; clarified --branch vs stable channel (v0.22.5 available).
- incidents/2026-10-03-ct450-hermes-update.md + incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md:
  Current-state sections recording current versions and embedding coverage.

All 9 corrected pages verified 200 at http://127.0.0.1:9120/<path>.
Vault mirror at /home/hermes/wiki synced for all touched files (22 copies,
sha256-verified) and committed locally; the vault is local-only and is not pushed.
No secrets written — IDs, URLs, key env var names, counts and timestamps only.
2026-10-26 04:27:49 +08:00

3.7 KiB

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
CT450 Hermes Update to v0.21.5 incident resolved 2026-10-04 2026-10-26 2026-10-26 high
hermes
ct450
chaos
update
incident
live-probe
pve-100.96.100.82
session-2026-10-03

CT450 Hermes Update to v0.21.5

Summary

The Chaos instance (LXC CT450, hermesagent) was updated v0.19.0 → v0.21.5 on 2026-10-03 — its first update since July. CT460 (production) was rebuilt the same morning, so both instances now carry the same semver token from separate builds.

Result (live-verified 2026-10-04)

CT460 (production) CT450 (chaos)
Version v0.21.5+6169.gab86a20 (2026.9.24) v0.21.5+6170.ga510d6c (2026.9.24)
Upstream ab86a20a a510d6c6 — 3 commits behind, hermes update available
Python managed runtime 3.14 (legacy venv/ 3.11.15) 3.14.7
Install tree mtime 2026-10-03 10:11 AWST 2026-10-03 10:22 AWST

Access

CT450 has no SSH. Reach it through the PVE host:

ssh root@100.96.100.82
pct exec 450 -- <command>            # or: pct enter 450
pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"

Two gotchas that cost time:

  • pct exec 450 -- hermes ... fails with Failed to exec "hermes" — the binary is not on lxc-attach's default PATH. Use the absolute path, or su - hermes -c (the dash matters: only a login shell puts ~/.local/bin on PATH).
  • A hermes login shell prints a hermes-setup banner ahead of command output; parse with | tail -n rather than expecting clean stdout.

Pre-flight checks (each cost one failed run on 2026-10-03)

  1. Build toolchain. clang must be present before updating — source-compiled Python dependencies will not build without it. Verified present on CT450 (/usr/bin/clang, 2026-10-04).
  2. Stale update locks. A leftover .install.lock / .lazy-refresh-incomplete in the Hermes home makes the updater refuse or silently skip the refresh step. Neither exists on CT450 now.
  3. Install-tree ownership. ~/.hermes/hermes-agent must stay owned by hermes:hermes (uid 1000); a root-owned tree left by an earlier root-run update fails the in-place build.
  4. Caddy keyring permissions. Host-level Caddy/keyring file permissions (0600) blocked a step mid-update — check them before starting, not during.

Verification

  • hermes --version on CT450 → Hermes Agent v0.21.5+6170.ga510d6c (2026.9.24) · upstream a510d6c6, Python 3.14.7.
  • Install tree mtime 2026-10-03 10:22 matches the update window.
  • clang present; no stale locks present.
  • CT460 production untouched — separate config, skills, cron and memory state.

Current state (re-verified 2026-10-26)

Both instances have since moved on: CT460 and CT450 now both run v0.22.3 (2026.9.24), local 5b147ab036d6 — the same build. CT450 is 0 commits behind origin/main (the "3 commits behind" note below is superseded), though hermes update --check reports stable v0.22.5 available on both. CT450's default model is now smart-route on custom → http://100.78.63.20:20128. The CT450 Mattermost timeout remains — CT255 is stopped on pve and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.

Follow-ups

  • CT450 reports Update available: 3 commits behind. Harmless; re-run hermes update if the chaos box should track upstream exactly.
  • CT450's Mattermost connection is failing (mattermost connect timed out after 30s, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See agents/hermes-chaos.