Files
agent-estate-wiki/runbooks/update-hermes-safely.md
T
Tony0410 4126093644 stocktake(2026-10-26): correct 8 stale pages against live state — 0 stale remaining
wiki-maintain.sh ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted,
service restarted). The freshness report listed 8 stale pages; each was diffed
against live state and corrected only where reality could be verified. A second
script run rewrote the same-day report with 0 stale / 103 fresh / 4 no-date,
so the committed report matches the tree it ships with.

- infrastructure/hosts.md: CT600 is LIVE (not retired); CT601 does not exist on
  either hypervisor; CT333 verified running; both hypervisors verified; config
  33->51; skills 66->302; cron 25->28 (25 enabled / 3 disabled); Xvfb :102->:99;
  providers split into Omniroute:20128 (CT600) and omni-101 (CT101, default).
- agents/hermes-production.md: Tailscale IP 100.96.100.82 -> 100.118.5.51 (the
  old value is the PVE hypervisor); v0.21.5 -> v0.22.3; dead 8082 claim removed;
  fallback chain rewritten from live config (first entry is the dead nous
  credential); all 8 auxiliary slots now no-think/freeapi/auto; cron 30->28.
- agents/hermes-chaos.md: v0.22.3 (same build as CT460); "3 commits behind" ->
  0 behind origin/master; default model smart-route on custom:omni-101; cron
  "11 paused" -> 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460.
- infrastructure/tailscale.md: node table rebuilt from status --json; octoagent
  and omarchymac-1 were missing entirely (52 -> 54 non-funnel nodes); m93p now
  online; every offline age refreshed; omni row corrected to "CT601 removed
  from both hypervisors"; Serve tables re-probed per host.
- systems/memory-backend.md: re-verified against live config; coverage 1899/2350
  (80.8%), working 96.8%, episodic 99.4% — above the 95% reindex threshold.
- runbooks/provider-health.md: the step-3 OmniRoute snippet raised IndexError
  (provider renamed omniroute -> Omniroute:20128; key moved to key_env) — fixed;
  new step 3b documents the omni-101 /v1/models false-failure quirk.
- runbooks/update-hermes-safely.md: added managed-runtime dependency re-assert
  step; clarified --branch vs stable channel (v0.22.5 available).
- incidents/2026-10-03-ct450-hermes-update.md + incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md:
  Current-state sections recording current versions and embedding coverage.

All 9 corrected pages verified 200 at http://127.0.0.1:9120/<path>.
Vault mirror at /home/hermes/wiki synced for all touched files (22 copies,
sha256-verified) and committed locally; the vault is local-only and is not pushed.
No secrets written — IDs, URLs, key env var names, counts and timestamps only.
2026-10-26 04:27:49 +08:00

5.0 KiB

title, type, status, created, updated, verified_on, last_tested, confidence, tags, sources
title type status created updated verified_on last_tested confidence tags sources
Update Hermes safely runbook active 2026-07-22 2026-10-26 2026-10-26 2026-10-26 high
runbook
hermes
update
raw/configs/hermes-config-sanitized.txt

Update Hermes safely

Purpose

Update Hermes to a newer release without losing configuration, skills, or sessions.

Symptoms that match this runbook

  • User wants to move to a newer Hermes version
  • hermes version shows an older build than expected
  • Update-related errors after manual changes in the install directory

Prerequisites

  • Hermes install method: git-based install at /home/hermes/.hermes/hermes-agent
  • Working Python virtualenv at /home/hermes/.hermes/hermes-agent/venv
  • Managed runtimes live under ~/.hermes/installs/<install>/environments/<hash>/venv — these are what the gateway actually imports from; a missing dep there breaks cron jobs with no obvious trace (see incidents/2026-09-28-hermes-managed-runtime-deps)
  • Available disk space in Hermes install and backup paths
  • No active Hermes chat/webui shell issues before starting

Procedure

1. Create a backup first

hermes backup --quick --label pre-update-$(date +%Y%m%d)

If you need a full backup instead:

hermes backup --label pre-update-$(date +%Y%m%d)

2. Check whether an update is available

hermes update --check

3. Review configuration version and backups

python3 -c "import yaml; print(yaml.safe_load(open('/home/hermes/.hermes/config.yaml')).get('_config_version'))"
ls -la ~/.hermes/config.yaml.bak*

4. Run the update with noninteractive flags

cd /home/hermes/.hermes/hermes-agent
git status --short
hermes update --yes

Use --branch main only when deliberately tracking a branch other than the configured channel. The stable channel is currently the default and offers v0.22.5 (hermes update --check, verified 2026-10-26; local install is v0.22.3 / 5b147ab036d6). --force-venv rebuilds the virtualenv — avoid unless the venv itself is broken.

After any update, re-assert the managed-runtime dependencies (the updater can leave a runtime without fastembed/dotenv/mnemosyne):

~/.hermes/scripts/hermes-runtime-deps-guard.sh

5. Verify the new version

hermes version

6. Restart the required Hermes pieces

hermes gateway status
hermes dashboard status
hermes gateway restart
hermes dashboard restart

7. Sanity check health

hermes gateway status
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8787/health

Verification

  • hermes version shows a newer version than before
  • hermes gateway status is active
  • WebUI /health returns 200
  • Providers reachable with provider-health

Rollback

  • Rollback Hermes config:
ls -lt ~/.hermes/config.yaml.bak*
cp ~/.hermes/config.yaml.bak.<timestamp> ~/.hermes/config.yaml
hermes gateway restart
hermes dashboard restart
  • Rollback entire Hermes install requires git revert or reinstall from previous checkout
  • If update warns about config migration, revert the backup and re-run manually

Notes

  • Hermes version on this install: v0.22.3 (2026.9.24), local 5b147ab036d6; install tree rebuilt 2026-10-21 22:47→22:57 AWST. Stable v0.22.5 available as of 2026-10-26
  • Install directory: /home/hermes/.hermes/hermes-agent
  • Update preserves venv by default unless --force-venv is used

Second instance — CT450 (hermesagent), updated 2026-10-03 (v0.19.0 → v0.21.5)

CT450 has no SSH. Reach it from the PVE host:

ssh root@100.96.100.82
pct exec 450 -- <command>            # or: pct enter 450
pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"

pct exec 450 -- hermes ... fails (Failed to exec "hermes"): the binary is not on lxc-attach's PATH. Use the absolute path or su - hermes -c (the dash gives a login shell, which sets ~/.local/bin). A hermes login shell also prints a hermes-setup banner before output.

Pre-flight before running the update (each of these cost one failed run on 2026-10-03):

  1. clang installed — source-compiled Python dependencies will not build without it (pct exec 450 -- bash -lc "which clang").
  2. No stale .install.lock / .lazy-refresh-incomplete in the Hermes home.
  3. ~/.hermes/hermes-agent owned by hermes:hermes (a root-owned tree from an earlier root-run update breaks the in-place build).
  4. Host Caddy/keyring file permissions sane (0600 keyring blocked a step mid-update).

After: pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version", confirm CT460 production is untouched (separate config/skills/cron/memory). Full record: incidents/2026-10-03-ct450-hermes-update.

Last tested

2026-10-26 on debian 13, git-based Hermes install — hermes update --check (stable v0.22.5 available), hermes backup --help and hermes update --help flags confirmed