Files
agent-estate-wiki/incidents/2026-09-28-hermes-managed-runtime-deps.md

9.1 KiB

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Hermes Managed-Runtime Dependency Failure incident resolved 2026-09-30 2026-11-04 2026-11-04 high
hermes
cron
runtime
python
worker
incident
live-jobs-json
live-guard-script
live-cron-list

Hermes Managed-Runtime Dependency Failure

Window: ~2026-09-28 → 2026-09-30 19:13 AWST Status: Resolved — dependency guard installed, watchdog green Impact: Cron jobs dispatched through the external worker; Mnemosyne Watchdog failed 176 consecutive runs

Symptoms

Affected jobs reported the same dispatch error, which points at the wrong layer:

Restart-safe cron worker dispatch failed ... cron external worker exited
before ownership acknowledgement (exit 1)
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in
sys.modules after import of package 'cron', but prior to execution of
'cron.scheduler'

The RuntimeWarning is noise. The real error is further down the worker stderr.

Root cause

Hermes migrated to uv-managed runtimes. Each runtime gets its own interpreter and venv:

~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv      # 10 venvs at time of incident
                                                          # (now **3** — see 2026-11-04 re-verification below)

The migration did not carry every dependency into every managed venv. Workers launched against a venv missing dotenv and mnemosyne died during import — before acknowledging ownership of the job. The failure therefore surfaced as a scheduler/dispatch bug while the real cause was:

  • ModuleNotFoundError: No module named 'dotenv' (459 occurrences in worker stderr)
  • ModuleNotFoundError: No module named 'mnemosyne'

Diagnosis

Read worker stderr, not just the job's error field:

grep -rl "ModuleNotFoundError" ~/.hermes/logs/          # find the worker log
hermes cron list                                        # which jobs, which streaks
ls -d ~/.hermes/installs/*/environments/*/venv          # how many runtimes exist

Test the live interpreter directly — never trust *.dist-info.

Fix

/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh (v2, installed 2026-09-30 19:13 AWST):

  • discovers every managed venv under ~/.hermes/installs/*/environments/*/venv (v1 checked only 2)
  • verifies import dotenv and import mnemosyne against each venv
  • repairs missing packages with pip install --target <venv site-packages>

Wiring: the Mnemosyne Watchdog job (96eb95ce12f2, */15 * * * *) runs the guard every 15 minutes, so a future managed-runtime migration self-heals within 15 minutes.

Pitfalls found while fixing

  • uv pip install --python <venv>/bin/python walks up to the ~/.hermes/hermes-agent workspace and installs there — the target venv stays empty. Use pip install --target.
  • Stale *.dist-info directories make uv skip a package as "already installed" even when the module is not importable.
  • write_file silently no-ops on .hermes/scripts/ paths (Hermes library bug). Stage the script in /tmp and install it into place.

Re-verification (2026-11-04)

The incident remains resolved, but the estate's runtime topology has changed since 2026-09-30 and the page's "10 venvs" figures are now stale. Live state on CT460:

Item 2026-09-30 (page) 2026-11-04 (live)
Managed venvs under ~/.hermes/installs/*/environments/*/venv 10 3 — 19da945341ad4794ac2bc8ff9fdd3643, d41b16c8b6674454a4a819bd4acced1f, f51d6e4902414114974145d8041c79cb
Total runtimes the guard checks 10 4 (3 managed venvs + the tools installer python)
Guard result 10/10 OK hermes-runtime-deps-guard: 4/4 runtimes OK (exit 0)
Bootstrap-selected runtime n/a f51d6e4902414114974145d8041c79cb — confirmed by running hermes_bootstrap under the tools python: sys.path → …/f51d6e4902414114974145d8041c79cb/venv/lib/python3.14/site-packages
fastembed in the bootstrap-selected venv repaired OK — guard log: fastembed OK in f51d6e4902414114974145d8041c79cb
Fastembed model cache n/a present — ~/.hermes/cache/fastembed/models--qdrant--bge-small-en-v1.5-onnx-q
Old gateway committed env 795020dbf24343cd9c52df24723dca83 the one that broke on 2026-10-10 no longer exists — the environments/ directory now holds only the three venvs above

The guard script itself is unchanged and still wired in: ~/.hermes/scripts/hermes-runtime-deps-guard.sh exists (executable, 6479 bytes) and is invoked by ~/.hermes/scripts/mnemosyne-watchdog.py at line 62.

Two notes for the next time this class recurs. First, the "10 venvs" figure is not a stable invariant — Hermes prunes and re-creates uv environments, so count the live venvs rather than trusting a recorded number; the guard's own 4/4 runtimes OK line is the authoritative reading. Second, the gateway's committed env is no longer a separate directory — the 795020dbf… env from the 2026-10-10 recurrence has been garbage-collected, and hermes_bootstrap now selects f51d6e49… (one of the three surviving managed venvs). The 2026-10-10 failure mode ("the gateway loads an env the guard does not check") may therefore no longer reproduce in that exact shape, but the underlying trap — a sys.path that points somewhere the guard never inspects — is unchanged, so the preventive rule below still stands.

Verification (2026-09-30)

Check Result
Managed venvs importing dotenv + mnemosyne 10/10
Mnemosyne Watchdog 96eb95ce12f2 last_status ok (19:15 AWST), streak 0
Jobs recovered 12 of 13
Residual cbfe434a3b5e (Wiki Audit) and 1a8e90ea2271 (wiki-stocktake) still carry the pre-fix error — both last ran before the 19:13 fix. Expected to clear on their next runs.

Preventive rules

  • Treat "worker exited before ownership acknowledgement" as a dependency/import failure first. Read worker stderr before touching provider or model pins.
  • After any Hermes runtime or Python upgrade, re-run the guard and confirm every runtime it reports imports dotenv and mnemosyne — read the guard's own N/N runtimes OK line rather than a recorded venv count, because the number of managed venvs changes as Hermes prunes and re-creates uv environments (10 on 2026-09-30, 3 on 2026-11-04).
  • Mass-cron-failure triage order: live interpreter imports → worker stderr → runtime layout → job logic. See runbooks/restart-hermes and systems/scheduled-tasks.

Recurrence - 2026-10-10 (this time the gateway runtime env, plus a new failure mode)

The class recurred twice in one day, on packages the 15-minute guard does not cover.

1. mnemosyne missing from the committed gateway env (repaired 21:26-21:34, verified 22:01:19). The gateway does not run from a normal venv: it launches as /home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 -I -c '<bootstrap>' gateway run and activate_dependencies() replaces sys.path with one committed uv environment - ~/.hermes/installs/<id>/environments/795020dbf24343cd9c52df24723dca83/venv. Of the uv environments on CT460, that one alone was missing mnemosyne / mnemosyne_hermes, so sessions logged mnemosyne_batch helpers unavailable (No module named 'mnemosyne') and Memory provider 'mnemosyne' is selected but reports unavailable. Fix: install hermes_memory_provider + mnemosyne_hermes + mnemosyne together with the native dependencies (fastembed, onnxruntime, numpy + numpy.libs, tokenizers, huggingface_hub, sqlite-vec, loguru, mmh3) into that committed venv. Verified live: Memory provider 'mnemosyne' registered (40 tools) -> activated at 2026-10-10 22:01:19, again at 23:59 / 00:00 / 00:02; and hermes memory status -> Plugin: installed / Status: available.

2. elevenlabs missing from the live managed env (21:13 -> 21:24). TTS was configured but the package was absent from the managed runtime in use; reinstalled and verified by generating a real clip.

Trap: ~/.local/lib/python3.14/site-packages is invisible to the gateway. The first repair attempt installed into the user site directory; with ENABLE_USER_SITE=False and a replaced sys.path, the gateway never sees it. Install into the committed uv env that activate_dependencies() selects - not the user site dir, not a sibling venv.

Trap: hermes memory status can be green while the gateway is still broken. It is a separate process with its own runtime resolution. The gateway's own log lines (registered (40 tools) / activated versus reports unavailable) are the authoritative check.

Trap: this recurrence produced no cron failures at all (2026-10-10: 174 completed, 0 failed). It degraded the gateway's memory and TTS paths only. A green cron day does not mean the runtime is healthy - and the 15-minute guard still does not check the gateway's committed env, so guard coverage remains open.