9.1 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Hermes Managed-Runtime Dependency Failure | incident | resolved | 2026-09-30 | 2026-11-04 | 2026-11-04 | high |
|
|
Hermes Managed-Runtime Dependency Failure
Window: ~2026-09-28 → 2026-09-30 19:13 AWST Status: Resolved — dependency guard installed, watchdog green Impact: Cron jobs dispatched through the external worker; Mnemosyne Watchdog failed 176 consecutive runs
Symptoms
Affected jobs reported the same dispatch error, which points at the wrong layer:
Restart-safe cron worker dispatch failed ... cron external worker exited
before ownership acknowledgement (exit 1)
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in
sys.modules after import of package 'cron', but prior to execution of
'cron.scheduler'
The RuntimeWarning is noise. The real error is further down the worker stderr.
Root cause
Hermes migrated to uv-managed runtimes. Each runtime gets its own interpreter and venv:
~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv # 10 venvs at time of incident
# (now **3** — see 2026-11-04 re-verification below)
The migration did not carry every dependency into every managed venv. Workers launched against a venv missing dotenv and mnemosyne died during import — before acknowledging ownership of the job. The failure therefore surfaced as a scheduler/dispatch bug while the real cause was:
ModuleNotFoundError: No module named 'dotenv'(459 occurrences in worker stderr)ModuleNotFoundError: No module named 'mnemosyne'
Diagnosis
Read worker stderr, not just the job's error field:
grep -rl "ModuleNotFoundError" ~/.hermes/logs/ # find the worker log
hermes cron list # which jobs, which streaks
ls -d ~/.hermes/installs/*/environments/*/venv # how many runtimes exist
Test the live interpreter directly — never trust *.dist-info.
Fix
/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh (v2, installed 2026-09-30 19:13 AWST):
- discovers every managed venv under
~/.hermes/installs/*/environments/*/venv(v1 checked only 2) - verifies
import dotenvandimport mnemosyneagainst each venv - repairs missing packages with
pip install --target <venv site-packages>
Wiring: the Mnemosyne Watchdog job (96eb95ce12f2, */15 * * * *) runs the guard every 15 minutes, so a future managed-runtime migration self-heals within 15 minutes.
Pitfalls found while fixing
uv pip install --python <venv>/bin/pythonwalks up to the~/.hermes/hermes-agentworkspace and installs there — the target venv stays empty. Usepip install --target.- Stale
*.dist-infodirectories makeuvskip a package as "already installed" even when the module is not importable. write_filesilently no-ops on.hermes/scripts/paths (Hermes library bug). Stage the script in/tmpandinstallit into place.
Re-verification (2026-11-04)
The incident remains resolved, but the estate's runtime topology has changed since 2026-09-30 and the page's "10 venvs" figures are now stale. Live state on CT460:
| Item | 2026-09-30 (page) | 2026-11-04 (live) |
|---|---|---|
Managed venvs under ~/.hermes/installs/*/environments/*/venv |
10 | 3 — 19da945341ad4794ac2bc8ff9fdd3643, d41b16c8b6674454a4a819bd4acced1f, f51d6e4902414114974145d8041c79cb |
| Total runtimes the guard checks | 10 | 4 (3 managed venvs + the tools installer python) |
| Guard result | 10/10 OK | hermes-runtime-deps-guard: 4/4 runtimes OK (exit 0) |
| Bootstrap-selected runtime | n/a | f51d6e4902414114974145d8041c79cb — confirmed by running hermes_bootstrap under the tools python: sys.path → …/f51d6e4902414114974145d8041c79cb/venv/lib/python3.14/site-packages |
fastembed in the bootstrap-selected venv |
repaired | OK — guard log: fastembed OK in f51d6e4902414114974145d8041c79cb |
| Fastembed model cache | n/a | present — ~/.hermes/cache/fastembed/models--qdrant--bge-small-en-v1.5-onnx-q |
Old gateway committed env 795020dbf24343cd9c52df24723dca83 |
the one that broke on 2026-10-10 | no longer exists — the environments/ directory now holds only the three venvs above |
The guard script itself is unchanged and still wired in: ~/.hermes/scripts/hermes-runtime-deps-guard.sh exists
(executable, 6479 bytes) and is invoked by ~/.hermes/scripts/mnemosyne-watchdog.py at line 62.
Two notes for the next time this class recurs. First, the "10 venvs" figure is not a stable invariant —
Hermes prunes and re-creates uv environments, so count the live venvs rather than trusting a recorded number;
the guard's own 4/4 runtimes OK line is the authoritative reading. Second, the gateway's committed env is no
longer a separate directory — the 795020dbf… env from the 2026-10-10 recurrence has been garbage-collected,
and hermes_bootstrap now selects f51d6e49… (one of the three surviving managed venvs). The 2026-10-10
failure mode ("the gateway loads an env the guard does not check") may therefore no longer reproduce in that exact
shape, but the underlying trap — a sys.path that points somewhere the guard never inspects — is unchanged, so
the preventive rule below still stands.
Verification (2026-09-30)
| Check | Result |
|---|---|
Managed venvs importing dotenv + mnemosyne |
10/10 |
Mnemosyne Watchdog 96eb95ce12f2 last_status |
ok (19:15 AWST), streak 0 |
| Jobs recovered | 12 of 13 |
| Residual | cbfe434a3b5e (Wiki Audit) and 1a8e90ea2271 (wiki-stocktake) still carry the pre-fix error — both last ran before the 19:13 fix. Expected to clear on their next runs. |
Preventive rules
- Treat "worker exited before ownership acknowledgement" as a dependency/import failure first. Read worker stderr before touching provider or model pins.
- After any Hermes runtime or Python upgrade, re-run the guard and confirm every runtime it reports imports
dotenvandmnemosyne— read the guard's ownN/N runtimes OKline rather than a recorded venv count, because the number of managed venvs changes as Hermes prunes and re-creates uv environments (10 on 2026-09-30, 3 on 2026-11-04). - Mass-cron-failure triage order: live interpreter imports → worker stderr → runtime layout → job logic. See runbooks/restart-hermes and systems/scheduled-tasks.
Related
Recurrence - 2026-10-10 (this time the gateway runtime env, plus a new failure mode)
The class recurred twice in one day, on packages the 15-minute guard does not cover.
1. mnemosyne missing from the committed gateway env (repaired 21:26-21:34, verified 22:01:19).
The gateway does not run from a normal venv: it launches as
/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 -I -c '<bootstrap>' gateway run
and activate_dependencies() replaces sys.path with one committed uv environment -
~/.hermes/installs/<id>/environments/795020dbf24343cd9c52df24723dca83/venv. Of the uv
environments on CT460, that one alone was missing mnemosyne / mnemosyne_hermes, so sessions
logged mnemosyne_batch helpers unavailable (No module named 'mnemosyne') and
Memory provider 'mnemosyne' is selected but reports unavailable. Fix: install
hermes_memory_provider + mnemosyne_hermes + mnemosyne together with the native dependencies
(fastembed, onnxruntime, numpy + numpy.libs, tokenizers, huggingface_hub, sqlite-vec,
loguru, mmh3) into that committed venv. Verified live: Memory provider 'mnemosyne' registered (40 tools) -> activated at 2026-10-10 22:01:19, again at 23:59 / 00:00 / 00:02; and
hermes memory status -> Plugin: installed / Status: available.
2. elevenlabs missing from the live managed env (21:13 -> 21:24). TTS was configured but the
package was absent from the managed runtime in use; reinstalled and verified by generating a real
clip.
Trap: ~/.local/lib/python3.14/site-packages is invisible to the gateway. The first repair
attempt installed into the user site directory; with ENABLE_USER_SITE=False and a replaced
sys.path, the gateway never sees it. Install into the committed uv env that
activate_dependencies() selects - not the user site dir, not a sibling venv.
Trap: hermes memory status can be green while the gateway is still broken. It is a separate
process with its own runtime resolution. The gateway's own log lines
(registered (40 tools) / activated versus reports unavailable) are the authoritative check.
Trap: this recurrence produced no cron failures at all (2026-10-10: 174 completed, 0 failed).
It degraded the gateway's memory and TTS paths only. A green cron day does not mean the runtime
is healthy - and the 15-minute guard still does not check the gateway's committed env, so guard
coverage remains open.