4.0 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Hermes Managed-Runtime Dependency Failure | incident | resolved | 2026-09-30 | 2026-09-30 | 2026-09-30 | high |
|
|
Hermes Managed-Runtime Dependency Failure
Window: ~2026-09-28 → 2026-09-30 19:13 AWST Status: Resolved — dependency guard installed, watchdog green Impact: Cron jobs dispatched through the external worker; Mnemosyne Watchdog failed 176 consecutive runs
Symptoms
Affected jobs reported the same dispatch error, which points at the wrong layer:
Restart-safe cron worker dispatch failed ... cron external worker exited
before ownership acknowledgement (exit 1)
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in
sys.modules after import of package 'cron', but prior to execution of
'cron.scheduler'
The RuntimeWarning is noise. The real error is further down the worker stderr.
Root cause
Hermes migrated to uv-managed runtimes. Each runtime gets its own interpreter and venv:
~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv # 10 venvs at time of incident
The migration did not carry every dependency into every managed venv. Workers launched against a venv missing dotenv and mnemosyne died during import — before acknowledging ownership of the job. The failure therefore surfaced as a scheduler/dispatch bug while the real cause was:
ModuleNotFoundError: No module named 'dotenv'(459 occurrences in worker stderr)ModuleNotFoundError: No module named 'mnemosyne'
Diagnosis
Read worker stderr, not just the job's error field:
grep -rl "ModuleNotFoundError" ~/.hermes/logs/ # find the worker log
hermes cron list # which jobs, which streaks
ls -d ~/.hermes/installs/*/environments/*/venv # how many runtimes exist
Test the live interpreter directly — never trust *.dist-info.
Fix
/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh (v2, installed 2026-09-30 19:13 AWST):
- discovers every managed venv under
~/.hermes/installs/*/environments/*/venv(v1 checked only 2) - verifies
import dotenvandimport mnemosyneagainst each venv - repairs missing packages with
pip install --target <venv site-packages>
Wiring: the Mnemosyne Watchdog job (96eb95ce12f2, */15 * * * *) runs the guard every 15 minutes, so a future managed-runtime migration self-heals within 15 minutes.
Pitfalls found while fixing
uv pip install --python <venv>/bin/pythonwalks up to the~/.hermes/hermes-agentworkspace and installs there — the target venv stays empty. Usepip install --target.- Stale
*.dist-infodirectories makeuvskip a package as "already installed" even when the module is not importable. write_filesilently no-ops on.hermes/scripts/paths (Hermes library bug). Stage the script in/tmpandinstallit into place.
Verification (2026-09-30)
| Check | Result |
|---|---|
Managed venvs importing dotenv + mnemosyne |
10/10 |
Mnemosyne Watchdog 96eb95ce12f2 last_status |
ok (19:15 AWST), streak 0 |
| Jobs recovered | 12 of 13 |
| Residual | cbfe434a3b5e (Wiki Audit) and 1a8e90ea2271 (wiki-stocktake) still carry the pre-fix error — both last ran before the 19:13 fix. Expected to clear on their next runs. |
Preventive rules
- Treat "worker exited before ownership acknowledgement" as a dependency/import failure first. Read worker stderr before touching provider or model pins.
- After any Hermes runtime or Python upgrade, re-run the guard and confirm 10/10 managed venvs import
dotenvandmnemosyne. - Mass-cron-failure triage order: live interpreter imports → worker stderr → runtime layout → job logic. See runbooks/restart-hermes and systems/scheduled-tasks.