--- title: Hermes Managed-Runtime Dependency Failure type: incident status: resolved created: 2026-09-30 updated: 2026-10-11 verified_on: 2026-10-11 confidence: high tags: [hermes, cron, runtime, python, worker, incident] sources: [live-jobs-json, live-guard-script, live-cron-list] --- # Hermes Managed-Runtime Dependency Failure **Window:** ~2026-09-28 → 2026-09-30 19:13 AWST **Status:** Resolved — dependency guard installed, watchdog green **Impact:** Cron jobs dispatched through the external worker; Mnemosyne Watchdog failed 176 consecutive runs ## Symptoms Affected jobs reported the same dispatch error, which points at the wrong layer: ``` Restart-safe cron worker dispatch failed ... cron external worker exited before ownership acknowledgement (exit 1) worker stderr: :130: RuntimeWarning: 'cron.scheduler' found in sys.modules after import of package 'cron', but prior to execution of 'cron.scheduler' ``` The `RuntimeWarning` is noise. The real error is further down the worker stderr. ## Root cause Hermes migrated to **uv-managed runtimes**. Each runtime gets its own interpreter and venv: ``` ~/.hermes/installs/019d0d114d22b7e6/environments//venv # 10 venvs at time of incident ``` The migration did not carry every dependency into every managed venv. Workers launched against a venv missing `dotenv` and `mnemosyne` died during import — *before* acknowledging ownership of the job. The failure therefore surfaced as a scheduler/dispatch bug while the real cause was: - `ModuleNotFoundError: No module named 'dotenv'` (459 occurrences in worker stderr) - `ModuleNotFoundError: No module named 'mnemosyne'` ## Diagnosis Read worker stderr, not just the job's error field: ```bash grep -rl "ModuleNotFoundError" ~/.hermes/logs/ # find the worker log hermes cron list # which jobs, which streaks ls -d ~/.hermes/installs/*/environments/*/venv # how many runtimes exist ``` Test the live interpreter directly — never trust `*.dist-info`. ## Fix `/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh` (v2, installed 2026-09-30 19:13 AWST): - discovers **every** managed venv under `~/.hermes/installs/*/environments/*/venv` (v1 checked only 2) - verifies `import dotenv` and `import mnemosyne` against each venv - repairs missing packages with `pip install --target ` Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the guard every 15 minutes, so a future managed-runtime migration self-heals within 15 minutes. ## Pitfalls found while fixing - `uv pip install --python /bin/python` **walks up** to the `~/.hermes/hermes-agent` workspace and installs there — the target venv stays empty. Use `pip install --target`. - Stale `*.dist-info` directories make `uv` skip a package as "already installed" even when the module is not importable. - `write_file` silently no-ops on `.hermes/scripts/` paths (Hermes library bug). Stage the script in `/tmp` and `install` it into place. ## Verification (2026-09-30) | Check | Result | |---|---| | Managed venvs importing `dotenv` + `mnemosyne` | 10/10 | | Mnemosyne Watchdog `96eb95ce12f2` last_status | `ok` (19:15 AWST), streak 0 | | Jobs recovered | 12 of 13 | | Residual | `cbfe434a3b5e` (Wiki Audit) and `1a8e90ea2271` (wiki-stocktake) still carry the pre-fix error — both last ran **before** the 19:13 fix. Expected to clear on their next runs. | ## Preventive rules - Treat "worker exited before ownership acknowledgement" as a **dependency/import** failure first. Read worker stderr before touching provider or model pins. - After any Hermes runtime or Python upgrade, re-run the guard and confirm 10/10 managed venvs import `dotenv` and `mnemosyne`. - Mass-cron-failure triage order: live interpreter imports → worker stderr → runtime layout → job logic. See [[runbooks/restart-hermes]] and [[systems/scheduled-tasks]]. ## Related - [[systems/scheduled-tasks]] - [[systems/hermes-agent]] - [[systems/mnemosyne]] - [[runbooks/restart-hermes]] --- ## Recurrence - 2026-10-10 (this time the *gateway* runtime env, plus a new failure mode) **The class recurred twice in one day, on packages the 15-minute guard does not cover.** **1. `mnemosyne` missing from the committed gateway env (repaired 21:26-21:34, verified 22:01:19).** The gateway does not run from a normal venv: it launches as `/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 -I -c '' gateway run` and `activate_dependencies()` **replaces `sys.path`** with one committed uv environment - `~/.hermes/installs//environments/795020dbf24343cd9c52df24723dca83/venv`. Of the uv environments on CT460, that one alone was missing `mnemosyne` / `mnemosyne_hermes`, so sessions logged `mnemosyne_batch helpers unavailable (No module named 'mnemosyne')` and `Memory provider 'mnemosyne' is selected but reports unavailable`. Fix: install `hermes_memory_provider` + `mnemosyne_hermes` + `mnemosyne` **together with the native dependencies** (`fastembed`, `onnxruntime`, `numpy` + `numpy.libs`, `tokenizers`, `huggingface_hub`, `sqlite-vec`, `loguru`, `mmh3`) into that committed venv. Verified live: `Memory provider 'mnemosyne' registered (40 tools)` -> `activated` at 2026-10-10 22:01:19, again at 23:59 / 00:00 / 00:02; and `hermes memory status` -> `Plugin: installed / Status: available`. **2. `elevenlabs` missing from the live managed env (21:13 -> 21:24).** TTS was configured but the package was absent from the managed runtime in use; reinstalled and verified by generating a real clip. **Trap: `~/.local/lib/python3.14/site-packages` is invisible to the gateway.** The first repair attempt installed into the user site directory; with `ENABLE_USER_SITE=False` **and** a replaced `sys.path`, the gateway never sees it. Install into the committed uv env that `activate_dependencies()` selects - not the user site dir, not a sibling venv. **Trap: `hermes memory status` can be green while the gateway is still broken.** It is a separate process with its own runtime resolution. The gateway's own log lines (`registered (40 tools)` / `activated` versus `reports unavailable`) are the authoritative check. **Trap: this recurrence produced no cron failures at all** (2026-10-10: 174 `completed`, 0 `failed`). It degraded the gateway's memory and TTS paths only. A green cron day does **not** mean the runtime is healthy - and the 15-minute guard still does not check the gateway's committed env, so **guard coverage remains open.**