162 lines
9.1 KiB
Markdown
162 lines
9.1 KiB
Markdown
---
|
|
title: Hermes Managed-Runtime Dependency Failure
|
|
type: incident
|
|
status: resolved
|
|
created: 2026-09-30
|
|
updated: 2026-11-04
|
|
verified_on: 2026-11-04
|
|
confidence: high
|
|
tags: [hermes, cron, runtime, python, worker, incident]
|
|
sources: [live-jobs-json, live-guard-script, live-cron-list]
|
|
---
|
|
|
|
# Hermes Managed-Runtime Dependency Failure
|
|
|
|
**Window:** ~2026-09-28 → 2026-09-30 19:13 AWST
|
|
**Status:** Resolved — dependency guard installed, watchdog green
|
|
**Impact:** Cron jobs dispatched through the external worker; Mnemosyne Watchdog failed 176 consecutive runs
|
|
|
|
## Symptoms
|
|
|
|
Affected jobs reported the same dispatch error, which points at the wrong layer:
|
|
|
|
```
|
|
Restart-safe cron worker dispatch failed ... cron external worker exited
|
|
before ownership acknowledgement (exit 1)
|
|
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in
|
|
sys.modules after import of package 'cron', but prior to execution of
|
|
'cron.scheduler'
|
|
```
|
|
|
|
The `RuntimeWarning` is noise. The real error is further down the worker stderr.
|
|
|
|
## Root cause
|
|
|
|
Hermes migrated to **uv-managed runtimes**. Each runtime gets its own interpreter and venv:
|
|
|
|
```
|
|
~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv # 10 venvs at time of incident
|
|
# (now **3** — see 2026-11-04 re-verification below)
|
|
```
|
|
|
|
The migration did not carry every dependency into every managed venv. Workers launched against a venv missing `dotenv` and `mnemosyne` died during import — *before* acknowledging ownership of the job. The failure therefore surfaced as a scheduler/dispatch bug while the real cause was:
|
|
|
|
- `ModuleNotFoundError: No module named 'dotenv'` (459 occurrences in worker stderr)
|
|
- `ModuleNotFoundError: No module named 'mnemosyne'`
|
|
|
|
## Diagnosis
|
|
|
|
Read worker stderr, not just the job's error field:
|
|
|
|
```bash
|
|
grep -rl "ModuleNotFoundError" ~/.hermes/logs/ # find the worker log
|
|
hermes cron list # which jobs, which streaks
|
|
ls -d ~/.hermes/installs/*/environments/*/venv # how many runtimes exist
|
|
```
|
|
|
|
Test the live interpreter directly — never trust `*.dist-info`.
|
|
|
|
## Fix
|
|
|
|
`/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh` (v2, installed 2026-09-30 19:13 AWST):
|
|
|
|
- discovers **every** managed venv under `~/.hermes/installs/*/environments/*/venv` (v1 checked only 2)
|
|
- verifies `import dotenv` and `import mnemosyne` against each venv
|
|
- repairs missing packages with `pip install --target <venv site-packages>`
|
|
|
|
Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the guard every 15 minutes, so a future managed-runtime migration self-heals within 15 minutes.
|
|
|
|
## Pitfalls found while fixing
|
|
|
|
- `uv pip install --python <venv>/bin/python` **walks up** to the `~/.hermes/hermes-agent` workspace and installs there — the target venv stays empty. Use `pip install --target`.
|
|
- Stale `*.dist-info` directories make `uv` skip a package as "already installed" even when the module is not importable.
|
|
- `write_file` silently no-ops on `.hermes/scripts/` paths (Hermes library bug). Stage the script in `/tmp` and `install` it into place.
|
|
|
|
## Re-verification (2026-11-04)
|
|
|
|
**The incident remains resolved, but the estate's runtime topology has changed since 2026-09-30 and the page's
|
|
"10 venvs" figures are now stale.** Live state on CT460:
|
|
|
|
| Item | 2026-09-30 (page) | 2026-11-04 (live) |
|
|
|---|---|---|
|
|
| Managed venvs under `~/.hermes/installs/*/environments/*/venv` | **10** | **3** — `19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`, `f51d6e4902414114974145d8041c79cb` |
|
|
| Total runtimes the guard checks | 10 | **4** (3 managed venvs + the tools installer python) |
|
|
| Guard result | 10/10 OK | **`hermes-runtime-deps-guard: 4/4 runtimes OK`** (exit 0) |
|
|
| Bootstrap-selected runtime | n/a | **`f51d6e4902414114974145d8041c79cb`** — confirmed by running `hermes_bootstrap` under the tools python: `sys.path` → `…/f51d6e4902414114974145d8041c79cb/venv/lib/python3.14/site-packages` |
|
|
| `fastembed` in the bootstrap-selected venv | repaired | **OK** — guard log: `fastembed OK in f51d6e4902414114974145d8041c79cb` |
|
|
| Fastembed model cache | n/a | **present** — `~/.hermes/cache/fastembed/models--qdrant--bge-small-en-v1.5-onnx-q` |
|
|
| Old gateway committed env `795020dbf24343cd9c52df24723dca83` | the one that broke on 2026-10-10 | **no longer exists** — the `environments/` directory now holds only the three venvs above |
|
|
|
|
The guard script itself is unchanged and still wired in: `~/.hermes/scripts/hermes-runtime-deps-guard.sh` exists
|
|
(executable, 6479 bytes) and is invoked by `~/.hermes/scripts/mnemosyne-watchdog.py` at line 62.
|
|
|
|
**Two notes for the next time this class recurs.** First, the "10 venvs" figure is not a stable invariant —
|
|
Hermes prunes and re-creates uv environments, so **count the live venvs rather than trusting a recorded number**;
|
|
the guard's own `4/4 runtimes OK` line is the authoritative reading. Second, the **gateway's committed env is no
|
|
longer a separate directory** — the `795020dbf…` env from the 2026-10-10 recurrence has been garbage-collected,
|
|
and `hermes_bootstrap` now selects `f51d6e49…` (one of the three surviving managed venvs). The 2026-10-10
|
|
failure mode ("the gateway loads an env the guard does not check") may therefore no longer reproduce in that exact
|
|
shape, but the underlying trap — *a `sys.path` that points somewhere the guard never inspects* — is unchanged, so
|
|
the preventive rule below still stands.
|
|
|
|
## Verification (2026-09-30)
|
|
|
|
| Check | Result |
|
|
|---|---|
|
|
| Managed venvs importing `dotenv` + `mnemosyne` | 10/10 |
|
|
| Mnemosyne Watchdog `96eb95ce12f2` last_status | `ok` (19:15 AWST), streak 0 |
|
|
| Jobs recovered | 12 of 13 |
|
|
| Residual | `cbfe434a3b5e` (Wiki Audit) and `1a8e90ea2271` (wiki-stocktake) still carry the pre-fix error — both last ran **before** the 19:13 fix. Expected to clear on their next runs. |
|
|
|
|
## Preventive rules
|
|
|
|
- Treat "worker exited before ownership acknowledgement" as a **dependency/import** failure first. Read worker stderr before touching provider or model pins.
|
|
- After any Hermes runtime or Python upgrade, re-run the guard and confirm **every runtime it reports** imports `dotenv` and `mnemosyne` — read the guard's own `N/N runtimes OK` line rather than a recorded venv count, because the number of managed venvs changes as Hermes prunes and re-creates uv environments (10 on 2026-09-30, 3 on 2026-11-04).
|
|
- Mass-cron-failure triage order: live interpreter imports → worker stderr → runtime layout → job logic. See [[runbooks/restart-hermes]] and [[systems/scheduled-tasks]].
|
|
|
|
## Related
|
|
|
|
- [[systems/scheduled-tasks]]
|
|
- [[systems/hermes-agent]]
|
|
- [[systems/mnemosyne]]
|
|
- [[runbooks/restart-hermes]]
|
|
|
|
|
|
---
|
|
|
|
## Recurrence - 2026-10-10 (this time the *gateway* runtime env, plus a new failure mode)
|
|
|
|
**The class recurred twice in one day, on packages the 15-minute guard does not cover.**
|
|
|
|
**1. `mnemosyne` missing from the committed gateway env (repaired 21:26-21:34, verified 22:01:19).**
|
|
The gateway does not run from a normal venv: it launches as
|
|
`/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 -I -c '<bootstrap>' gateway run`
|
|
and `activate_dependencies()` **replaces `sys.path`** with one committed uv environment -
|
|
`~/.hermes/installs/<id>/environments/795020dbf24343cd9c52df24723dca83/venv`. Of the uv
|
|
environments on CT460, that one alone was missing `mnemosyne` / `mnemosyne_hermes`, so sessions
|
|
logged `mnemosyne_batch helpers unavailable (No module named 'mnemosyne')` and
|
|
`Memory provider 'mnemosyne' is selected but reports unavailable`. Fix: install
|
|
`hermes_memory_provider` + `mnemosyne_hermes` + `mnemosyne` **together with the native dependencies**
|
|
(`fastembed`, `onnxruntime`, `numpy` + `numpy.libs`, `tokenizers`, `huggingface_hub`, `sqlite-vec`,
|
|
`loguru`, `mmh3`) into that committed venv. Verified live: `Memory provider 'mnemosyne' registered
|
|
(40 tools)` -> `activated` at 2026-10-10 22:01:19, again at 23:59 / 00:00 / 00:02; and
|
|
`hermes memory status` -> `Plugin: installed / Status: available`.
|
|
|
|
**2. `elevenlabs` missing from the live managed env (21:13 -> 21:24).** TTS was configured but the
|
|
package was absent from the managed runtime in use; reinstalled and verified by generating a real
|
|
clip.
|
|
|
|
**Trap: `~/.local/lib/python3.14/site-packages` is invisible to the gateway.** The first repair
|
|
attempt installed into the user site directory; with `ENABLE_USER_SITE=False` **and** a replaced
|
|
`sys.path`, the gateway never sees it. Install into the committed uv env that
|
|
`activate_dependencies()` selects - not the user site dir, not a sibling venv.
|
|
|
|
**Trap: `hermes memory status` can be green while the gateway is still broken.** It is a separate
|
|
process with its own runtime resolution. The gateway's own log lines
|
|
(`registered (40 tools)` / `activated` versus `reports unavailable`) are the authoritative check.
|
|
|
|
**Trap: this recurrence produced no cron failures at all** (2026-10-10: 174 `completed`, 0 `failed`).
|
|
It degraded the gateway's memory and TTS paths only. A green cron day does **not** mean the runtime
|
|
is healthy - and the 15-minute guard still does not check the gateway's committed env, so **guard
|
|
coverage remains open.**
|