Files
agent-estate-wiki/incidents/2026-09-28-hermes-managed-runtime-deps.md
T

4.0 KiB

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Hermes Managed-Runtime Dependency Failure incident resolved 2026-09-30 2026-09-30 2026-09-30 high
hermes
cron
runtime
python
worker
incident
live-jobs-json
live-guard-script
live-cron-list

Hermes Managed-Runtime Dependency Failure

Window: ~2026-09-28 → 2026-09-30 19:13 AWST Status: Resolved — dependency guard installed, watchdog green Impact: Cron jobs dispatched through the external worker; Mnemosyne Watchdog failed 176 consecutive runs

Symptoms

Affected jobs reported the same dispatch error, which points at the wrong layer:

Restart-safe cron worker dispatch failed ... cron external worker exited
before ownership acknowledgement (exit 1)
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in
sys.modules after import of package 'cron', but prior to execution of
'cron.scheduler'

The RuntimeWarning is noise. The real error is further down the worker stderr.

Root cause

Hermes migrated to uv-managed runtimes. Each runtime gets its own interpreter and venv:

~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv      # 10 venvs at time of incident

The migration did not carry every dependency into every managed venv. Workers launched against a venv missing dotenv and mnemosyne died during import — before acknowledging ownership of the job. The failure therefore surfaced as a scheduler/dispatch bug while the real cause was:

  • ModuleNotFoundError: No module named 'dotenv' (459 occurrences in worker stderr)
  • ModuleNotFoundError: No module named 'mnemosyne'

Diagnosis

Read worker stderr, not just the job's error field:

grep -rl "ModuleNotFoundError" ~/.hermes/logs/          # find the worker log
hermes cron list                                        # which jobs, which streaks
ls -d ~/.hermes/installs/*/environments/*/venv          # how many runtimes exist

Test the live interpreter directly — never trust *.dist-info.

Fix

/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh (v2, installed 2026-09-30 19:13 AWST):

  • discovers every managed venv under ~/.hermes/installs/*/environments/*/venv (v1 checked only 2)
  • verifies import dotenv and import mnemosyne against each venv
  • repairs missing packages with pip install --target <venv site-packages>

Wiring: the Mnemosyne Watchdog job (96eb95ce12f2, */15 * * * *) runs the guard every 15 minutes, so a future managed-runtime migration self-heals within 15 minutes.

Pitfalls found while fixing

  • uv pip install --python <venv>/bin/python walks up to the ~/.hermes/hermes-agent workspace and installs there — the target venv stays empty. Use pip install --target.
  • Stale *.dist-info directories make uv skip a package as "already installed" even when the module is not importable.
  • write_file silently no-ops on .hermes/scripts/ paths (Hermes library bug). Stage the script in /tmp and install it into place.

Verification (2026-09-30)

Check Result
Managed venvs importing dotenv + mnemosyne 10/10
Mnemosyne Watchdog 96eb95ce12f2 last_status ok (19:15 AWST), streak 0
Jobs recovered 12 of 13
Residual cbfe434a3b5e (Wiki Audit) and 1a8e90ea2271 (wiki-stocktake) still carry the pre-fix error — both last ran before the 19:13 fix. Expected to clear on their next runs.

Preventive rules

  • Treat "worker exited before ownership acknowledgement" as a dependency/import failure first. Read worker stderr before touching provider or model pins.
  • After any Hermes runtime or Python upgrade, re-run the guard and confirm 10/10 managed venvs import dotenv and mnemosyne.
  • Mass-cron-failure triage order: live interpreter imports → worker stderr → runtime layout → job logic. See runbooks/restart-hermes and systems/scheduled-tasks.