Wiki audit 2026-09-30: managed-runtime dependency incident, cron fleet reconciliation, provider/model refresh
This commit is contained in:
@@ -0,0 +1,94 @@
|
||||
---
|
||||
title: Hermes Managed-Runtime Dependency Failure
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-09-30
|
||||
updated: 2026-09-30
|
||||
verified_on: 2026-09-30
|
||||
confidence: high
|
||||
tags: [hermes, cron, runtime, python, worker, incident]
|
||||
sources: [live-jobs-json, live-guard-script, live-cron-list]
|
||||
---
|
||||
|
||||
# Hermes Managed-Runtime Dependency Failure
|
||||
|
||||
**Window:** ~2026-09-28 → 2026-09-30 19:13 AWST
|
||||
**Status:** Resolved — dependency guard installed, watchdog green
|
||||
**Impact:** Cron jobs dispatched through the external worker; Mnemosyne Watchdog failed 176 consecutive runs
|
||||
|
||||
## Symptoms
|
||||
|
||||
Affected jobs reported the same dispatch error, which points at the wrong layer:
|
||||
|
||||
```
|
||||
Restart-safe cron worker dispatch failed ... cron external worker exited
|
||||
before ownership acknowledgement (exit 1)
|
||||
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in
|
||||
sys.modules after import of package 'cron', but prior to execution of
|
||||
'cron.scheduler'
|
||||
```
|
||||
|
||||
The `RuntimeWarning` is noise. The real error is further down the worker stderr.
|
||||
|
||||
## Root cause
|
||||
|
||||
Hermes migrated to **uv-managed runtimes**. Each runtime gets its own interpreter and venv:
|
||||
|
||||
```
|
||||
~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv # 10 venvs at time of incident
|
||||
```
|
||||
|
||||
The migration did not carry every dependency into every managed venv. Workers launched against a venv missing `dotenv` and `mnemosyne` died during import — *before* acknowledging ownership of the job. The failure therefore surfaced as a scheduler/dispatch bug while the real cause was:
|
||||
|
||||
- `ModuleNotFoundError: No module named 'dotenv'` (459 occurrences in worker stderr)
|
||||
- `ModuleNotFoundError: No module named 'mnemosyne'`
|
||||
|
||||
## Diagnosis
|
||||
|
||||
Read worker stderr, not just the job's error field:
|
||||
|
||||
```bash
|
||||
grep -rl "ModuleNotFoundError" ~/.hermes/logs/ # find the worker log
|
||||
hermes cron list # which jobs, which streaks
|
||||
ls -d ~/.hermes/installs/*/environments/*/venv # how many runtimes exist
|
||||
```
|
||||
|
||||
Test the live interpreter directly — never trust `*.dist-info`.
|
||||
|
||||
## Fix
|
||||
|
||||
`/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh` (v2, installed 2026-09-30 19:13 AWST):
|
||||
|
||||
- discovers **every** managed venv under `~/.hermes/installs/*/environments/*/venv` (v1 checked only 2)
|
||||
- verifies `import dotenv` and `import mnemosyne` against each venv
|
||||
- repairs missing packages with `pip install --target <venv site-packages>`
|
||||
|
||||
Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the guard every 15 minutes, so a future managed-runtime migration self-heals within 15 minutes.
|
||||
|
||||
## Pitfalls found while fixing
|
||||
|
||||
- `uv pip install --python <venv>/bin/python` **walks up** to the `~/.hermes/hermes-agent` workspace and installs there — the target venv stays empty. Use `pip install --target`.
|
||||
- Stale `*.dist-info` directories make `uv` skip a package as "already installed" even when the module is not importable.
|
||||
- `write_file` silently no-ops on `.hermes/scripts/` paths (Hermes library bug). Stage the script in `/tmp` and `install` it into place.
|
||||
|
||||
## Verification (2026-09-30)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Managed venvs importing `dotenv` + `mnemosyne` | 10/10 |
|
||||
| Mnemosyne Watchdog `96eb95ce12f2` last_status | `ok` (19:15 AWST), streak 0 |
|
||||
| Jobs recovered | 12 of 13 |
|
||||
| Residual | `cbfe434a3b5e` (Wiki Audit) and `1a8e90ea2271` (wiki-stocktake) still carry the pre-fix error — both last ran **before** the 19:13 fix. Expected to clear on their next runs. |
|
||||
|
||||
## Preventive rules
|
||||
|
||||
- Treat "worker exited before ownership acknowledgement" as a **dependency/import** failure first. Read worker stderr before touching provider or model pins.
|
||||
- After any Hermes runtime or Python upgrade, re-run the guard and confirm 10/10 managed venvs import `dotenv` and `mnemosyne`.
|
||||
- Mass-cron-failure triage order: live interpreter imports → worker stderr → runtime layout → job logic. See [[runbooks/restart-hermes]] and [[systems/scheduled-tasks]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/hermes-agent]]
|
||||
- [[systems/mnemosyne]]
|
||||
- [[runbooks/restart-hermes]]
|
||||
Reference in New Issue
Block a user