audit(2026-10-10): DHL monitor retired (29 jobs / 26 enabled, verified by ID lookup); 10-10 fleet clean 174/0, 7 incidents closed - 5 open; gateway runtime env recurrence (mnemosyne + elevenlabs) verified fixed; new systems/paseo.md; mattermost unreachable; current-state + index + log updated

This commit is contained in:
hermes
2026-10-11 00:22:20 +08:00
parent c11f11e652
commit 7bffeab157
9 changed files with 295 additions and 18 deletions
@@ -3,8 +3,8 @@ title: Hermes Managed-Runtime Dependency Failure
type: incident
status: resolved
created: 2026-09-30
updated: 2026-09-30
verified_on: 2026-09-30
updated: 2026-10-11
verified_on: 2026-10-11
confidence: high
tags: [hermes, cron, runtime, python, worker, incident]
sources: [live-jobs-json, live-guard-script, live-cron-list]
@@ -92,3 +92,42 @@ Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the
- [[systems/hermes-agent]]
- [[systems/mnemosyne]]
- [[runbooks/restart-hermes]]
---
## Recurrence - 2026-10-10 (this time the *gateway* runtime env, plus a new failure mode)
**The class recurred twice in one day, on packages the 15-minute guard does not cover.**
**1. `mnemosyne` missing from the committed gateway env (repaired 21:26-21:34, verified 22:01:19).**
The gateway does not run from a normal venv: it launches as
`/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 -I -c '<bootstrap>' gateway run`
and `activate_dependencies()` **replaces `sys.path`** with one committed uv environment -
`~/.hermes/installs/<id>/environments/795020dbf24343cd9c52df24723dca83/venv`. Of the uv
environments on CT460, that one alone was missing `mnemosyne` / `mnemosyne_hermes`, so sessions
logged `mnemosyne_batch helpers unavailable (No module named 'mnemosyne')` and
`Memory provider 'mnemosyne' is selected but reports unavailable`. Fix: install
`hermes_memory_provider` + `mnemosyne_hermes` + `mnemosyne` **together with the native dependencies**
(`fastembed`, `onnxruntime`, `numpy` + `numpy.libs`, `tokenizers`, `huggingface_hub`, `sqlite-vec`,
`loguru`, `mmh3`) into that committed venv. Verified live: `Memory provider 'mnemosyne' registered
(40 tools)` -> `activated` at 2026-10-10 22:01:19, again at 23:59 / 00:00 / 00:02; and
`hermes memory status` -> `Plugin: installed / Status: available`.
**2. `elevenlabs` missing from the live managed env (21:13 -> 21:24).** TTS was configured but the
package was absent from the managed runtime in use; reinstalled and verified by generating a real
clip.
**Trap: `~/.local/lib/python3.14/site-packages` is invisible to the gateway.** The first repair
attempt installed into the user site directory; with `ENABLE_USER_SITE=False` **and** a replaced
`sys.path`, the gateway never sees it. Install into the committed uv env that
`activate_dependencies()` selects - not the user site dir, not a sibling venv.
**Trap: `hermes memory status` can be green while the gateway is still broken.** It is a separate
process with its own runtime resolution. The gateway's own log lines
(`registered (40 tools)` / `activated` versus `reports unavailable`) are the authoritative check.
**Trap: this recurrence produced no cron failures at all** (2026-10-10: 174 `completed`, 0 `failed`).
It degraded the gateway's memory and TTS paths only. A green cron day does **not** mean the runtime
is healthy - and the 15-minute guard still does not check the gateway's committed env, so **guard
coverage remains open.**