wiki: nightly audit 2026-11-05 window — network-map re-verified by live probe (9377=Camofox not Stratos, 8790/5174 gone, +9120/9222/7777/6767/8085/9900, full Tailscale Serve table, two dead routes 8769/9123); commit pending 11-04 stocktake re-verifications; log + skills-index

This commit is contained in:
hermes
2026-11-05 01:22:45 +08:00
parent e4086ec18e
commit 3b5daccb4d
15 changed files with 942 additions and 552 deletions
@@ -3,8 +3,8 @@ title: Hermes Managed-Runtime Dependency Failure
type: incident
status: resolved
created: 2026-09-30
updated: 2026-10-11
verified_on: 2026-10-11
updated: 2026-11-04
verified_on: 2026-11-04
confidence: high
tags: [hermes, cron, runtime, python, worker, incident]
sources: [live-jobs-json, live-guard-script, live-cron-list]
@@ -36,6 +36,7 @@ Hermes migrated to **uv-managed runtimes**. Each runtime gets its own interprete
```
~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv # 10 venvs at time of incident
# (now **3** — see 2026-11-04 re-verification below)
```
The migration did not carry every dependency into every managed venv. Workers launched against a venv missing `dotenv` and `mnemosyne` died during import — *before* acknowledging ownership of the job. The failure therefore surfaced as a scheduler/dispatch bug while the real cause was:
@@ -71,6 +72,33 @@ Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the
- Stale `*.dist-info` directories make `uv` skip a package as "already installed" even when the module is not importable.
- `write_file` silently no-ops on `.hermes/scripts/` paths (Hermes library bug). Stage the script in `/tmp` and `install` it into place.
## Re-verification (2026-11-04)
**The incident remains resolved, but the estate's runtime topology has changed since 2026-09-30 and the page's
"10 venvs" figures are now stale.** Live state on CT460:
| Item | 2026-09-30 (page) | 2026-11-04 (live) |
|---|---|---|
| Managed venvs under `~/.hermes/installs/*/environments/*/venv` | **10** | **3** — `19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`, `f51d6e4902414114974145d8041c79cb` |
| Total runtimes the guard checks | 10 | **4** (3 managed venvs + the tools installer python) |
| Guard result | 10/10 OK | **`hermes-runtime-deps-guard: 4/4 runtimes OK`** (exit 0) |
| Bootstrap-selected runtime | n/a | **`f51d6e4902414114974145d8041c79cb`** — confirmed by running `hermes_bootstrap` under the tools python: `sys.path` → `…/f51d6e4902414114974145d8041c79cb/venv/lib/python3.14/site-packages` |
| `fastembed` in the bootstrap-selected venv | repaired | **OK** — guard log: `fastembed OK in f51d6e4902414114974145d8041c79cb` |
| Fastembed model cache | n/a | **present** — `~/.hermes/cache/fastembed/models--qdrant--bge-small-en-v1.5-onnx-q` |
| Old gateway committed env `795020dbf24343cd9c52df24723dca83` | the one that broke on 2026-10-10 | **no longer exists** — the `environments/` directory now holds only the three venvs above |
The guard script itself is unchanged and still wired in: `~/.hermes/scripts/hermes-runtime-deps-guard.sh` exists
(executable, 6479 bytes) and is invoked by `~/.hermes/scripts/mnemosyne-watchdog.py` at line 62.
**Two notes for the next time this class recurs.** First, the "10 venvs" figure is not a stable invariant —
Hermes prunes and re-creates uv environments, so **count the live venvs rather than trusting a recorded number**;
the guard's own `4/4 runtimes OK` line is the authoritative reading. Second, the **gateway's committed env is no
longer a separate directory** — the `795020dbf…` env from the 2026-10-10 recurrence has been garbage-collected,
and `hermes_bootstrap` now selects `f51d6e49…` (one of the three surviving managed venvs). The 2026-10-10
failure mode ("the gateway loads an env the guard does not check") may therefore no longer reproduce in that exact
shape, but the underlying trap — *a `sys.path` that points somewhere the guard never inspects* — is unchanged, so
the preventive rule below still stands.
## Verification (2026-09-30)
| Check | Result |
@@ -83,7 +111,7 @@ Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the
## Preventive rules
- Treat "worker exited before ownership acknowledgement" as a **dependency/import** failure first. Read worker stderr before touching provider or model pins.
- After any Hermes runtime or Python upgrade, re-run the guard and confirm 10/10 managed venvs import `dotenv` and `mnemosyne`.
- After any Hermes runtime or Python upgrade, re-run the guard and confirm **every runtime it reports** imports `dotenv` and `mnemosyne` — read the guard's own `N/N runtimes OK` line rather than a recorded venv count, because the number of managed venvs changes as Hermes prunes and re-creates uv environments (10 on 2026-09-30, 3 on 2026-11-04).
- Mass-cron-failure triage order: live interpreter imports → worker stderr → runtime layout → job logic. See [[runbooks/restart-hermes]] and [[systems/scheduled-tasks]].
## Related