wiki: nightly audit 2026-11-05 window — network-map re-verified by live probe (9377=Camofox not Stratos, 8790/5174 gone, +9120/9222/7777/6767/8085/9900, full Tailscale Serve table, two dead routes 8769/9123); commit pending 11-04 stocktake re-verifications; log + skills-index
This commit is contained in:
@@ -3,8 +3,8 @@ title: "Mnemosyne Capture Pipeline Broken (Aug 18)"
|
||||
type: incident
|
||||
status: superseded
|
||||
created: 2026-08-18
|
||||
updated: 2026-10-12
|
||||
verified_on: 2026-10-12
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, hermes, incident, capture]
|
||||
sources: [mnemosyne-db, health-monitor, run_agent.py]
|
||||
@@ -68,6 +68,29 @@ embedding coverage **98.3 % working / 99.6 % episodic**, FTS aligned (`fts_worki
|
||||
The three guard conditions diagnosed below were never individually confirmed as the fix — the path resumed
|
||||
capturing and has captured through every window since.
|
||||
|
||||
### Re-verified 2026-11-04 — capture still healthy, and the counts have moved on
|
||||
|
||||
Live state of `~/.hermes/mnemosyne/data/mnemosyne.db` (the path in the Verification snippet above is still
|
||||
correct; note the DB is under `data/`, not the `~/.hermes/mnemosyne/*.db` glob path):
|
||||
|
||||
| Metric | 2026-10-12 (page) | 2026-11-04 (live) |
|
||||
|---|---|---|
|
||||
| `working_memory` | 786 | **970** |
|
||||
| `episodic_memory` | 1,056 | **1,146** |
|
||||
| `memoria_facts` | — | **446** |
|
||||
| `memory_embeddings` | — | **943** |
|
||||
| Working-memory vector coverage | 98.3 % | **98.6 %** (`vec_working` 956 / `working_memory` 970) |
|
||||
| Episodic vector coverage | 99.6 % | **100 %** (`vec_episodes` 1,146 / `episodic_memory` 1,146) |
|
||||
| `fts_working` | 786 | **970** — aligned with `working_memory` |
|
||||
| `fts_facts` | 876 | **927** (vs `memoria_facts` 446 — FTS indexes a superset) |
|
||||
| Last `working_memory` entry | 2026-10-11 | **2026-11-03 17:12:40** |
|
||||
|
||||
`mnemosyne stats` independently reports **Total 2,562 / Working 970 / Episodic 1,146**, and the `vec0` extension
|
||||
loads cleanly (`sqlite-vec` from the agent venv), confirming vector search is live rather than silently FTS-only.
|
||||
**Working-memory coverage is above the 95 % reindex threshold**, so no `mnemosyne reindex` was triggered by this
|
||||
audit. **Capture has continued to work without incident for a full month** — the original failure has not
|
||||
recurred.
|
||||
|
||||
**Caution for the next alarm:** the health monitor raised a false EXCEPTION on 2026-10-11 00:08 by comparing
|
||||
naive-UTC `created_at` against AWST `now` ("stalled 10.7 h" — real gap 2.7 h). Do not triage that as a repeat
|
||||
of this incident; see [[systems/mnemosyne]].
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Hermes Managed-Runtime Dependency Failure
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-09-30
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
confidence: high
|
||||
tags: [hermes, cron, runtime, python, worker, incident]
|
||||
sources: [live-jobs-json, live-guard-script, live-cron-list]
|
||||
@@ -36,6 +36,7 @@ Hermes migrated to **uv-managed runtimes**. Each runtime gets its own interprete
|
||||
|
||||
```
|
||||
~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv # 10 venvs at time of incident
|
||||
# (now **3** — see 2026-11-04 re-verification below)
|
||||
```
|
||||
|
||||
The migration did not carry every dependency into every managed venv. Workers launched against a venv missing `dotenv` and `mnemosyne` died during import — *before* acknowledging ownership of the job. The failure therefore surfaced as a scheduler/dispatch bug while the real cause was:
|
||||
@@ -71,6 +72,33 @@ Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the
|
||||
- Stale `*.dist-info` directories make `uv` skip a package as "already installed" even when the module is not importable.
|
||||
- `write_file` silently no-ops on `.hermes/scripts/` paths (Hermes library bug). Stage the script in `/tmp` and `install` it into place.
|
||||
|
||||
## Re-verification (2026-11-04)
|
||||
|
||||
**The incident remains resolved, but the estate's runtime topology has changed since 2026-09-30 and the page's
|
||||
"10 venvs" figures are now stale.** Live state on CT460:
|
||||
|
||||
| Item | 2026-09-30 (page) | 2026-11-04 (live) |
|
||||
|---|---|---|
|
||||
| Managed venvs under `~/.hermes/installs/*/environments/*/venv` | **10** | **3** — `19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`, `f51d6e4902414114974145d8041c79cb` |
|
||||
| Total runtimes the guard checks | 10 | **4** (3 managed venvs + the tools installer python) |
|
||||
| Guard result | 10/10 OK | **`hermes-runtime-deps-guard: 4/4 runtimes OK`** (exit 0) |
|
||||
| Bootstrap-selected runtime | n/a | **`f51d6e4902414114974145d8041c79cb`** — confirmed by running `hermes_bootstrap` under the tools python: `sys.path` → `…/f51d6e4902414114974145d8041c79cb/venv/lib/python3.14/site-packages` |
|
||||
| `fastembed` in the bootstrap-selected venv | repaired | **OK** — guard log: `fastembed OK in f51d6e4902414114974145d8041c79cb` |
|
||||
| Fastembed model cache | n/a | **present** — `~/.hermes/cache/fastembed/models--qdrant--bge-small-en-v1.5-onnx-q` |
|
||||
| Old gateway committed env `795020dbf24343cd9c52df24723dca83` | the one that broke on 2026-10-10 | **no longer exists** — the `environments/` directory now holds only the three venvs above |
|
||||
|
||||
The guard script itself is unchanged and still wired in: `~/.hermes/scripts/hermes-runtime-deps-guard.sh` exists
|
||||
(executable, 6479 bytes) and is invoked by `~/.hermes/scripts/mnemosyne-watchdog.py` at line 62.
|
||||
|
||||
**Two notes for the next time this class recurs.** First, the "10 venvs" figure is not a stable invariant —
|
||||
Hermes prunes and re-creates uv environments, so **count the live venvs rather than trusting a recorded number**;
|
||||
the guard's own `4/4 runtimes OK` line is the authoritative reading. Second, the **gateway's committed env is no
|
||||
longer a separate directory** — the `795020dbf…` env from the 2026-10-10 recurrence has been garbage-collected,
|
||||
and `hermes_bootstrap` now selects `f51d6e49…` (one of the three surviving managed venvs). The 2026-10-10
|
||||
failure mode ("the gateway loads an env the guard does not check") may therefore no longer reproduce in that exact
|
||||
shape, but the underlying trap — *a `sys.path` that points somewhere the guard never inspects* — is unchanged, so
|
||||
the preventive rule below still stands.
|
||||
|
||||
## Verification (2026-09-30)
|
||||
|
||||
| Check | Result |
|
||||
@@ -83,7 +111,7 @@ Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the
|
||||
## Preventive rules
|
||||
|
||||
- Treat "worker exited before ownership acknowledgement" as a **dependency/import** failure first. Read worker stderr before touching provider or model pins.
|
||||
- After any Hermes runtime or Python upgrade, re-run the guard and confirm 10/10 managed venvs import `dotenv` and `mnemosyne`.
|
||||
- After any Hermes runtime or Python upgrade, re-run the guard and confirm **every runtime it reports** imports `dotenv` and `mnemosyne` — read the guard's own `N/N runtimes OK` line rather than a recorded venv count, because the number of managed venvs changes as Hermes prunes and re-creates uv environments (10 on 2026-09-30, 3 on 2026-11-04).
|
||||
- Mass-cron-failure triage order: live interpreter imports → worker stderr → runtime layout → job logic. See [[runbooks/restart-hermes]] and [[systems/scheduled-tasks]].
|
||||
|
||||
## Related
|
||||
|
||||
Reference in New Issue
Block a user