wiki audit 2026-10-04: CT450 v0.21.5 update incident, mnemosyne vector-recall FTS-only repair + coverage refresh, managed runtimes 10->3, doctor vec0 caveat

This commit is contained in:
Tony0410
2026-10-04 00:15:35 +08:00
parent b9f69d3f75
commit 78048bdf0b
14 changed files with 241 additions and 49 deletions
+24
View File
@@ -298,3 +298,27 @@
- **`systems/skills-index.md` regenerated 341 → 344** — including a new skill, `fleet-wide-failure-triage` (devops). This was done **by hand**: job `1a8e90ea2271` is still blocked on HTTP 410 (retired `qwen3.5:397b`) and cannot self-recover. `wiki-maintain.sh` itself is healthy and was run manually, also writing `raw/stocktake/freshness-20261003.txt` — recorded so the next reader knows the stocktake is on manual refresh, not on schedule.
- **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live under `wiki/`; `index.md` sits at the vault root.
- No secrets written; all credential-shaped values in config remain redacted.
## [2026-10-04] Nightly audit — CT450 update + Mnemosyne vector-recall repair
Reviewed window: 2026-10-03 00:12 → 2026-10-04 00:12 AWST.
**New pages**
- `incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md` — `mnemosyne_recall` had been scoring keyword-only: `fastembed` was absent from every uv-managed runtime, so the dense leg produced no candidates and recall fell back to FTS5 *without raising an error*. Repaired 2026-10-03; the runtime guard now asserts fastembed + `FASTEMBED_CACHE_PATH` + model dir every 15 min. Live 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%).
- `incidents/2026-10-03-ct450-hermes-update.md` — CT450 (`hermesagent`) updated v0.19.0 → v0.21.5, its first update since July. Records the no-SSH access path (`ssh root@100.96.100.82` → `pct exec 450 -- …`) and the four pre-flight checks (clang, stale locks, tree ownership, Caddy keyring perms).
**Updated**
- `current-state.md` — new Recent Changes (2026-10-04) block: the two findings above, the managed-runtime prune 10 → 3, cron re-verification, Mattermost unreachable, and the `mnemosyne doctor` / `sqlite_vec` misread gotcha.
- `systems/mnemosyne.md` — Database Stats refreshed to live 2026-10-04 numbers; the 2026-08-02 "vector coverage gap" marked RESOLVED (retired as a current issue); new Known Issues subsection for the FTS-only degradation plus the doctor caveat; History entry; embedding-coverage job `b63ed5817212` documented.
- `systems/hermes-agent.md` — version corrected to `v0.21.5+6169.gab86a20` (was `v0.21.5+3837.gbfda74c`), managed-runtime count corrected 10 → 3, guard description extended, profile line corrected (no `profiles/default/`; `ops` is the only named sub-profile), Known Issues bullet added.
- `systems/memory-backend.md` — Known Issues bullet for the FTS-only degradation.
- `systems/scheduled-tasks.md` — 2026-10-04 re-verification (30 jobs, failing set still 6, no new failures in the window), `79f9bb41ecf5` due 2026-10-04 18:00; bare-`telegram` delivery warnings confirmed live.
- `agents/hermes-production.md` — version, Python, profile, provider routing, cron count and messaging refreshed; stale `custom:litellm` / `freellmapi-auto` references replaced.
- `agents/hermes-chaos.md` — CT450 version bump recorded, no-SSH access path added, Mattermost failure noted.
- `infrastructure/hosts.md` — CT450 row updated to v0.21.5.
- `runbooks/update-hermes-safely.md` — new "Second instance — CT450" section (access path + pre-flight checks).
- `runbooks/provider-health.md` — stale v0.19.0 version reference corrected.
- `index.md` — last-updated bumped; both new incidents linked; cron-fleet line re-dated.
**Note** — `mnemosyne doctor` on this host prints `vec0: present_but_unloadable` and marks the vector tiers unavailable. That is a Python-3.11-legacy-`venv` artefact, not the gateway's state. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`.