wiki audit 2026-10-04: CT450 v0.21.5 update incident, mnemosyne vector-recall FTS-only repair + coverage refresh, managed runtimes 10->3, doctor vec0 caveat

This commit is contained in:
Tony0410
2026-10-04 00:15:35 +08:00
parent b9f69d3f75
commit 78048bdf0b
14 changed files with 241 additions and 49 deletions
+8 -7
View File
@@ -3,8 +3,8 @@ title: Hermes Agent
type: system
status: active
created: 2026-07-22
updated: 2026-10-03
verified_on: 2026-10-03
updated: 2026-10-04
verified_on: 2026-10-04
confidence: high
tags: [system, hermes, agent, llm]
sources: [live-hermes-version, live-config, live-skill-inventory]
@@ -16,16 +16,16 @@ sources: [live-hermes-version, live-config, live-skill-inventory]
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
## Current Status
- **Version:** v0.21.5+3837.gbfda74c (2026.9.24), upstream `27062c34` — verified 2026-09-30
- **Version:** v0.21.5+6169.gab86a20 (2026.9.24), upstream `ab86a20a` — verified live 2026-10-04. Same semver token as the earlier `v0.21.5+3837.gbfda74c` (`27062c34`), but a newer build: the install tree was rebuilt 2026-10-03 10:11 AWST.
- **Install method:** git
- **Python:** 3.11.15
- **Upstream status:** Update available (`hermes update`)
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
- **Upstream status:** tree rebuilt 2026-10-03 from upstream `ab86a20a`; update availability not re-checked since.
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
- **Managed runtimes:** 10 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv`. Dependency guard `hermes-runtime-deps-guard.sh` verifies `dotenv` + `mnemosyne` in every venv every 15 min — see [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **Managed runtimes:** 3 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv` (was 10 — the 2026-10-03 rebuild pruned superseded environments and created two new ones at 09:59/10:00; the third survives from 2026-09-28). `hermes-runtime-deps-guard.sh` runs every 15 min via `mnemosyne-watchdog.py` and now asserts `dotenv` + `mnemosyne` in the installer python *and every managed venv*, **plus** `fastembed` + `FASTEMBED_CACHE_PATH` + the local model dir in the bootstrap-selected venv — see [[incidents/2026-09-28-hermes-managed-runtime-deps]] and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
- **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions
- **Binary:** `/home/hermes/.local/bin/hermes`
- **Config:** `~/.hermes/config.yaml`
- **Profile:** `default` (`~/.hermes/profiles/default/`)
- **Profile:** `default` (top-level config `~/.hermes/config.yaml`; the only named sub-profile on this host is `ops` at `~/.hermes/profiles/ops/`)
## Location
- **Host:** Local Linux host (no containerized Hermes runtime confirmed)
@@ -131,6 +131,7 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
## Known Issues
- Managed-runtime venvs previously drifted on dependencies, breaking cron workers — now guarded every 15 min. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **Vector recall silently ran keyword-only until 2026-10-03 (repaired).** `fastembed` was missing from every uv-managed runtime and `FASTEMBED_CACHE_PATH` was unset, so the dense leg of `mnemosyne_recall` returned no candidates and recall degraded to FTS5 — with no error raised. Repaired 2026-10-03 and the guard extended (above). Live coverage 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%). See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
- **Four cron jobs cannot self-recover — three pinned to models that no longer exist upstream, plus a credential fault:** `1a8e90ea2271` → `qwen3.5:397b` retired 2026-09-25 (HTTP 410); `dca8482e4f76` → `gpt-5.6-terra` plan-gated (403 `MODEL_NOT_IN_PLAN`); `fad5a86979d4` + `0e753e9720fd` → `deepseek-v4-flash` absent from the NVIDIA live catalog (HTTP 400); and `06a0a1c8704f` + `1b23c042e17d` → the `cliprox/terra` provider *connection* auth expired (HTTP 401), now confirmed on 2 of the 3 jobs sharing that pin. Repinning to the `smart-route` alias is the durable fix for the model faults; the cliprox pair needs the OmniRoute connection re-authenticated first. See [[systems/scheduled-tasks]], [[systems/model-providers]] and [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- **The daily skills stock-take is not running.** Job `1a8e90ea2271` is blocked on the retired model above, so [[skills-index]] and the freshness audit are only refreshed when `wiki-maintain.sh` is run by hand (last done 2026-10-03). The script itself is healthy — only the scheduled wrapper is broken.
- **Telegram thread delivery failures — root-caused and fixed 2026-09-30.** A bare `deliver: "telegram"` resolves to the home DM (`1793951355`), where group forum topic IDs cannot exist; the send hard-failed and the message was lost. Two jobs were retargeted to `telegram:-1004321904721:102` and verified. Three jobs still use bare `telegram` targets. See [[systems/messaging-integrations]].
+3 -2
View File
@@ -3,8 +3,8 @@ title: Memory Backend
type: system
status: active
created: 2026-07-22
updated: 2026-08-01
verified_on: 2026-08-01
updated: 2026-10-04
verified_on: 2026-10-04
confidence: high
tags: [memory, mnemosyne, agent]
sources: [raw/configs/hermes-config-sanitized.txt]
@@ -43,6 +43,7 @@ Persist and recall agent memory across sessions.
## Known Issues
- Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods.
- The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]].
- **Vector recall degraded silently until 2026-10-03** — `fastembed` was missing from every uv-managed runtime, so `mnemosyne_recall` fell back to FTS5 keyword scoring with no error raised. Repaired 2026-10-03; the 15-min dependency guard now asserts `fastembed` and the model cache path. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
**Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]].
+25 -14
View File
@@ -3,8 +3,8 @@ title: Mnemosyne
type: system
status: active
created: 2026-07-22
updated: 2026-10-01
verified_on: 2026-10-01
updated: 2026-10-04
verified_on: 2026-10-04
confidence: high
tags: [memory, mnemosyne, backend]
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
@@ -33,25 +33,34 @@ Primary agent memory store for Hermes.
- **Embedding model:** `BAAI/bge-small-en-v1.5` (384 dims)
- **Embeddings via API:** false (local fastembed)
## Database Stats (2026-08-02)
## Database Stats (live, 2026-10-04)
| Table | Rows | Notes |
|---|---|---|
| `working_memory` | 1,476 | Active memories |
| `episodic_memory` | 458 | Consolidated summaries |
| `memory_embeddings` | 158 | Only working memories — **0 episodic** |
| `working_memory` | 752 | Active memories |
| `episodic_memory` | 1,039 | Consolidated summaries |
| `memory_embeddings` | 739 | Working-tier embeddings |
| `memory` (legacy) | 2,181 | Old table, not migrated |
| `vec_working` | 159 | Vector index rows |
| `vec_working` / `vec_working_rowids` | 739 | Working vector index |
| `vec_episodes` / `vec_episodes_rowids` | 1,035 | Episodic vector index |
| `vec_facts_rowids` | 0 | `memoria_facts` is intentionally left unvectorised |
| Shared surface | 0 | Empty — cross-agent sharing not functional |
**Embedding coverage (live, 2026-10-04):** 1,759/2,194 memories embedded (80.2%) — working **726/752 (96.5%)**, episodic **1,033/1,039 (99.4%)**; 13 embeddings generated in the previous 24h. Working coverage is above the 95% alert threshold enforced by the Embedding Coverage job (`b63ed5817212`, every 6h, Telegram topic 102).
## Known Issues
### Vector embedding coverage gap
- **Only 159/1,476 working memories (10.8%)** have vector embeddings
- **0/458 episodic memories** have embeddings
- Semantic search (`mnemosyne_recall`) falls back to FTS5 keyword matching for ~90% of working and 100% of episodic memories
- This means recall quality is degraded — vector similarity can only search a tiny fraction of the memory store
- **Root cause:** The embedding pipeline (fastembed) never processed the bulk of memories. The `vec_working` table was rebuilt once (Jul 18) but only captured 159 rows.
- **Fix needed:** Backfill embeddings for all working and episodic memories. This requires running the embedding pipeline across the full dataset.
### Vector embedding coverage gap — RESOLVED (2026-08-02)
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
- **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only.
- Standard repair if coverage drops below 95% working: `~/.hermes/hermes-agent/venv/bin/mnemosyne reindex --yes` (the Embedding Coverage job `b63ed5817212` triggers the same path every 6h).
### Vector recall silently ran FTS-only (2026-10-03 — repaired)
- `fastembed` was absent from **every** uv-managed runtime, so the dense leg of `mnemosyne_recall` produced no candidates and recall fell back to FTS5/keyword scoring **without raising an error**. Sessions kept working, which is why it went unnoticed.
- Compounded by `FASTEMBED_CACHE_PATH` being unset in `~/.hermes/.env`, and by the embedding worker sometimes launching in isolated (`python -I`) mode, which does not inherit `PYTHONPATH`.
- **Repaired 2026-10-03:** `fastembed` installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min alongside its existing `dotenv`/`mnemosyne` checks.
- **CLI caveat — do not misread `mnemosyne doctor`.** Doctor and the recall CLI run under the legacy `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable; doctor therefore prints `vec0: present_but_unloadable` and marks the vector tiers `unavailable` even while the gateway embeds normally. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line.
- Benign noise: `pthread_setaffinity_np failed … mask: {0,}` from onnxruntime in `mcp-stderr.log` is a container affinity warning emitted on every model load — not an error.
- See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
### Profile isolation chosen over shared surface
|- The ops profile was enabled with mnemosyne on 2026-08-02
@@ -75,9 +84,11 @@ Primary agent memory store for Hermes.
- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
## History and Evidence
- **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
- **2026-09-30 memory-hygiene pass:** 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (`hermes_cron_model_default` → `custom:omni-101`, job-radar model). No unresolved conflicts left in the working tier.
- **2026-08-19 Mnemosyne Watchdog incident:** Watchdog job (`96eb95ce12f2`) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See [[systems/scheduled-tasks]] Known Issues section.
- **2026-08-18 incident:** Working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See [[incidents/2026-08-18-mnemosyne-capture-pipeline]].
+4 -3
View File
@@ -3,8 +3,8 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-03
verified_on: 2026-10-03
updated: 2026-10-04
verified_on: 2026-10-04
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -25,6 +25,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
- **Resolved since the last entry:** 1 — `1b23c042e17d`'s stale `delivery_failed` flag cleared exactly as predicted. Its 2026-10-02 run delivered to topic `102` (`delivery_outcome: delivered`); it then failed on the model pin, above.
- **Shared model pin at risk:** 3 enabled jobs run `cliprox/terra` — `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`. **Two of the three have now failed on that pin.** The third (`79f9bb41ecf5`) last ran 2026-09-27 — before the credential lapsed — and is next due 2026-10-04 18:00. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- **Streak recount (2026-10-03):** recounted directly from `~/.hermes/cron/executions.db`. `dca8482e4f76` 6 → 7 (one more daily run since), and `1a8e90ea2271` 8 → **7** — the ×8 carried in the 2026-10-02 entry was an overcount: that job has exactly 7 recorded runs and all 7 failed.
- **Re-verified 2026-10-04 00:12 AWST — no change:** still 30 jobs (27 enabled / 3 disabled) and the same **6** failing jobs; no new failure event landed inside the audit window. `79f9bb41ecf5` (Weekly Review, the last untested `cliprox/terra` job) drops at **2026-10-04 18:00 AWST** — its result settles whether that pin is 3/3 broken. `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found; delivered without thread_id`), confirmed live in `hermes cron list`.
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03)
## Job Table
@@ -87,7 +88,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
### Open
- **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 10-02, streak 7):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change; it cannot self-recover. See [[runbooks/job-radar]].
- **`cliprox/terra` provider connection auth expired — 2 of 3 jobs confirmed (streak 1 each):** Jobs `06a0a1c8704f` (2026-10-01) and `1b23c042e17d` (2026-10-02) both failed with `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Failure notices reached their topics; no output was produced. This is a *credential/connection* problem at the proxy, not a missing model — it will not self-recover until the connection is re-authenticated. `79f9bb41ecf5` is the remaining untested job on this pin (next run 2026-10-04 18:00). Re-authenticate the OmniRoute connection before repinning anything. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- **`cliprox/terra` provider connection auth expired — 2 of 3 jobs confirmed (streak 1 each):** Jobs `06a0a1c8704f` (2026-10-01) and `1b23c042e17d` (2026-10-02) both failed with `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Failure notices reached their topics; no output was produced. This is a *credential/connection* problem at the proxy, not a missing model — it will not self-recover until the connection is re-authenticated. `79f9bb41ecf5` is the remaining untested job on this pin (next run 2026-10-04 18:00 — re-check on 2026-10-05). Re-authenticate the OmniRoute connection before repinning anything. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- **A delivered notice is not a successful job:** both `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output.
- **wiki-stocktake-maintenance — HTTP 410, model retired (streak 7):** Job `1a8e90ea2271` is pinned to `ollama-cloud / qwen3.5:397b`, which Ollama retired on 2026-09-25 (`qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT`). It cannot self-recover — it needs a new model pin. **The job's script is fine:** `wiki-maintain.sh` was run by hand on 2026-10-03 and completed cleanly, regenerating [[systems/skills-index]] (344 skills / 70 categories) and refreshing the freshness audit. Only the scheduled wrapper is broken — so every non-model step of the daily stocktake needs a manual run until it is repinned.
- **Facebook Birthday scrapes — HTTP 400, model missing from catalog (streak 3 / 1):** Job `fad5a86979d4` (weekly) is pinned to `nvidia/deepseek-ai/deepseek-v4-flash`; the provider reports the model is `not available in the active live catalog for provider 'nvidia'`. **Confirmed as a shared-pin issue on 2026-10-01**, when sibling job `0e753e9720fd` (monthly) failed on the same pin at its own 09:00 run. Safest durable fix is repinning both to a routing alias (`smart-route`) instead of a raw model name.