wiki audit 2026-10-04: CT450 v0.21.5 update incident, mnemosyne vector-recall FTS-only repair + coverage refresh, managed runtimes 10->3, doctor vec0 caveat

This commit is contained in:
Tony0410
2026-10-04 00:15:35 +08:00
parent b9f69d3f75
commit 78048bdf0b
14 changed files with 241 additions and 49 deletions
+5 -2
View File
@@ -3,8 +3,8 @@ title: Hermes Chaos
type: system
status: active
created: 2026-08-15
updated: 2026-08-17
verified_on: 2026-08-17
updated: 2026-10-04
verified_on: 2026-10-04
confidence: high
tags: [hermes, chaos, ct450, experimental, agent]
sources: [live-probe, config-inspection]
@@ -19,6 +19,7 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod
## Location
- **Container:** LXC CT450 (`hermesagent`)
- **Access:** **no SSH** — go through the PVE host: `ssh root@100.96.100.82`, then `pct exec 450 -- <cmd>` (or `pct enter 450`). `hermes` is not on `pct exec`'s default PATH; use `su - hermes -c "/home/hermes/.local/bin/hermes …"`.
- **Tailscale IP:** `100.98.205.112`
- **Hostname:** `hermesagent`
- **Software:** hermes-ui (Vite web app for browser use, port 5174)
@@ -27,6 +28,8 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod
## Current State
- **Status:** Active (experimental)
- **Hermes version:** `v0.21.5+6170.ga510d6c` (2026.9.24), upstream `a510d6c6` — **updated from v0.19.0 on 2026-10-03**, first update since July; 3 commits behind upstream. Python 3.14.7. See [[incidents/2026-10-03-ct450-hermes-update]].
- **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]].
- **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01)
- **Duplicate cron jobs:** 11 paused (no post-pause execution records since 2026-08-01)
- **Production counterparts:** CT460 has 11 enabled jobs with successful latest runs
+12 -10
View File
@@ -3,8 +3,8 @@ title: Hermes Production
type: system
status: active
created: 2026-08-15
updated: 2026-08-15
verified_on: 2026-08-15
updated: 2026-10-04
verified_on: 2026-10-04
confidence: high
tags: [hermes, production, ct460, agent]
sources: [live-probe, config-inspection]
@@ -27,11 +27,11 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
## Current State
- **Status:** Active
- **Hermes version:** v0.19.0
- **Python:** 3.11.15
- **Hermes version:** `v0.21.5+6169.gab86a20` (2026.9.24), upstream `ab86a20a`; install tree rebuilt 2026-10-03 10:11 AWST — verified live 2026-10-04
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
- **Config:** `~/.hermes/config.yaml`
- **Profile:** `default` (`~/.hermes/profiles/default/`)
- **Profile:** `default` — top-level config; **no `profiles/default/` directory exists** (`ops` at `~/.hermes/profiles/ops/` is the only named sub-profile)
- **Soul:** Rhino — trickster-mentor in a green jacket 🎩
## Profiles
@@ -43,9 +43,10 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
## Providers & Routing
- **Default provider:** `custom:litellm` → `http://litellm:4000/v1`
- **Fallback provider:** `custom:omniroute` → `http://omniroute:20128/v1`
- **Default model:** `freellmapi-auto`
- **Default provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — as attested 2026-09-30; the `custom:omni` provider key no longer exists
- **Default model:** `smart-route` (previously `freellmapi-auto`)
- **Fallback providers:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`, `custom / litellm:4000`
- **Auxiliary curator model:** `openai-codex / gpt-5.6-terra`
- See [[systems/model-providers]] for full breakdown.
## Memory
@@ -57,7 +58,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
## Scheduled Tasks
- **Cron jobs:** 25+ active jobs
- **Cron jobs:** 30 (27 enabled / 3 disabled; **6 failing** as of 2026-10-04)
- Delivery targets: Telegram (primary), Discord, origin, local
- Notable: Birthday checker, job radar, newsletter digests, mnemosyne health/watchdog/backup
- See [[systems/scheduled-tasks]] for full table.
@@ -65,6 +66,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
## Messaging
- Telegram, Discord, Slack, Email, Mattermost, ntfy
- **Mattermost is currently unreachable** — the gateway logs `mattermost connect timed out after 30s, next retry in 300s` (live, 2026-10-04) and the Mattermost host does not answer. Treat Mattermost delivery as down until probed.
- See [[systems/messaging-integrations]] for details.
## Key Differences from Chaos
@@ -72,7 +74,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
| Aspect | Production (CT460) | Chaos (CT450) |
|--------|-------------------|---------------|
| Purpose | Stable daily operations | Experimental/testing |
| Cron jobs | 25+ enabled | 11 paused duplicates |
| Cron jobs | 30 (27 enabled / 3 disabled) | 11 paused duplicates |
| Config changes | Requires verification | Safe to experiment |
| Memory | Production Mnemosyne | Separate state |
+11 -3
View File
@@ -3,8 +3,8 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-10-03
verified_on: 2026-10-03
updated: 2026-10-04
verified_on: 2026-10-04
review_after: 2026-10-31
confidence: high
tags: [system]
@@ -12,7 +12,15 @@ sources: [live-config, live-cron-list, live-hermes-version]
---
# Current State
Updated: 2026-10-03
Updated: 2026-10-04
## Recent Changes (2026-10-04)
- **Both Hermes instances updated on 2026-10-03, from opposite directions.** CT460 (production) was rebuilt to `v0.21.5+6169.gab86a20` (upstream `ab86a20a`, tree mtime 10:11 AWST) and CT450 (chaos) came up from `v0.19.0` to `v0.21.5+6170.ga510d6c` (upstream `a510d6c6`, tree mtime 10:22 AWST). Same semver token, different builds, separate state. CT450 now reports `Update available: 3 commits behind`. CT450 has no SSH — it is reached via `ssh root@100.96.100.82` + `pct exec 450`, and `hermes` is not on the default `pct exec` PATH. See [[incidents/2026-10-03-ct450-hermes-update]].
- **Vector recall had been silently running keyword-only — found and fixed 2026-10-03.** `fastembed` was missing from every uv-managed runtime, so the dense leg of `mnemosyne_recall` contributed no candidates and recall fell back to FTS5 **without erroring**. Fixed the same day: fastembed 0.8.1 installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min. **Live coverage 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%)**, 13 embeddings generated in the previous 24h. The previous "coverage gap" entry (159/1,476) is retired — it was resolved by the 2026-08-02 reindex. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] and [[systems/mnemosyne]].
- **Managed runtimes pruned 10 → 3.** Only 3 uv-managed venvs remain under `~/.hermes/installs/019d0d114d22b7e6/environments/` (two created 2026-10-03 09:59/10:00, one surviving from 2026-09-28). Any note that says "ten managed venvs" is stale.
- **Cron fleet re-verified 2026-10-04 00:12 AWST:** 30 jobs (27 enabled / 3 disabled), failing set unchanged at 6, no new failure inside the window. `79f9bb41ecf5` — the last untested `cliprox/terra` job — is due 2026-10-04 18:00 AWST. See [[systems/scheduled-tasks]].
- **Mattermost is unreachable from CT460 right now** (`mattermost connect timed out after 30s, next retry in 300s`, every 5 min, live 2026-10-04). Host-side condition; Mattermost delivery should be treated as down until probed.
- **Wiki maintenance gotcha worth remembering:** `mnemosyne doctor` runs under the legacy Python 3.11 `venv/`, where `sqlite_vec` is not importable, so it prints `vec0: present_but_unloadable` and marks vector tiers `unavailable` *even when the gateway is embedding fine*. Judge vector health with `check_embedding_coverage.py`, never with doctor's `vec0` line.
## Recent Changes (2026-10-03)
- **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set stays at **6**, but the composition changed: `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) failed on 2026-10-02 with the *identical* HTTP 401 the cliprox incident predicted — `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. That makes **2 of the 3** `cliprox/terra` jobs confirmed broken, so the provider-connection auth expiry is a real credential fault, not a one-job fluke. `79f9bb41ecf5` (Weekly Review, Sun 18:00) is the last untested one — its only run (2026-09-27) predates the expiry and succeeded; next due **2026-10-04 18:00**. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
@@ -0,0 +1,59 @@
---
title: CT450 Hermes Update to v0.21.5
type: incident
status: resolved
created: 2026-10-04
updated: 2026-10-04
verified_on: 2026-10-04
confidence: high
tags: [hermes, ct450, chaos, update, incident]
sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
---
# CT450 Hermes Update to v0.21.5
## Summary
The Chaos instance (LXC CT450, `hermesagent`) was updated **v0.19.0 → v0.21.5** on 2026-10-03 — its first update since July. CT460 (production) was rebuilt the same morning, so both instances now carry the same semver token from separate builds.
## Result (live-verified 2026-10-04)
| | CT460 (production) | CT450 (chaos) |
|---|---|---|
| Version | `v0.21.5+6169.gab86a20` (2026.9.24) | `v0.21.5+6170.ga510d6c` (2026.9.24) |
| Upstream | `ab86a20a` | `a510d6c6` — 3 commits behind, `hermes update` available |
| Python | managed runtime 3.14 (legacy `venv/` 3.11.15) | 3.14.7 |
| Install tree mtime | 2026-10-03 10:11 AWST | 2026-10-03 10:22 AWST |
## Access
**CT450 has no SSH.** Reach it through the PVE host:
```bash
ssh root@100.96.100.82
pct exec 450 -- <command> # or: pct enter 450
pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"
```
Two gotchas that cost time:
- `pct exec 450 -- hermes ...` **fails** with `Failed to exec "hermes"` — the binary is not on `lxc-attach`'s default PATH. Use the absolute path, or `su - hermes -c` (the dash matters: only a login shell puts `~/.local/bin` on PATH).
- A `hermes` login shell prints a `hermes-setup` banner ahead of command output; parse with `| tail -n` rather than expecting clean stdout.
## Pre-flight checks (each cost one failed run on 2026-10-03)
1. **Build toolchain.** `clang` must be present *before* updating — source-compiled Python dependencies will not build without it. Verified present on CT450 (`/usr/bin/clang`, 2026-10-04).
2. **Stale update locks.** A leftover `.install.lock` / `.lazy-refresh-incomplete` in the Hermes home makes the updater refuse or silently skip the refresh step. Neither exists on CT450 now.
3. **Install-tree ownership.** `~/.hermes/hermes-agent` must stay owned by `hermes:hermes` (uid 1000); a root-owned tree left by an earlier root-run update fails the in-place build.
4. **Caddy keyring permissions.** Host-level Caddy/`keyring` file permissions (`0600`) blocked a step mid-update — check them before starting, not during.
## Verification
- `hermes --version` on CT450 → `Hermes Agent v0.21.5+6170.ga510d6c (2026.9.24) · upstream a510d6c6`, Python 3.14.7.
- Install tree mtime `2026-10-03 10:22` matches the update window.
- `clang` present; no stale locks present.
- CT460 production untouched — separate config, skills, cron and memory state.
## Follow-ups
- CT450 reports `Update available: 3 commits behind`. Harmless; re-run `hermes update` if the chaos box should track upstream exactly.
- CT450's Mattermost connection is failing (`mattermost connect timed out after 30s`, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See [[agents/hermes-chaos]].
## Related
- [[agents/hermes-chaos]]
- [[agents/hermes-production]]
- [[runbooks/update-hermes-safely]]
- [[infrastructure/hosts]]
@@ -0,0 +1,60 @@
---
title: Mnemosyne Vector Recall Ran FTS-Only
type: incident
status: resolved
created: 2026-10-04
updated: 2026-10-04
verified_on: 2026-10-04
confidence: high
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
---
# Mnemosyne Vector Recall Ran FTS-Only
## Summary
Until 2026-10-03, `mnemosyne_recall` was scoring **keyword-only**. `fastembed` was missing from every UV-managed runtime, so the dense (vector) leg of recall contributed no candidates. Nothing errored — recall still returned results, just worse ones. Detected and repaired 2026-10-03; re-verified live 2026-10-04.
## Impact
- Semantic similarity search silently degraded to keyword/`FTS5` matching.
- No user-visible error, no failed cron job, no alert. The failure mode of a missing embedding backend is **quiet recall degradation**, not a crash — which is why it survived unnoticed.
- Nothing was lost: the residue is *un-embedded* rows, not missing rows (26 working / 6 episodic as of 2026-10-04). New writes embed normally.
## Root Cause
Three overlapping conditions:
1. **`fastembed` absent from the managed runtimes.** The gateway runs from a uv-managed venv under `~/.hermes/installs/<install>/environments/<hash>/venv`, and none of them had `fastembed`. The *legacy* `/home/hermes/.hermes/hermes-agent/venv` had it (0.8.0), which is why CLI-side checks looked healthy while the gateway could not embed.
2. **`FASTEMBED_CACHE_PATH` unset** in `~/.hermes/.env`, so the local model directory was resolved implicitly rather than pinned.
3. **Isolated-mode launches.** The embedding worker can be launched with `python -I`, which ignores `PYTHONPATH`/`PYTHONHOME` — any path-based injection of the dependency does not survive that launch mode.
## Fix (2026-10-03)
- Installed `fastembed` (0.8.1) into the runtime the bootstrap actually selects.
- Pinned `FASTEMBED_CACHE_PATH` (models cached under `~/.hermes/cache/fastembed`, `BAAI/bge-small-en-v1.5`).
- Extended `~/.hermes/scripts/hermes-runtime-deps-guard.sh` — run every 15 min by `mnemosyne-watchdog.py` — so it now asserts, in the bootstrap-selected venv: `fastembed` importable, `FASTEMBED_CACHE_PATH` set, and the model directory present, on top of its existing `dotenv` + `mnemosyne` checks in the installer python and every managed venv.
- No gateway restart required.
## Verification (live, 2026-10-04)
```
Memories with embeddings: 1759 / 2194 (80.2%)
Working memory: 726 / 752 (96.5%)
Episodic memory: 1033 / 1039 (99.4%)
vec_episodes_rowids: 1035
vec_working_rowids: 739
Embeddings generated (24h): 13 (working: 7, episodic: 6)
```
`onnxruntime` threads now appear in `~/.hermes/logs/mcp-stderr.log` on each model load — direct evidence the embedding model is executing inside the gateway runtime. The dependency guard exits 0.
## Caveat — do not misread `mnemosyne doctor`
`mnemosyne doctor` and the recall CLI run under the **legacy** `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable. Doctor therefore prints `vec0: present_but_unloadable` and marks the working/episodic vector tiers `unavailable`, even while the gateway embeds normally. The managed Python 3.14 venvs cannot import `sqlite_vec` either when probed directly. **Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line.**
## Benign noise
`[E:onnxruntime:Default, env.cc:228 ThreadMain] pthread_setaffinity_np failed for thread: … mask: {0,}` — a container CPU-affinity warning emitted on every model load. Not an error; do not chase it.
## Follow-ups
- Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.
- Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.
## Related
- [[systems/mnemosyne]]
- [[systems/memory-backend]]
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]]
+4 -2
View File
@@ -1,6 +1,6 @@
# Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-10-03
> model routing and automation. Last updated: 2026-10-04
## Start here
- [[current-state]] — Current systems status
@@ -28,10 +28,12 @@
### Integrations
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; 6 failing as of 2026-10-03)
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; 6 failing as of 2026-10-04)
- [[systems/skills-index]] — Installed skills inventory (auto-regenerated daily; being refreshed by hand while the stocktake job is blocked)
## Incidents
- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
- [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired; **2 of 3 jobs sharing the pin now failed**, the third untested (open 2026-10-01)
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26)
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min
+2 -2
View File
@@ -3,7 +3,7 @@ title: Known Hosts and Paths
type: infrastructure
status: partial
created: 2026-07-22
updated: 2026-08-14
updated: 2026-10-04
verified_on: 2026-07-22
confidence: medium
tags: [infrastructure, hosts, containers, paths]
@@ -21,7 +21,7 @@ sources: []
| omni | OmniRoute proxy | `http://omni:20128/v1` | Yes (from config, 2026-08-13) |
| CT201 | Steel Browser, Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.166` (LAN) | Yes (live probe) |
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net` | Yes |
| CT450 (hermesagent) | Hermes Agent v0.19.0, hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes |
| CT450 (hermesagent) | Hermes Agent **v0.21.5+6170.ga510d6c** (updated 2026-10-03 from v0.19.0), hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes |
## Project Paths
+24
View File
@@ -298,3 +298,27 @@
- **`systems/skills-index.md` regenerated 341 → 344** — including a new skill, `fleet-wide-failure-triage` (devops). This was done **by hand**: job `1a8e90ea2271` is still blocked on HTTP 410 (retired `qwen3.5:397b`) and cannot self-recover. `wiki-maintain.sh` itself is healthy and was run manually, also writing `raw/stocktake/freshness-20261003.txt` — recorded so the next reader knows the stocktake is on manual refresh, not on schedule.
- **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live under `wiki/`; `index.md` sits at the vault root.
- No secrets written; all credential-shaped values in config remain redacted.
## [2026-10-04] Nightly audit — CT450 update + Mnemosyne vector-recall repair
Reviewed window: 2026-10-03 00:12 → 2026-10-04 00:12 AWST.
**New pages**
- `incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md` — `mnemosyne_recall` had been scoring keyword-only: `fastembed` was absent from every uv-managed runtime, so the dense leg produced no candidates and recall fell back to FTS5 *without raising an error*. Repaired 2026-10-03; the runtime guard now asserts fastembed + `FASTEMBED_CACHE_PATH` + model dir every 15 min. Live 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%).
- `incidents/2026-10-03-ct450-hermes-update.md` — CT450 (`hermesagent`) updated v0.19.0 → v0.21.5, its first update since July. Records the no-SSH access path (`ssh root@100.96.100.82` → `pct exec 450 -- …`) and the four pre-flight checks (clang, stale locks, tree ownership, Caddy keyring perms).
**Updated**
- `current-state.md` — new Recent Changes (2026-10-04) block: the two findings above, the managed-runtime prune 10 → 3, cron re-verification, Mattermost unreachable, and the `mnemosyne doctor` / `sqlite_vec` misread gotcha.
- `systems/mnemosyne.md` — Database Stats refreshed to live 2026-10-04 numbers; the 2026-08-02 "vector coverage gap" marked RESOLVED (retired as a current issue); new Known Issues subsection for the FTS-only degradation plus the doctor caveat; History entry; embedding-coverage job `b63ed5817212` documented.
- `systems/hermes-agent.md` — version corrected to `v0.21.5+6169.gab86a20` (was `v0.21.5+3837.gbfda74c`), managed-runtime count corrected 10 → 3, guard description extended, profile line corrected (no `profiles/default/`; `ops` is the only named sub-profile), Known Issues bullet added.
- `systems/memory-backend.md` — Known Issues bullet for the FTS-only degradation.
- `systems/scheduled-tasks.md` — 2026-10-04 re-verification (30 jobs, failing set still 6, no new failures in the window), `79f9bb41ecf5` due 2026-10-04 18:00; bare-`telegram` delivery warnings confirmed live.
- `agents/hermes-production.md` — version, Python, profile, provider routing, cron count and messaging refreshed; stale `custom:litellm` / `freellmapi-auto` references replaced.
- `agents/hermes-chaos.md` — CT450 version bump recorded, no-SSH access path added, Mattermost failure noted.
- `infrastructure/hosts.md` — CT450 row updated to v0.21.5.
- `runbooks/update-hermes-safely.md` — new "Second instance — CT450" section (access path + pre-flight checks).
- `runbooks/provider-health.md` — stale v0.19.0 version reference corrected.
- `index.md` — last-updated bumped; both new incidents linked; cron-fleet line re-dated.
**Note** — `mnemosyne doctor` on this host prints `vec0: present_but_unloadable` and marks the vector tiers unavailable. That is a Python-3.11-legacy-`venv` artefact, not the gateway's state. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`.
+2 -2
View File
@@ -3,7 +3,7 @@ title: Provider Health Check
type: runbook
status: active
created: 2026-07-22
updated: 2026-07-22
updated: 2026-10-04
verified_on: 2026-07-22
last_tested: 2026-07-22
confidence: high
@@ -84,7 +84,7 @@ Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or
- Gateway journal shows no red alarm pattern.
## Current install references
- Hermes: v0.19.0, git install: `/home/hermes/.hermes/hermes-agent`
- Hermes: v0.21.5+6169.gab86a20 (2026.9.24), git install: `/home/hermes/.hermes/hermes-agent`
- LiteLLM base URL: `http://litellm:4000/v1`
- OmniRoute base URL: `http://omniroute:20128/v1`
- OmniRoute direct IP: `100.88.81.19:20128`
+22 -2
View File
@@ -3,7 +3,7 @@ title: Update Hermes safely
type: runbook
status: active
created: 2026-07-22
updated: 2026-07-22
updated: 2026-10-04
verified_on: 2026-07-22
last_tested: 2026-07-22
confidence: high
@@ -93,10 +93,30 @@ hermes dashboard restart
- If update warns about config migration, revert the backup and re-run manually
## Notes
- Hermes version on this install: v0.19.0 (2026.7.20)
- Hermes version on this install: v0.21.5+6169.gab86a20 (2026.9.24); install tree rebuilt 2026-10-03 10:11 AWST
- Install directory: `/home/hermes/.hermes/hermes-agent`
- Update preserves venv by default unless `--force-venv` is used
## Second instance — CT450 (`hermesagent`), updated 2026-10-03 (v0.19.0 → v0.21.5)
CT450 has **no SSH**. Reach it from the PVE host:
```bash
ssh root@100.96.100.82
pct exec 450 -- <command> # or: pct enter 450
pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"
```
`pct exec 450 -- hermes ...` fails (`Failed to exec "hermes"`): the binary is not on `lxc-attach`'s PATH. Use the absolute path or `su - hermes -c` (the dash gives a login shell, which sets `~/.local/bin`). A `hermes` login shell also prints a `hermes-setup` banner before output.
**Pre-flight before running the update** (each of these cost one failed run on 2026-10-03):
1. `clang` installed — source-compiled Python dependencies will not build without it (`pct exec 450 -- bash -lc "which clang"`).
2. No stale `.install.lock` / `.lazy-refresh-incomplete` in the Hermes home.
3. `~/.hermes/hermes-agent` owned by `hermes:hermes` (a root-owned tree from an earlier root-run update breaks the in-place build).
4. Host Caddy/`keyring` file permissions sane (`0600` keyring blocked a step mid-update).
**After:** `pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"`, confirm CT460 production is untouched (separate config/skills/cron/memory). Full record: [[incidents/2026-10-03-ct450-hermes-update]].
## Last tested
2026-07-22 on debian 13, git-based Hermes install
+8 -7
View File
@@ -3,8 +3,8 @@ title: Hermes Agent
type: system
status: active
created: 2026-07-22
updated: 2026-10-03
verified_on: 2026-10-03
updated: 2026-10-04
verified_on: 2026-10-04
confidence: high
tags: [system, hermes, agent, llm]
sources: [live-hermes-version, live-config, live-skill-inventory]
@@ -16,16 +16,16 @@ sources: [live-hermes-version, live-config, live-skill-inventory]
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
## Current Status
- **Version:** v0.21.5+3837.gbfda74c (2026.9.24), upstream `27062c34` — verified 2026-09-30
- **Version:** v0.21.5+6169.gab86a20 (2026.9.24), upstream `ab86a20a` — verified live 2026-10-04. Same semver token as the earlier `v0.21.5+3837.gbfda74c` (`27062c34`), but a newer build: the install tree was rebuilt 2026-10-03 10:11 AWST.
- **Install method:** git
- **Python:** 3.11.15
- **Upstream status:** Update available (`hermes update`)
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
- **Upstream status:** tree rebuilt 2026-10-03 from upstream `ab86a20a`; update availability not re-checked since.
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
- **Managed runtimes:** 10 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv`. Dependency guard `hermes-runtime-deps-guard.sh` verifies `dotenv` + `mnemosyne` in every venv every 15 min — see [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **Managed runtimes:** 3 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv` (was 10 — the 2026-10-03 rebuild pruned superseded environments and created two new ones at 09:59/10:00; the third survives from 2026-09-28). `hermes-runtime-deps-guard.sh` runs every 15 min via `mnemosyne-watchdog.py` and now asserts `dotenv` + `mnemosyne` in the installer python *and every managed venv*, **plus** `fastembed` + `FASTEMBED_CACHE_PATH` + the local model dir in the bootstrap-selected venv — see [[incidents/2026-09-28-hermes-managed-runtime-deps]] and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
- **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions
- **Binary:** `/home/hermes/.local/bin/hermes`
- **Config:** `~/.hermes/config.yaml`
- **Profile:** `default` (`~/.hermes/profiles/default/`)
- **Profile:** `default` (top-level config `~/.hermes/config.yaml`; the only named sub-profile on this host is `ops` at `~/.hermes/profiles/ops/`)
## Location
- **Host:** Local Linux host (no containerized Hermes runtime confirmed)
@@ -131,6 +131,7 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
## Known Issues
- Managed-runtime venvs previously drifted on dependencies, breaking cron workers — now guarded every 15 min. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **Vector recall silently ran keyword-only until 2026-10-03 (repaired).** `fastembed` was missing from every uv-managed runtime and `FASTEMBED_CACHE_PATH` was unset, so the dense leg of `mnemosyne_recall` returned no candidates and recall degraded to FTS5 — with no error raised. Repaired 2026-10-03 and the guard extended (above). Live coverage 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%). See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
- **Four cron jobs cannot self-recover — three pinned to models that no longer exist upstream, plus a credential fault:** `1a8e90ea2271` → `qwen3.5:397b` retired 2026-09-25 (HTTP 410); `dca8482e4f76` → `gpt-5.6-terra` plan-gated (403 `MODEL_NOT_IN_PLAN`); `fad5a86979d4` + `0e753e9720fd` → `deepseek-v4-flash` absent from the NVIDIA live catalog (HTTP 400); and `06a0a1c8704f` + `1b23c042e17d` → the `cliprox/terra` provider *connection* auth expired (HTTP 401), now confirmed on 2 of the 3 jobs sharing that pin. Repinning to the `smart-route` alias is the durable fix for the model faults; the cliprox pair needs the OmniRoute connection re-authenticated first. See [[systems/scheduled-tasks]], [[systems/model-providers]] and [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- **The daily skills stock-take is not running.** Job `1a8e90ea2271` is blocked on the retired model above, so [[skills-index]] and the freshness audit are only refreshed when `wiki-maintain.sh` is run by hand (last done 2026-10-03). The script itself is healthy — only the scheduled wrapper is broken.
- **Telegram thread delivery failures — root-caused and fixed 2026-09-30.** A bare `deliver: "telegram"` resolves to the home DM (`1793951355`), where group forum topic IDs cannot exist; the send hard-failed and the message was lost. Two jobs were retargeted to `telegram:-1004321904721:102` and verified. Three jobs still use bare `telegram` targets. See [[systems/messaging-integrations]].
+3 -2
View File
@@ -3,8 +3,8 @@ title: Memory Backend
type: system
status: active
created: 2026-07-22
updated: 2026-08-01
verified_on: 2026-08-01
updated: 2026-10-04
verified_on: 2026-10-04
confidence: high
tags: [memory, mnemosyne, agent]
sources: [raw/configs/hermes-config-sanitized.txt]
@@ -43,6 +43,7 @@ Persist and recall agent memory across sessions.
## Known Issues
- Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods.
- The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]].
- **Vector recall degraded silently until 2026-10-03** — `fastembed` was missing from every uv-managed runtime, so `mnemosyne_recall` fell back to FTS5 keyword scoring with no error raised. Repaired 2026-10-03; the 15-min dependency guard now asserts `fastembed` and the model cache path. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
**Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]].
+25 -14
View File
@@ -3,8 +3,8 @@ title: Mnemosyne
type: system
status: active
created: 2026-07-22
updated: 2026-10-01
verified_on: 2026-10-01
updated: 2026-10-04
verified_on: 2026-10-04
confidence: high
tags: [memory, mnemosyne, backend]
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
@@ -33,25 +33,34 @@ Primary agent memory store for Hermes.
- **Embedding model:** `BAAI/bge-small-en-v1.5` (384 dims)
- **Embeddings via API:** false (local fastembed)
## Database Stats (2026-08-02)
## Database Stats (live, 2026-10-04)
| Table | Rows | Notes |
|---|---|---|
| `working_memory` | 1,476 | Active memories |
| `episodic_memory` | 458 | Consolidated summaries |
| `memory_embeddings` | 158 | Only working memories — **0 episodic** |
| `working_memory` | 752 | Active memories |
| `episodic_memory` | 1,039 | Consolidated summaries |
| `memory_embeddings` | 739 | Working-tier embeddings |
| `memory` (legacy) | 2,181 | Old table, not migrated |
| `vec_working` | 159 | Vector index rows |
| `vec_working` / `vec_working_rowids` | 739 | Working vector index |
| `vec_episodes` / `vec_episodes_rowids` | 1,035 | Episodic vector index |
| `vec_facts_rowids` | 0 | `memoria_facts` is intentionally left unvectorised |
| Shared surface | 0 | Empty — cross-agent sharing not functional |
**Embedding coverage (live, 2026-10-04):** 1,759/2,194 memories embedded (80.2%) — working **726/752 (96.5%)**, episodic **1,033/1,039 (99.4%)**; 13 embeddings generated in the previous 24h. Working coverage is above the 95% alert threshold enforced by the Embedding Coverage job (`b63ed5817212`, every 6h, Telegram topic 102).
## Known Issues
### Vector embedding coverage gap
- **Only 159/1,476 working memories (10.8%)** have vector embeddings
- **0/458 episodic memories** have embeddings
- Semantic search (`mnemosyne_recall`) falls back to FTS5 keyword matching for ~90% of working and 100% of episodic memories
- This means recall quality is degraded — vector similarity can only search a tiny fraction of the memory store
- **Root cause:** The embedding pipeline (fastembed) never processed the bulk of memories. The `vec_working` table was rebuilt once (Jul 18) but only captured 159 rows.
- **Fix needed:** Backfill embeddings for all working and episodic memories. This requires running the embedding pipeline across the full dataset.
### Vector embedding coverage gap — RESOLVED (2026-08-02)
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
- **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only.
- Standard repair if coverage drops below 95% working: `~/.hermes/hermes-agent/venv/bin/mnemosyne reindex --yes` (the Embedding Coverage job `b63ed5817212` triggers the same path every 6h).
### Vector recall silently ran FTS-only (2026-10-03 — repaired)
- `fastembed` was absent from **every** uv-managed runtime, so the dense leg of `mnemosyne_recall` produced no candidates and recall fell back to FTS5/keyword scoring **without raising an error**. Sessions kept working, which is why it went unnoticed.
- Compounded by `FASTEMBED_CACHE_PATH` being unset in `~/.hermes/.env`, and by the embedding worker sometimes launching in isolated (`python -I`) mode, which does not inherit `PYTHONPATH`.
- **Repaired 2026-10-03:** `fastembed` installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min alongside its existing `dotenv`/`mnemosyne` checks.
- **CLI caveat — do not misread `mnemosyne doctor`.** Doctor and the recall CLI run under the legacy `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable; doctor therefore prints `vec0: present_but_unloadable` and marks the vector tiers `unavailable` even while the gateway embeds normally. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line.
- Benign noise: `pthread_setaffinity_np failed … mask: {0,}` from onnxruntime in `mcp-stderr.log` is a container affinity warning emitted on every model load — not an error.
- See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
### Profile isolation chosen over shared surface
|- The ops profile was enabled with mnemosyne on 2026-08-02
@@ -75,9 +84,11 @@ Primary agent memory store for Hermes.
- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
## History and Evidence
- **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
- **2026-09-30 memory-hygiene pass:** 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (`hermes_cron_model_default` → `custom:omni-101`, job-radar model). No unresolved conflicts left in the working tier.
- **2026-08-19 Mnemosyne Watchdog incident:** Watchdog job (`96eb95ce12f2`) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See [[systems/scheduled-tasks]] Known Issues section.
- **2026-08-18 incident:** Working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See [[incidents/2026-08-18-mnemosyne-capture-pipeline]].
+4 -3
View File
@@ -3,8 +3,8 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-03
verified_on: 2026-10-03
updated: 2026-10-04
verified_on: 2026-10-04
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -25,6 +25,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
- **Resolved since the last entry:** 1 — `1b23c042e17d`'s stale `delivery_failed` flag cleared exactly as predicted. Its 2026-10-02 run delivered to topic `102` (`delivery_outcome: delivered`); it then failed on the model pin, above.
- **Shared model pin at risk:** 3 enabled jobs run `cliprox/terra` — `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`. **Two of the three have now failed on that pin.** The third (`79f9bb41ecf5`) last ran 2026-09-27 — before the credential lapsed — and is next due 2026-10-04 18:00. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- **Streak recount (2026-10-03):** recounted directly from `~/.hermes/cron/executions.db`. `dca8482e4f76` 6 → 7 (one more daily run since), and `1a8e90ea2271` 8 → **7** — the ×8 carried in the 2026-10-02 entry was an overcount: that job has exactly 7 recorded runs and all 7 failed.
- **Re-verified 2026-10-04 00:12 AWST — no change:** still 30 jobs (27 enabled / 3 disabled) and the same **6** failing jobs; no new failure event landed inside the audit window. `79f9bb41ecf5` (Weekly Review, the last untested `cliprox/terra` job) drops at **2026-10-04 18:00 AWST** — its result settles whether that pin is 3/3 broken. `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found; delivered without thread_id`), confirmed live in `hermes cron list`.
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03)
## Job Table
@@ -87,7 +88,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
### Open
- **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 10-02, streak 7):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change; it cannot self-recover. See [[runbooks/job-radar]].
- **`cliprox/terra` provider connection auth expired — 2 of 3 jobs confirmed (streak 1 each):** Jobs `06a0a1c8704f` (2026-10-01) and `1b23c042e17d` (2026-10-02) both failed with `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Failure notices reached their topics; no output was produced. This is a *credential/connection* problem at the proxy, not a missing model — it will not self-recover until the connection is re-authenticated. `79f9bb41ecf5` is the remaining untested job on this pin (next run 2026-10-04 18:00). Re-authenticate the OmniRoute connection before repinning anything. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- **`cliprox/terra` provider connection auth expired — 2 of 3 jobs confirmed (streak 1 each):** Jobs `06a0a1c8704f` (2026-10-01) and `1b23c042e17d` (2026-10-02) both failed with `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Failure notices reached their topics; no output was produced. This is a *credential/connection* problem at the proxy, not a missing model — it will not self-recover until the connection is re-authenticated. `79f9bb41ecf5` is the remaining untested job on this pin (next run 2026-10-04 18:00 — re-check on 2026-10-05). Re-authenticate the OmniRoute connection before repinning anything. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- **A delivered notice is not a successful job:** both `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output.
- **wiki-stocktake-maintenance — HTTP 410, model retired (streak 7):** Job `1a8e90ea2271` is pinned to `ollama-cloud / qwen3.5:397b`, which Ollama retired on 2026-09-25 (`qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT`). It cannot self-recover — it needs a new model pin. **The job's script is fine:** `wiki-maintain.sh` was run by hand on 2026-10-03 and completed cleanly, regenerating [[systems/skills-index]] (344 skills / 70 categories) and refreshing the freshness audit. Only the scheduled wrapper is broken — so every non-model step of the daily stocktake needs a manual run until it is repinned.
- **Facebook Birthday scrapes — HTTP 400, model missing from catalog (streak 3 / 1):** Job `fad5a86979d4` (weekly) is pinned to `nvidia/deepseek-ai/deepseek-v4-flash`; the provider reports the model is `not available in the active live catalog for provider 'nvidia'`. **Confirmed as a shared-pin issue on 2026-10-01**, when sibling job `0e753e9720fd` (monthly) failed on the same pin at its own 09:00 run. Safest durable fix is repinning both to a routing alias (`smart-route`) instead of a raw model name.