diff --git a/agents/hermes-chaos.md b/agents/hermes-chaos.md index 2791dfc..dd68c1f 100644 --- a/agents/hermes-chaos.md +++ b/agents/hermes-chaos.md @@ -3,8 +3,8 @@ title: Hermes Chaos type: system status: active created: 2026-08-15 -updated: 2026-08-17 -verified_on: 2026-08-17 +updated: 2026-10-04 +verified_on: 2026-10-04 confidence: high tags: [hermes, chaos, ct450, experimental, agent] sources: [live-probe, config-inspection] @@ -19,6 +19,7 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod ## Location - **Container:** LXC CT450 (`hermesagent`) +- **Access:** **no SSH** — go through the PVE host: `ssh root@100.96.100.82`, then `pct exec 450 -- ` (or `pct enter 450`). `hermes` is not on `pct exec`'s default PATH; use `su - hermes -c "/home/hermes/.local/bin/hermes …"`. - **Tailscale IP:** `100.98.205.112` - **Hostname:** `hermesagent` - **Software:** hermes-ui (Vite web app for browser use, port 5174) @@ -27,6 +28,8 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod ## Current State - **Status:** Active (experimental) +- **Hermes version:** `v0.21.5+6170.ga510d6c` (2026.9.24), upstream `a510d6c6` — **updated from v0.19.0 on 2026-10-03**, first update since July; 3 commits behind upstream. Python 3.14.7. See [[incidents/2026-10-03-ct450-hermes-update]]. +- **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]]. - **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01) - **Duplicate cron jobs:** 11 paused (no post-pause execution records since 2026-08-01) - **Production counterparts:** CT460 has 11 enabled jobs with successful latest runs diff --git a/agents/hermes-production.md b/agents/hermes-production.md index a085e18..a124395 100644 --- a/agents/hermes-production.md +++ b/agents/hermes-production.md @@ -3,8 +3,8 @@ title: Hermes Production type: system status: active created: 2026-08-15 -updated: 2026-08-15 -verified_on: 2026-08-15 +updated: 2026-10-04 +verified_on: 2026-10-04 confidence: high tags: [hermes, production, ct460, agent] sources: [live-probe, config-inspection] @@ -27,11 +27,11 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn ## Current State - **Status:** Active -- **Hermes version:** v0.19.0 -- **Python:** 3.11.15 +- **Hermes version:** `v0.21.5+6169.gab86a20` (2026.9.24), upstream `ab86a20a`; install tree rebuilt 2026-10-03 10:11 AWST — verified live 2026-10-04 +- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15) - **Install directory:** `/home/hermes/.hermes/hermes-agent` - **Config:** `~/.hermes/config.yaml` -- **Profile:** `default` (`~/.hermes/profiles/default/`) +- **Profile:** `default` — top-level config; **no `profiles/default/` directory exists** (`ops` at `~/.hermes/profiles/ops/` is the only named sub-profile) - **Soul:** Rhino — trickster-mentor in a green jacket 🎩 ## Profiles @@ -43,9 +43,10 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn ## Providers & Routing -- **Default provider:** `custom:litellm` → `http://litellm:4000/v1` -- **Fallback provider:** `custom:omniroute` → `http://omniroute:20128/v1` -- **Default model:** `freellmapi-auto` +- **Default provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — as attested 2026-09-30; the `custom:omni` provider key no longer exists +- **Default model:** `smart-route` (previously `freellmapi-auto`) +- **Fallback providers:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`, `custom / litellm:4000` +- **Auxiliary curator model:** `openai-codex / gpt-5.6-terra` - See [[systems/model-providers]] for full breakdown. ## Memory @@ -57,7 +58,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn ## Scheduled Tasks -- **Cron jobs:** 25+ active jobs +- **Cron jobs:** 30 (27 enabled / 3 disabled; **6 failing** as of 2026-10-04) - Delivery targets: Telegram (primary), Discord, origin, local - Notable: Birthday checker, job radar, newsletter digests, mnemosyne health/watchdog/backup - See [[systems/scheduled-tasks]] for full table. @@ -65,6 +66,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn ## Messaging - Telegram, Discord, Slack, Email, Mattermost, ntfy +- **Mattermost is currently unreachable** — the gateway logs `mattermost connect timed out after 30s, next retry in 300s` (live, 2026-10-04) and the Mattermost host does not answer. Treat Mattermost delivery as down until probed. - See [[systems/messaging-integrations]] for details. ## Key Differences from Chaos @@ -72,7 +74,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn | Aspect | Production (CT460) | Chaos (CT450) | |--------|-------------------|---------------| | Purpose | Stable daily operations | Experimental/testing | -| Cron jobs | 25+ enabled | 11 paused duplicates | +| Cron jobs | 30 (27 enabled / 3 disabled) | 11 paused duplicates | | Config changes | Requires verification | Safe to experiment | | Memory | Production Mnemosyne | Separate state | diff --git a/current-state.md b/current-state.md index e9e80a7..4313a4d 100644 --- a/current-state.md +++ b/current-state.md @@ -3,8 +3,8 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-03 -verified_on: 2026-10-03 +updated: 2026-10-04 +verified_on: 2026-10-04 review_after: 2026-10-31 confidence: high tags: [system] @@ -12,7 +12,15 @@ sources: [live-config, live-cron-list, live-hermes-version] --- # Current State -Updated: 2026-10-03 +Updated: 2026-10-04 + +## Recent Changes (2026-10-04) +- **Both Hermes instances updated on 2026-10-03, from opposite directions.** CT460 (production) was rebuilt to `v0.21.5+6169.gab86a20` (upstream `ab86a20a`, tree mtime 10:11 AWST) and CT450 (chaos) came up from `v0.19.0` to `v0.21.5+6170.ga510d6c` (upstream `a510d6c6`, tree mtime 10:22 AWST). Same semver token, different builds, separate state. CT450 now reports `Update available: 3 commits behind`. CT450 has no SSH — it is reached via `ssh root@100.96.100.82` + `pct exec 450`, and `hermes` is not on the default `pct exec` PATH. See [[incidents/2026-10-03-ct450-hermes-update]]. +- **Vector recall had been silently running keyword-only — found and fixed 2026-10-03.** `fastembed` was missing from every uv-managed runtime, so the dense leg of `mnemosyne_recall` contributed no candidates and recall fell back to FTS5 **without erroring**. Fixed the same day: fastembed 0.8.1 installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min. **Live coverage 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%)**, 13 embeddings generated in the previous 24h. The previous "coverage gap" entry (159/1,476) is retired — it was resolved by the 2026-08-02 reindex. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] and [[systems/mnemosyne]]. +- **Managed runtimes pruned 10 → 3.** Only 3 uv-managed venvs remain under `~/.hermes/installs/019d0d114d22b7e6/environments/` (two created 2026-10-03 09:59/10:00, one surviving from 2026-09-28). Any note that says "ten managed venvs" is stale. +- **Cron fleet re-verified 2026-10-04 00:12 AWST:** 30 jobs (27 enabled / 3 disabled), failing set unchanged at 6, no new failure inside the window. `79f9bb41ecf5` — the last untested `cliprox/terra` job — is due 2026-10-04 18:00 AWST. See [[systems/scheduled-tasks]]. +- **Mattermost is unreachable from CT460 right now** (`mattermost connect timed out after 30s, next retry in 300s`, every 5 min, live 2026-10-04). Host-side condition; Mattermost delivery should be treated as down until probed. +- **Wiki maintenance gotcha worth remembering:** `mnemosyne doctor` runs under the legacy Python 3.11 `venv/`, where `sqlite_vec` is not importable, so it prints `vec0: present_but_unloadable` and marks vector tiers `unavailable` *even when the gateway is embedding fine*. Judge vector health with `check_embedding_coverage.py`, never with doctor's `vec0` line. ## Recent Changes (2026-10-03) - **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set stays at **6**, but the composition changed: `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) failed on 2026-10-02 with the *identical* HTTP 401 the cliprox incident predicted — `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. That makes **2 of the 3** `cliprox/terra` jobs confirmed broken, so the provider-connection auth expiry is a real credential fault, not a one-job fluke. `79f9bb41ecf5` (Weekly Review, Sun 18:00) is the last untested one — its only run (2026-09-27) predates the expiry and succeeded; next due **2026-10-04 18:00**. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. diff --git a/incidents/2026-10-03-ct450-hermes-update.md b/incidents/2026-10-03-ct450-hermes-update.md new file mode 100644 index 0000000..44b919c --- /dev/null +++ b/incidents/2026-10-03-ct450-hermes-update.md @@ -0,0 +1,59 @@ +--- +title: CT450 Hermes Update to v0.21.5 +type: incident +status: resolved +created: 2026-10-04 +updated: 2026-10-04 +verified_on: 2026-10-04 +confidence: high +tags: [hermes, ct450, chaos, update, incident] +sources: [live-probe, pve-100.96.100.82, session-2026-10-03] +--- + +# CT450 Hermes Update to v0.21.5 + +## Summary +The Chaos instance (LXC CT450, `hermesagent`) was updated **v0.19.0 → v0.21.5** on 2026-10-03 — its first update since July. CT460 (production) was rebuilt the same morning, so both instances now carry the same semver token from separate builds. + +## Result (live-verified 2026-10-04) +| | CT460 (production) | CT450 (chaos) | +|---|---|---| +| Version | `v0.21.5+6169.gab86a20` (2026.9.24) | `v0.21.5+6170.ga510d6c` (2026.9.24) | +| Upstream | `ab86a20a` | `a510d6c6` — 3 commits behind, `hermes update` available | +| Python | managed runtime 3.14 (legacy `venv/` 3.11.15) | 3.14.7 | +| Install tree mtime | 2026-10-03 10:11 AWST | 2026-10-03 10:22 AWST | + +## Access +**CT450 has no SSH.** Reach it through the PVE host: + +```bash +ssh root@100.96.100.82 +pct exec 450 -- # or: pct enter 450 +pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version" +``` + +Two gotchas that cost time: +- `pct exec 450 -- hermes ...` **fails** with `Failed to exec "hermes"` — the binary is not on `lxc-attach`'s default PATH. Use the absolute path, or `su - hermes -c` (the dash matters: only a login shell puts `~/.local/bin` on PATH). +- A `hermes` login shell prints a `hermes-setup` banner ahead of command output; parse with `| tail -n` rather than expecting clean stdout. + +## Pre-flight checks (each cost one failed run on 2026-10-03) +1. **Build toolchain.** `clang` must be present *before* updating — source-compiled Python dependencies will not build without it. Verified present on CT450 (`/usr/bin/clang`, 2026-10-04). +2. **Stale update locks.** A leftover `.install.lock` / `.lazy-refresh-incomplete` in the Hermes home makes the updater refuse or silently skip the refresh step. Neither exists on CT450 now. +3. **Install-tree ownership.** `~/.hermes/hermes-agent` must stay owned by `hermes:hermes` (uid 1000); a root-owned tree left by an earlier root-run update fails the in-place build. +4. **Caddy keyring permissions.** Host-level Caddy/`keyring` file permissions (`0600`) blocked a step mid-update — check them before starting, not during. + +## Verification +- `hermes --version` on CT450 → `Hermes Agent v0.21.5+6170.ga510d6c (2026.9.24) · upstream a510d6c6`, Python 3.14.7. +- Install tree mtime `2026-10-03 10:22` matches the update window. +- `clang` present; no stale locks present. +- CT460 production untouched — separate config, skills, cron and memory state. + +## Follow-ups +- CT450 reports `Update available: 3 commits behind`. Harmless; re-run `hermes update` if the chaos box should track upstream exactly. +- CT450's Mattermost connection is failing (`mattermost connect timed out after 30s`, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See [[agents/hermes-chaos]]. + +## Related +- [[agents/hermes-chaos]] +- [[agents/hermes-production]] +- [[runbooks/update-hermes-safely]] +- [[infrastructure/hosts]] diff --git a/incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md b/incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md new file mode 100644 index 0000000..685833c --- /dev/null +++ b/incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md @@ -0,0 +1,60 @@ +--- +title: Mnemosyne Vector Recall Ran FTS-Only +type: incident +status: resolved +created: 2026-10-04 +updated: 2026-10-04 +verified_on: 2026-10-04 +confidence: high +tags: [mnemosyne, memory, embeddings, fastembed, recall, incident] +sources: [live-probe, check_embedding_coverage.py, session-2026-10-03] +--- + +# Mnemosyne Vector Recall Ran FTS-Only + +## Summary +Until 2026-10-03, `mnemosyne_recall` was scoring **keyword-only**. `fastembed` was missing from every UV-managed runtime, so the dense (vector) leg of recall contributed no candidates. Nothing errored — recall still returned results, just worse ones. Detected and repaired 2026-10-03; re-verified live 2026-10-04. + +## Impact +- Semantic similarity search silently degraded to keyword/`FTS5` matching. +- No user-visible error, no failed cron job, no alert. The failure mode of a missing embedding backend is **quiet recall degradation**, not a crash — which is why it survived unnoticed. +- Nothing was lost: the residue is *un-embedded* rows, not missing rows (26 working / 6 episodic as of 2026-10-04). New writes embed normally. + +## Root Cause +Three overlapping conditions: +1. **`fastembed` absent from the managed runtimes.** The gateway runs from a uv-managed venv under `~/.hermes/installs//environments//venv`, and none of them had `fastembed`. The *legacy* `/home/hermes/.hermes/hermes-agent/venv` had it (0.8.0), which is why CLI-side checks looked healthy while the gateway could not embed. +2. **`FASTEMBED_CACHE_PATH` unset** in `~/.hermes/.env`, so the local model directory was resolved implicitly rather than pinned. +3. **Isolated-mode launches.** The embedding worker can be launched with `python -I`, which ignores `PYTHONPATH`/`PYTHONHOME` — any path-based injection of the dependency does not survive that launch mode. + +## Fix (2026-10-03) +- Installed `fastembed` (0.8.1) into the runtime the bootstrap actually selects. +- Pinned `FASTEMBED_CACHE_PATH` (models cached under `~/.hermes/cache/fastembed`, `BAAI/bge-small-en-v1.5`). +- Extended `~/.hermes/scripts/hermes-runtime-deps-guard.sh` — run every 15 min by `mnemosyne-watchdog.py` — so it now asserts, in the bootstrap-selected venv: `fastembed` importable, `FASTEMBED_CACHE_PATH` set, and the model directory present, on top of its existing `dotenv` + `mnemosyne` checks in the installer python and every managed venv. +- No gateway restart required. + +## Verification (live, 2026-10-04) +``` +Memories with embeddings: 1759 / 2194 (80.2%) +Working memory: 726 / 752 (96.5%) +Episodic memory: 1033 / 1039 (99.4%) +vec_episodes_rowids: 1035 +vec_working_rowids: 739 +Embeddings generated (24h): 13 (working: 7, episodic: 6) +``` +`onnxruntime` threads now appear in `~/.hermes/logs/mcp-stderr.log` on each model load — direct evidence the embedding model is executing inside the gateway runtime. The dependency guard exits 0. + +## Caveat — do not misread `mnemosyne doctor` +`mnemosyne doctor` and the recall CLI run under the **legacy** `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable. Doctor therefore prints `vec0: present_but_unloadable` and marks the working/episodic vector tiers `unavailable`, even while the gateway embeds normally. The managed Python 3.14 venvs cannot import `sqlite_vec` either when probed directly. **Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line.** + +## Benign noise +`[E:onnxruntime:Default, env.cc:228 ThreadMain] pthread_setaffinity_np failed for thread: … mask: {0,}` — a container CPU-affinity warning emitted on every model load. Not an error; do not chase it. + +## Follow-ups +- Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%. +- Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this. + +## Related +- [[systems/mnemosyne]] +- [[systems/memory-backend]] +- [[incidents/2026-09-28-hermes-managed-runtime-deps]] +- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] diff --git a/index.md b/index.md index 325dfff..1f748c4 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-03 +> model routing and automation. Last updated: 2026-10-04 ## Start here - [[current-state]] — Current systems status @@ -28,10 +28,12 @@ ### Integrations - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. -- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; 6 failing as of 2026-10-03) +- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; 6 failing as of 2026-10-04) - [[systems/skills-index]] — Installed skills inventory (auto-regenerated daily; being refreshed by hand while the stocktake job is blocked) ## Incidents +- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic +- [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented - [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired; **2 of 3 jobs sharing the pin now failed**, the third untested (open 2026-10-01) - [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26) - [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min diff --git a/infrastructure/hosts.md b/infrastructure/hosts.md index 57b5831..2c9af59 100644 --- a/infrastructure/hosts.md +++ b/infrastructure/hosts.md @@ -3,7 +3,7 @@ title: Known Hosts and Paths type: infrastructure status: partial created: 2026-07-22 -updated: 2026-08-14 +updated: 2026-10-04 verified_on: 2026-07-22 confidence: medium tags: [infrastructure, hosts, containers, paths] @@ -21,7 +21,7 @@ sources: [] | omni | OmniRoute proxy | `http://omni:20128/v1` | Yes (from config, 2026-08-13) | | CT201 | Steel Browser, Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.166` (LAN) | Yes (live probe) | | CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net` | Yes | -| CT450 (hermesagent) | Hermes Agent v0.19.0, hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes | +| CT450 (hermesagent) | Hermes Agent **v0.21.5+6170.ga510d6c** (updated 2026-10-03 from v0.19.0), hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes | ## Project Paths diff --git a/log.md b/log.md index 5f0e119..8371d0b 100644 --- a/log.md +++ b/log.md @@ -298,3 +298,27 @@ - **`systems/skills-index.md` regenerated 341 → 344** — including a new skill, `fleet-wide-failure-triage` (devops). This was done **by hand**: job `1a8e90ea2271` is still blocked on HTTP 410 (retired `qwen3.5:397b`) and cannot self-recover. `wiki-maintain.sh` itself is healthy and was run manually, also writing `raw/stocktake/freshness-20261003.txt` — recorded so the next reader knows the stocktake is on manual refresh, not on schedule. - **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live under `wiki/`; `index.md` sits at the vault root. - No secrets written; all credential-shaped values in config remain redacted. + +## [2026-10-04] Nightly audit — CT450 update + Mnemosyne vector-recall repair + +Reviewed window: 2026-10-03 00:12 → 2026-10-04 00:12 AWST. + +**New pages** +- `incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md` — `mnemosyne_recall` had been scoring keyword-only: `fastembed` was absent from every uv-managed runtime, so the dense leg produced no candidates and recall fell back to FTS5 *without raising an error*. Repaired 2026-10-03; the runtime guard now asserts fastembed + `FASTEMBED_CACHE_PATH` + model dir every 15 min. Live 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%). +- `incidents/2026-10-03-ct450-hermes-update.md` — CT450 (`hermesagent`) updated v0.19.0 → v0.21.5, its first update since July. Records the no-SSH access path (`ssh root@100.96.100.82` → `pct exec 450 -- …`) and the four pre-flight checks (clang, stale locks, tree ownership, Caddy keyring perms). + +**Updated** +- `current-state.md` — new Recent Changes (2026-10-04) block: the two findings above, the managed-runtime prune 10 → 3, cron re-verification, Mattermost unreachable, and the `mnemosyne doctor` / `sqlite_vec` misread gotcha. +- `systems/mnemosyne.md` — Database Stats refreshed to live 2026-10-04 numbers; the 2026-08-02 "vector coverage gap" marked RESOLVED (retired as a current issue); new Known Issues subsection for the FTS-only degradation plus the doctor caveat; History entry; embedding-coverage job `b63ed5817212` documented. +- `systems/hermes-agent.md` — version corrected to `v0.21.5+6169.gab86a20` (was `v0.21.5+3837.gbfda74c`), managed-runtime count corrected 10 → 3, guard description extended, profile line corrected (no `profiles/default/`; `ops` is the only named sub-profile), Known Issues bullet added. +- `systems/memory-backend.md` — Known Issues bullet for the FTS-only degradation. +- `systems/scheduled-tasks.md` — 2026-10-04 re-verification (30 jobs, failing set still 6, no new failures in the window), `79f9bb41ecf5` due 2026-10-04 18:00; bare-`telegram` delivery warnings confirmed live. +- `agents/hermes-production.md` — version, Python, profile, provider routing, cron count and messaging refreshed; stale `custom:litellm` / `freellmapi-auto` references replaced. +- `agents/hermes-chaos.md` — CT450 version bump recorded, no-SSH access path added, Mattermost failure noted. +- `infrastructure/hosts.md` — CT450 row updated to v0.21.5. +- `runbooks/update-hermes-safely.md` — new "Second instance — CT450" section (access path + pre-flight checks). +- `runbooks/provider-health.md` — stale v0.19.0 version reference corrected. +- `index.md` — last-updated bumped; both new incidents linked; cron-fleet line re-dated. + +**Note** — `mnemosyne doctor` on this host prints `vec0: present_but_unloadable` and marks the vector tiers unavailable. That is a Python-3.11-legacy-`venv` artefact, not the gateway's state. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`. + diff --git a/runbooks/provider-health.md b/runbooks/provider-health.md index f106f41..e2a62af 100644 --- a/runbooks/provider-health.md +++ b/runbooks/provider-health.md @@ -3,7 +3,7 @@ title: Provider Health Check type: runbook status: active created: 2026-07-22 -updated: 2026-07-22 +updated: 2026-10-04 verified_on: 2026-07-22 last_tested: 2026-07-22 confidence: high @@ -84,7 +84,7 @@ Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or - Gateway journal shows no red alarm pattern. ## Current install references -- Hermes: v0.19.0, git install: `/home/hermes/.hermes/hermes-agent` +- Hermes: v0.21.5+6169.gab86a20 (2026.9.24), git install: `/home/hermes/.hermes/hermes-agent` - LiteLLM base URL: `http://litellm:4000/v1` - OmniRoute base URL: `http://omniroute:20128/v1` - OmniRoute direct IP: `100.88.81.19:20128` diff --git a/runbooks/update-hermes-safely.md b/runbooks/update-hermes-safely.md index 6e5ada6..36e8dcc 100644 --- a/runbooks/update-hermes-safely.md +++ b/runbooks/update-hermes-safely.md @@ -3,7 +3,7 @@ title: Update Hermes safely type: runbook status: active created: 2026-07-22 -updated: 2026-07-22 +updated: 2026-10-04 verified_on: 2026-07-22 last_tested: 2026-07-22 confidence: high @@ -93,10 +93,30 @@ hermes dashboard restart - If update warns about config migration, revert the backup and re-run manually ## Notes -- Hermes version on this install: v0.19.0 (2026.7.20) +- Hermes version on this install: v0.21.5+6169.gab86a20 (2026.9.24); install tree rebuilt 2026-10-03 10:11 AWST - Install directory: `/home/hermes/.hermes/hermes-agent` - Update preserves venv by default unless `--force-venv` is used +## Second instance — CT450 (`hermesagent`), updated 2026-10-03 (v0.19.0 → v0.21.5) + +CT450 has **no SSH**. Reach it from the PVE host: + +```bash +ssh root@100.96.100.82 +pct exec 450 -- # or: pct enter 450 +pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version" +``` + +`pct exec 450 -- hermes ...` fails (`Failed to exec "hermes"`): the binary is not on `lxc-attach`'s PATH. Use the absolute path or `su - hermes -c` (the dash gives a login shell, which sets `~/.local/bin`). A `hermes` login shell also prints a `hermes-setup` banner before output. + +**Pre-flight before running the update** (each of these cost one failed run on 2026-10-03): +1. `clang` installed — source-compiled Python dependencies will not build without it (`pct exec 450 -- bash -lc "which clang"`). +2. No stale `.install.lock` / `.lazy-refresh-incomplete` in the Hermes home. +3. `~/.hermes/hermes-agent` owned by `hermes:hermes` (a root-owned tree from an earlier root-run update breaks the in-place build). +4. Host Caddy/`keyring` file permissions sane (`0600` keyring blocked a step mid-update). + +**After:** `pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"`, confirm CT460 production is untouched (separate config/skills/cron/memory). Full record: [[incidents/2026-10-03-ct450-hermes-update]]. + ## Last tested 2026-07-22 on debian 13, git-based Hermes install diff --git a/systems/hermes-agent.md b/systems/hermes-agent.md index e05320c..969c369 100644 --- a/systems/hermes-agent.md +++ b/systems/hermes-agent.md @@ -3,8 +3,8 @@ title: Hermes Agent type: system status: active created: 2026-07-22 -updated: 2026-10-03 -verified_on: 2026-10-03 +updated: 2026-10-04 +verified_on: 2026-10-04 confidence: high tags: [system, hermes, agent, llm] sources: [live-hermes-version, live-config, live-skill-inventory] @@ -16,16 +16,16 @@ sources: [live-hermes-version, live-config, live-skill-inventory] Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks. ## Current Status -- **Version:** v0.21.5+3837.gbfda74c (2026.9.24), upstream `27062c34` — verified 2026-09-30 +- **Version:** v0.21.5+6169.gab86a20 (2026.9.24), upstream `ab86a20a` — verified live 2026-10-04. Same semver token as the earlier `v0.21.5+3837.gbfda74c` (`27062c34`), but a newer build: the install tree was rebuilt 2026-10-03 10:11 AWST. - **Install method:** git -- **Python:** 3.11.15 -- **Upstream status:** Update available (`hermes update`) +- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15) +- **Upstream status:** tree rebuilt 2026-10-03 from upstream `ab86a20a`; update availability not re-checked since. - **Install directory:** `/home/hermes/.hermes/hermes-agent` -- **Managed runtimes:** 10 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments//venv`. Dependency guard `hermes-runtime-deps-guard.sh` verifies `dotenv` + `mnemosyne` in every venv every 15 min — see [[incidents/2026-09-28-hermes-managed-runtime-deps]]. +- **Managed runtimes:** 3 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments//venv` (was 10 — the 2026-10-03 rebuild pruned superseded environments and created two new ones at 09:59/10:00; the third survives from 2026-09-28). `hermes-runtime-deps-guard.sh` runs every 15 min via `mnemosyne-watchdog.py` and now asserts `dotenv` + `mnemosyne` in the installer python *and every managed venv*, **plus** `fastembed` + `FASTEMBED_CACHE_PATH` + the local model dir in the bootstrap-selected venv — see [[incidents/2026-09-28-hermes-managed-runtime-deps]] and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]]. - **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions - **Binary:** `/home/hermes/.local/bin/hermes` - **Config:** `~/.hermes/config.yaml` -- **Profile:** `default` (`~/.hermes/profiles/default/`) +- **Profile:** `default` (top-level config `~/.hermes/config.yaml`; the only named sub-profile on this host is `ops` at `~/.hermes/profiles/ops/`) ## Location - **Host:** Local Linux host (no containerized Hermes runtime confirmed) @@ -131,6 +131,7 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi ## Known Issues - Managed-runtime venvs previously drifted on dependencies, breaking cron workers — now guarded every 15 min. See [[incidents/2026-09-28-hermes-managed-runtime-deps]]. +- **Vector recall silently ran keyword-only until 2026-10-03 (repaired).** `fastembed` was missing from every uv-managed runtime and `FASTEMBED_CACHE_PATH` was unset, so the dense leg of `mnemosyne_recall` returned no candidates and recall degraded to FTS5 — with no error raised. Repaired 2026-10-03 and the guard extended (above). Live coverage 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%). See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]]. - **Four cron jobs cannot self-recover — three pinned to models that no longer exist upstream, plus a credential fault:** `1a8e90ea2271` → `qwen3.5:397b` retired 2026-09-25 (HTTP 410); `dca8482e4f76` → `gpt-5.6-terra` plan-gated (403 `MODEL_NOT_IN_PLAN`); `fad5a86979d4` + `0e753e9720fd` → `deepseek-v4-flash` absent from the NVIDIA live catalog (HTTP 400); and `06a0a1c8704f` + `1b23c042e17d` → the `cliprox/terra` provider *connection* auth expired (HTTP 401), now confirmed on 2 of the 3 jobs sharing that pin. Repinning to the `smart-route` alias is the durable fix for the model faults; the cliprox pair needs the OmniRoute connection re-authenticated first. See [[systems/scheduled-tasks]], [[systems/model-providers]] and [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. - **The daily skills stock-take is not running.** Job `1a8e90ea2271` is blocked on the retired model above, so [[skills-index]] and the freshness audit are only refreshed when `wiki-maintain.sh` is run by hand (last done 2026-10-03). The script itself is healthy — only the scheduled wrapper is broken. - **Telegram thread delivery failures — root-caused and fixed 2026-09-30.** A bare `deliver: "telegram"` resolves to the home DM (`1793951355`), where group forum topic IDs cannot exist; the send hard-failed and the message was lost. Two jobs were retargeted to `telegram:-1004321904721:102` and verified. Three jobs still use bare `telegram` targets. See [[systems/messaging-integrations]]. diff --git a/systems/memory-backend.md b/systems/memory-backend.md index ba3111b..c67d6bb 100644 --- a/systems/memory-backend.md +++ b/systems/memory-backend.md @@ -3,8 +3,8 @@ title: Memory Backend type: system status: active created: 2026-07-22 -updated: 2026-08-01 -verified_on: 2026-08-01 +updated: 2026-10-04 +verified_on: 2026-10-04 confidence: high tags: [memory, mnemosyne, agent] sources: [raw/configs/hermes-config-sanitized.txt] @@ -43,6 +43,7 @@ Persist and recall agent memory across sessions. ## Known Issues - Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods. - The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]]. +- **Vector recall degraded silently until 2026-10-03** — `fastembed` was missing from every uv-managed runtime, so `mnemosyne_recall` fell back to FTS5 keyword scoring with no error raised. Repaired 2026-10-03; the 15-min dependency guard now asserts `fastembed` and the model cache path. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]]. **Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]]. diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index d8cd21d..1ac996b 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -3,8 +3,8 @@ title: Mnemosyne type: system status: active created: 2026-07-22 -updated: 2026-10-01 -verified_on: 2026-10-01 +updated: 2026-10-04 +verified_on: 2026-10-04 confidence: high tags: [memory, mnemosyne, backend] sources: [live-probe, mnemosyne-db, config, ops-profile-config] @@ -33,25 +33,34 @@ Primary agent memory store for Hermes. - **Embedding model:** `BAAI/bge-small-en-v1.5` (384 dims) - **Embeddings via API:** false (local fastembed) -## Database Stats (2026-08-02) +## Database Stats (live, 2026-10-04) | Table | Rows | Notes | |---|---|---| -| `working_memory` | 1,476 | Active memories | -| `episodic_memory` | 458 | Consolidated summaries | -| `memory_embeddings` | 158 | Only working memories — **0 episodic** | +| `working_memory` | 752 | Active memories | +| `episodic_memory` | 1,039 | Consolidated summaries | +| `memory_embeddings` | 739 | Working-tier embeddings | | `memory` (legacy) | 2,181 | Old table, not migrated | -| `vec_working` | 159 | Vector index rows | +| `vec_working` / `vec_working_rowids` | 739 | Working vector index | +| `vec_episodes` / `vec_episodes_rowids` | 1,035 | Episodic vector index | +| `vec_facts_rowids` | 0 | `memoria_facts` is intentionally left unvectorised | | Shared surface | 0 | Empty — cross-agent sharing not functional | +**Embedding coverage (live, 2026-10-04):** 1,759/2,194 memories embedded (80.2%) — working **726/752 (96.5%)**, episodic **1,033/1,039 (99.4%)**; 13 embeddings generated in the previous 24h. Working coverage is above the 95% alert threshold enforced by the Embedding Coverage job (`b63ed5817212`, every 6h, Telegram topic 102). + ## Known Issues -### Vector embedding coverage gap -- **Only 159/1,476 working memories (10.8%)** have vector embeddings -- **0/458 episodic memories** have embeddings -- Semantic search (`mnemosyne_recall`) falls back to FTS5 keyword matching for ~90% of working and 100% of episodic memories -- This means recall quality is degraded — vector similarity can only search a tiny fraction of the memory store -- **Root cause:** The embedding pipeline (fastembed) never processed the bulk of memories. The `vec_working` table was rebuilt once (Jul 18) but only captured 159 rows. -- **Fix needed:** Backfill embeddings for all working and episodic memories. This requires running the embedding pipeline across the full dataset. +### Vector embedding coverage gap — RESOLVED (2026-08-02) +- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories. +- **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only. +- Standard repair if coverage drops below 95% working: `~/.hermes/hermes-agent/venv/bin/mnemosyne reindex --yes` (the Embedding Coverage job `b63ed5817212` triggers the same path every 6h). + +### Vector recall silently ran FTS-only (2026-10-03 — repaired) +- `fastembed` was absent from **every** uv-managed runtime, so the dense leg of `mnemosyne_recall` produced no candidates and recall fell back to FTS5/keyword scoring **without raising an error**. Sessions kept working, which is why it went unnoticed. +- Compounded by `FASTEMBED_CACHE_PATH` being unset in `~/.hermes/.env`, and by the embedding worker sometimes launching in isolated (`python -I`) mode, which does not inherit `PYTHONPATH`. +- **Repaired 2026-10-03:** `fastembed` installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min alongside its existing `dotenv`/`mnemosyne` checks. +- **CLI caveat — do not misread `mnemosyne doctor`.** Doctor and the recall CLI run under the legacy `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable; doctor therefore prints `vec0: present_but_unloadable` and marks the vector tiers `unavailable` even while the gateway embeds normally. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line. +- Benign noise: `pthread_setaffinity_np failed … mask: {0,}` from onnxruntime in `mcp-stderr.log` is a container affinity warning emitted on every model load — not an error. +- See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]]. ### Profile isolation chosen over shared surface |- The ops profile was enabled with mnemosyne on 2026-08-02 @@ -75,9 +84,11 @@ Primary agent memory store for Hermes. - Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent - Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent - Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception +- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95% - Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent ## History and Evidence +- **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]]. - **2026-09-30 memory-hygiene pass:** 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (`hermes_cron_model_default` → `custom:omni-101`, job-radar model). No unresolved conflicts left in the working tier. - **2026-08-19 Mnemosyne Watchdog incident:** Watchdog job (`96eb95ce12f2`) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See [[systems/scheduled-tasks]] Known Issues section. - **2026-08-18 incident:** Working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See [[incidents/2026-08-18-mnemosyne-capture-pipeline]]. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 86d21a0..ccc0e28 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,8 +3,8 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-03 -verified_on: 2026-10-03 +updated: 2026-10-04 +verified_on: 2026-10-04 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db] @@ -25,6 +25,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] - **Resolved since the last entry:** 1 — `1b23c042e17d`'s stale `delivery_failed` flag cleared exactly as predicted. Its 2026-10-02 run delivered to topic `102` (`delivery_outcome: delivered`); it then failed on the model pin, above. - **Shared model pin at risk:** 3 enabled jobs run `cliprox/terra` — `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`. **Two of the three have now failed on that pin.** The third (`79f9bb41ecf5`) last ran 2026-09-27 — before the credential lapsed — and is next due 2026-10-04 18:00. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. - **Streak recount (2026-10-03):** recounted directly from `~/.hermes/cron/executions.db`. `dca8482e4f76` 6 → 7 (one more daily run since), and `1a8e90ea2271` 8 → **7** — the ×8 carried in the 2026-10-02 entry was an overcount: that job has exactly 7 recorded runs and all 7 failed. +- **Re-verified 2026-10-04 00:12 AWST — no change:** still 30 jobs (27 enabled / 3 disabled) and the same **6** failing jobs; no new failure event landed inside the audit window. `79f9bb41ecf5` (Weekly Review, the last untested `cliprox/terra` job) drops at **2026-10-04 18:00 AWST** — its result settles whether that pin is 3/3 broken. `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found; delivered without thread_id`), confirmed live in `hermes cron list`. - Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) ## Job Table @@ -87,7 +88,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] ### Open - **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 10-02, streak 7):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change; it cannot self-recover. See [[runbooks/job-radar]]. -- **`cliprox/terra` provider connection auth expired — 2 of 3 jobs confirmed (streak 1 each):** Jobs `06a0a1c8704f` (2026-10-01) and `1b23c042e17d` (2026-10-02) both failed with `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Failure notices reached their topics; no output was produced. This is a *credential/connection* problem at the proxy, not a missing model — it will not self-recover until the connection is re-authenticated. `79f9bb41ecf5` is the remaining untested job on this pin (next run 2026-10-04 18:00). Re-authenticate the OmniRoute connection before repinning anything. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. +- **`cliprox/terra` provider connection auth expired — 2 of 3 jobs confirmed (streak 1 each):** Jobs `06a0a1c8704f` (2026-10-01) and `1b23c042e17d` (2026-10-02) both failed with `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Failure notices reached their topics; no output was produced. This is a *credential/connection* problem at the proxy, not a missing model — it will not self-recover until the connection is re-authenticated. `79f9bb41ecf5` is the remaining untested job on this pin (next run 2026-10-04 18:00 — re-check on 2026-10-05). Re-authenticate the OmniRoute connection before repinning anything. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. - **A delivered notice is not a successful job:** both `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output. - **wiki-stocktake-maintenance — HTTP 410, model retired (streak 7):** Job `1a8e90ea2271` is pinned to `ollama-cloud / qwen3.5:397b`, which Ollama retired on 2026-09-25 (`qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT`). It cannot self-recover — it needs a new model pin. **The job's script is fine:** `wiki-maintain.sh` was run by hand on 2026-10-03 and completed cleanly, regenerating [[systems/skills-index]] (344 skills / 70 categories) and refreshing the freshness audit. Only the scheduled wrapper is broken — so every non-model step of the daily stocktake needs a manual run until it is repinned. - **Facebook Birthday scrapes — HTTP 400, model missing from catalog (streak 3 / 1):** Job `fad5a86979d4` (weekly) is pinned to `nvidia/deepseek-ai/deepseek-v4-flash`; the provider reports the model is `not available in the active live catalog for provider 'nvidia'`. **Confirmed as a shared-pin issue on 2026-10-01**, when sibling job `0e753e9720fd` (monthly) failed on the same pin at its own 09:00 run. Safest durable fix is repinning both to a routing alias (`smart-route`) instead of a raw model name.