wiki audit 2026-10-04: CT450 v0.21.5 update incident, mnemosyne vector-recall FTS-only repair + coverage refresh, managed runtimes 10->3, doctor vec0 caveat
This commit is contained in:
@@ -3,8 +3,8 @@ title: Hermes Chaos
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-15
|
created: 2026-08-15
|
||||||
updated: 2026-08-17
|
updated: 2026-10-04
|
||||||
verified_on: 2026-08-17
|
verified_on: 2026-10-04
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, chaos, ct450, experimental, agent]
|
tags: [hermes, chaos, ct450, experimental, agent]
|
||||||
sources: [live-probe, config-inspection]
|
sources: [live-probe, config-inspection]
|
||||||
@@ -19,6 +19,7 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod
|
|||||||
## Location
|
## Location
|
||||||
|
|
||||||
- **Container:** LXC CT450 (`hermesagent`)
|
- **Container:** LXC CT450 (`hermesagent`)
|
||||||
|
- **Access:** **no SSH** — go through the PVE host: `ssh root@100.96.100.82`, then `pct exec 450 -- <cmd>` (or `pct enter 450`). `hermes` is not on `pct exec`'s default PATH; use `su - hermes -c "/home/hermes/.local/bin/hermes …"`.
|
||||||
- **Tailscale IP:** `100.98.205.112`
|
- **Tailscale IP:** `100.98.205.112`
|
||||||
- **Hostname:** `hermesagent`
|
- **Hostname:** `hermesagent`
|
||||||
- **Software:** hermes-ui (Vite web app for browser use, port 5174)
|
- **Software:** hermes-ui (Vite web app for browser use, port 5174)
|
||||||
@@ -27,6 +28,8 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod
|
|||||||
## Current State
|
## Current State
|
||||||
|
|
||||||
- **Status:** Active (experimental)
|
- **Status:** Active (experimental)
|
||||||
|
- **Hermes version:** `v0.21.5+6170.ga510d6c` (2026.9.24), upstream `a510d6c6` — **updated from v0.19.0 on 2026-10-03**, first update since July; 3 commits behind upstream. Python 3.14.7. See [[incidents/2026-10-03-ct450-hermes-update]].
|
||||||
|
- **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]].
|
||||||
- **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01)
|
- **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01)
|
||||||
- **Duplicate cron jobs:** 11 paused (no post-pause execution records since 2026-08-01)
|
- **Duplicate cron jobs:** 11 paused (no post-pause execution records since 2026-08-01)
|
||||||
- **Production counterparts:** CT460 has 11 enabled jobs with successful latest runs
|
- **Production counterparts:** CT460 has 11 enabled jobs with successful latest runs
|
||||||
|
|||||||
+12
-10
@@ -3,8 +3,8 @@ title: Hermes Production
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-15
|
created: 2026-08-15
|
||||||
updated: 2026-08-15
|
updated: 2026-10-04
|
||||||
verified_on: 2026-08-15
|
verified_on: 2026-10-04
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, production, ct460, agent]
|
tags: [hermes, production, ct460, agent]
|
||||||
sources: [live-probe, config-inspection]
|
sources: [live-probe, config-inspection]
|
||||||
@@ -27,11 +27,11 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
## Current State
|
## Current State
|
||||||
|
|
||||||
- **Status:** Active
|
- **Status:** Active
|
||||||
- **Hermes version:** v0.19.0
|
- **Hermes version:** `v0.21.5+6169.gab86a20` (2026.9.24), upstream `ab86a20a`; install tree rebuilt 2026-10-03 10:11 AWST — verified live 2026-10-04
|
||||||
- **Python:** 3.11.15
|
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
|
||||||
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
|
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
|
||||||
- **Config:** `~/.hermes/config.yaml`
|
- **Config:** `~/.hermes/config.yaml`
|
||||||
- **Profile:** `default` (`~/.hermes/profiles/default/`)
|
- **Profile:** `default` — top-level config; **no `profiles/default/` directory exists** (`ops` at `~/.hermes/profiles/ops/` is the only named sub-profile)
|
||||||
- **Soul:** Rhino — trickster-mentor in a green jacket 🎩
|
- **Soul:** Rhino — trickster-mentor in a green jacket 🎩
|
||||||
|
|
||||||
## Profiles
|
## Profiles
|
||||||
@@ -43,9 +43,10 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
|
|
||||||
## Providers & Routing
|
## Providers & Routing
|
||||||
|
|
||||||
- **Default provider:** `custom:litellm` → `http://litellm:4000/v1`
|
- **Default provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — as attested 2026-09-30; the `custom:omni` provider key no longer exists
|
||||||
- **Fallback provider:** `custom:omniroute` → `http://omniroute:20128/v1`
|
- **Default model:** `smart-route` (previously `freellmapi-auto`)
|
||||||
- **Default model:** `freellmapi-auto`
|
- **Fallback providers:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`, `custom / litellm:4000`
|
||||||
|
- **Auxiliary curator model:** `openai-codex / gpt-5.6-terra`
|
||||||
- See [[systems/model-providers]] for full breakdown.
|
- See [[systems/model-providers]] for full breakdown.
|
||||||
|
|
||||||
## Memory
|
## Memory
|
||||||
@@ -57,7 +58,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
|
|
||||||
## Scheduled Tasks
|
## Scheduled Tasks
|
||||||
|
|
||||||
- **Cron jobs:** 25+ active jobs
|
- **Cron jobs:** 30 (27 enabled / 3 disabled; **6 failing** as of 2026-10-04)
|
||||||
- Delivery targets: Telegram (primary), Discord, origin, local
|
- Delivery targets: Telegram (primary), Discord, origin, local
|
||||||
- Notable: Birthday checker, job radar, newsletter digests, mnemosyne health/watchdog/backup
|
- Notable: Birthday checker, job radar, newsletter digests, mnemosyne health/watchdog/backup
|
||||||
- See [[systems/scheduled-tasks]] for full table.
|
- See [[systems/scheduled-tasks]] for full table.
|
||||||
@@ -65,6 +66,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
## Messaging
|
## Messaging
|
||||||
|
|
||||||
- Telegram, Discord, Slack, Email, Mattermost, ntfy
|
- Telegram, Discord, Slack, Email, Mattermost, ntfy
|
||||||
|
- **Mattermost is currently unreachable** — the gateway logs `mattermost connect timed out after 30s, next retry in 300s` (live, 2026-10-04) and the Mattermost host does not answer. Treat Mattermost delivery as down until probed.
|
||||||
- See [[systems/messaging-integrations]] for details.
|
- See [[systems/messaging-integrations]] for details.
|
||||||
|
|
||||||
## Key Differences from Chaos
|
## Key Differences from Chaos
|
||||||
@@ -72,7 +74,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
| Aspect | Production (CT460) | Chaos (CT450) |
|
| Aspect | Production (CT460) | Chaos (CT450) |
|
||||||
|--------|-------------------|---------------|
|
|--------|-------------------|---------------|
|
||||||
| Purpose | Stable daily operations | Experimental/testing |
|
| Purpose | Stable daily operations | Experimental/testing |
|
||||||
| Cron jobs | 25+ enabled | 11 paused duplicates |
|
| Cron jobs | 30 (27 enabled / 3 disabled) | 11 paused duplicates |
|
||||||
| Config changes | Requires verification | Safe to experiment |
|
| Config changes | Requires verification | Safe to experiment |
|
||||||
| Memory | Production Mnemosyne | Separate state |
|
| Memory | Production Mnemosyne | Separate state |
|
||||||
|
|
||||||
|
|||||||
+11
-3
@@ -3,8 +3,8 @@ title: Current State
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-03
|
updated: 2026-10-04
|
||||||
verified_on: 2026-10-03
|
verified_on: 2026-10-04
|
||||||
review_after: 2026-10-31
|
review_after: 2026-10-31
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system]
|
tags: [system]
|
||||||
@@ -12,7 +12,15 @@ sources: [live-config, live-cron-list, live-hermes-version]
|
|||||||
---
|
---
|
||||||
|
|
||||||
# Current State
|
# Current State
|
||||||
Updated: 2026-10-03
|
Updated: 2026-10-04
|
||||||
|
|
||||||
|
## Recent Changes (2026-10-04)
|
||||||
|
- **Both Hermes instances updated on 2026-10-03, from opposite directions.** CT460 (production) was rebuilt to `v0.21.5+6169.gab86a20` (upstream `ab86a20a`, tree mtime 10:11 AWST) and CT450 (chaos) came up from `v0.19.0` to `v0.21.5+6170.ga510d6c` (upstream `a510d6c6`, tree mtime 10:22 AWST). Same semver token, different builds, separate state. CT450 now reports `Update available: 3 commits behind`. CT450 has no SSH — it is reached via `ssh root@100.96.100.82` + `pct exec 450`, and `hermes` is not on the default `pct exec` PATH. See [[incidents/2026-10-03-ct450-hermes-update]].
|
||||||
|
- **Vector recall had been silently running keyword-only — found and fixed 2026-10-03.** `fastembed` was missing from every uv-managed runtime, so the dense leg of `mnemosyne_recall` contributed no candidates and recall fell back to FTS5 **without erroring**. Fixed the same day: fastembed 0.8.1 installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min. **Live coverage 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%)**, 13 embeddings generated in the previous 24h. The previous "coverage gap" entry (159/1,476) is retired — it was resolved by the 2026-08-02 reindex. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] and [[systems/mnemosyne]].
|
||||||
|
- **Managed runtimes pruned 10 → 3.** Only 3 uv-managed venvs remain under `~/.hermes/installs/019d0d114d22b7e6/environments/` (two created 2026-10-03 09:59/10:00, one surviving from 2026-09-28). Any note that says "ten managed venvs" is stale.
|
||||||
|
- **Cron fleet re-verified 2026-10-04 00:12 AWST:** 30 jobs (27 enabled / 3 disabled), failing set unchanged at 6, no new failure inside the window. `79f9bb41ecf5` — the last untested `cliprox/terra` job — is due 2026-10-04 18:00 AWST. See [[systems/scheduled-tasks]].
|
||||||
|
- **Mattermost is unreachable from CT460 right now** (`mattermost connect timed out after 30s, next retry in 300s`, every 5 min, live 2026-10-04). Host-side condition; Mattermost delivery should be treated as down until probed.
|
||||||
|
- **Wiki maintenance gotcha worth remembering:** `mnemosyne doctor` runs under the legacy Python 3.11 `venv/`, where `sqlite_vec` is not importable, so it prints `vec0: present_but_unloadable` and marks vector tiers `unavailable` *even when the gateway is embedding fine*. Judge vector health with `check_embedding_coverage.py`, never with doctor's `vec0` line.
|
||||||
|
|
||||||
## Recent Changes (2026-10-03)
|
## Recent Changes (2026-10-03)
|
||||||
- **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set stays at **6**, but the composition changed: `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) failed on 2026-10-02 with the *identical* HTTP 401 the cliprox incident predicted — `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. That makes **2 of the 3** `cliprox/terra` jobs confirmed broken, so the provider-connection auth expiry is a real credential fault, not a one-job fluke. `79f9bb41ecf5` (Weekly Review, Sun 18:00) is the last untested one — its only run (2026-09-27) predates the expiry and succeeded; next due **2026-10-04 18:00**. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
- **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set stays at **6**, but the composition changed: `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) failed on 2026-10-02 with the *identical* HTTP 401 the cliprox incident predicted — `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. That makes **2 of the 3** `cliprox/terra` jobs confirmed broken, so the provider-connection auth expiry is a real credential fault, not a one-job fluke. `79f9bb41ecf5` (Weekly Review, Sun 18:00) is the last untested one — its only run (2026-09-27) predates the expiry and succeeded; next due **2026-10-04 18:00**. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||||
|
|||||||
@@ -0,0 +1,59 @@
|
|||||||
|
---
|
||||||
|
title: CT450 Hermes Update to v0.21.5
|
||||||
|
type: incident
|
||||||
|
status: resolved
|
||||||
|
created: 2026-10-04
|
||||||
|
updated: 2026-10-04
|
||||||
|
verified_on: 2026-10-04
|
||||||
|
confidence: high
|
||||||
|
tags: [hermes, ct450, chaos, update, incident]
|
||||||
|
sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
|
||||||
|
---
|
||||||
|
|
||||||
|
# CT450 Hermes Update to v0.21.5
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
The Chaos instance (LXC CT450, `hermesagent`) was updated **v0.19.0 → v0.21.5** on 2026-10-03 — its first update since July. CT460 (production) was rebuilt the same morning, so both instances now carry the same semver token from separate builds.
|
||||||
|
|
||||||
|
## Result (live-verified 2026-10-04)
|
||||||
|
| | CT460 (production) | CT450 (chaos) |
|
||||||
|
|---|---|---|
|
||||||
|
| Version | `v0.21.5+6169.gab86a20` (2026.9.24) | `v0.21.5+6170.ga510d6c` (2026.9.24) |
|
||||||
|
| Upstream | `ab86a20a` | `a510d6c6` — 3 commits behind, `hermes update` available |
|
||||||
|
| Python | managed runtime 3.14 (legacy `venv/` 3.11.15) | 3.14.7 |
|
||||||
|
| Install tree mtime | 2026-10-03 10:11 AWST | 2026-10-03 10:22 AWST |
|
||||||
|
|
||||||
|
## Access
|
||||||
|
**CT450 has no SSH.** Reach it through the PVE host:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh root@100.96.100.82
|
||||||
|
pct exec 450 -- <command> # or: pct enter 450
|
||||||
|
pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"
|
||||||
|
```
|
||||||
|
|
||||||
|
Two gotchas that cost time:
|
||||||
|
- `pct exec 450 -- hermes ...` **fails** with `Failed to exec "hermes"` — the binary is not on `lxc-attach`'s default PATH. Use the absolute path, or `su - hermes -c` (the dash matters: only a login shell puts `~/.local/bin` on PATH).
|
||||||
|
- A `hermes` login shell prints a `hermes-setup` banner ahead of command output; parse with `| tail -n` rather than expecting clean stdout.
|
||||||
|
|
||||||
|
## Pre-flight checks (each cost one failed run on 2026-10-03)
|
||||||
|
1. **Build toolchain.** `clang` must be present *before* updating — source-compiled Python dependencies will not build without it. Verified present on CT450 (`/usr/bin/clang`, 2026-10-04).
|
||||||
|
2. **Stale update locks.** A leftover `.install.lock` / `.lazy-refresh-incomplete` in the Hermes home makes the updater refuse or silently skip the refresh step. Neither exists on CT450 now.
|
||||||
|
3. **Install-tree ownership.** `~/.hermes/hermes-agent` must stay owned by `hermes:hermes` (uid 1000); a root-owned tree left by an earlier root-run update fails the in-place build.
|
||||||
|
4. **Caddy keyring permissions.** Host-level Caddy/`keyring` file permissions (`0600`) blocked a step mid-update — check them before starting, not during.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
- `hermes --version` on CT450 → `Hermes Agent v0.21.5+6170.ga510d6c (2026.9.24) · upstream a510d6c6`, Python 3.14.7.
|
||||||
|
- Install tree mtime `2026-10-03 10:22` matches the update window.
|
||||||
|
- `clang` present; no stale locks present.
|
||||||
|
- CT460 production untouched — separate config, skills, cron and memory state.
|
||||||
|
|
||||||
|
## Follow-ups
|
||||||
|
- CT450 reports `Update available: 3 commits behind`. Harmless; re-run `hermes update` if the chaos box should track upstream exactly.
|
||||||
|
- CT450's Mattermost connection is failing (`mattermost connect timed out after 30s`, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See [[agents/hermes-chaos]].
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[agents/hermes-chaos]]
|
||||||
|
- [[agents/hermes-production]]
|
||||||
|
- [[runbooks/update-hermes-safely]]
|
||||||
|
- [[infrastructure/hosts]]
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
---
|
||||||
|
title: Mnemosyne Vector Recall Ran FTS-Only
|
||||||
|
type: incident
|
||||||
|
status: resolved
|
||||||
|
created: 2026-10-04
|
||||||
|
updated: 2026-10-04
|
||||||
|
verified_on: 2026-10-04
|
||||||
|
confidence: high
|
||||||
|
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
|
||||||
|
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Mnemosyne Vector Recall Ran FTS-Only
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
Until 2026-10-03, `mnemosyne_recall` was scoring **keyword-only**. `fastembed` was missing from every UV-managed runtime, so the dense (vector) leg of recall contributed no candidates. Nothing errored — recall still returned results, just worse ones. Detected and repaired 2026-10-03; re-verified live 2026-10-04.
|
||||||
|
|
||||||
|
## Impact
|
||||||
|
- Semantic similarity search silently degraded to keyword/`FTS5` matching.
|
||||||
|
- No user-visible error, no failed cron job, no alert. The failure mode of a missing embedding backend is **quiet recall degradation**, not a crash — which is why it survived unnoticed.
|
||||||
|
- Nothing was lost: the residue is *un-embedded* rows, not missing rows (26 working / 6 episodic as of 2026-10-04). New writes embed normally.
|
||||||
|
|
||||||
|
## Root Cause
|
||||||
|
Three overlapping conditions:
|
||||||
|
1. **`fastembed` absent from the managed runtimes.** The gateway runs from a uv-managed venv under `~/.hermes/installs/<install>/environments/<hash>/venv`, and none of them had `fastembed`. The *legacy* `/home/hermes/.hermes/hermes-agent/venv` had it (0.8.0), which is why CLI-side checks looked healthy while the gateway could not embed.
|
||||||
|
2. **`FASTEMBED_CACHE_PATH` unset** in `~/.hermes/.env`, so the local model directory was resolved implicitly rather than pinned.
|
||||||
|
3. **Isolated-mode launches.** The embedding worker can be launched with `python -I`, which ignores `PYTHONPATH`/`PYTHONHOME` — any path-based injection of the dependency does not survive that launch mode.
|
||||||
|
|
||||||
|
## Fix (2026-10-03)
|
||||||
|
- Installed `fastembed` (0.8.1) into the runtime the bootstrap actually selects.
|
||||||
|
- Pinned `FASTEMBED_CACHE_PATH` (models cached under `~/.hermes/cache/fastembed`, `BAAI/bge-small-en-v1.5`).
|
||||||
|
- Extended `~/.hermes/scripts/hermes-runtime-deps-guard.sh` — run every 15 min by `mnemosyne-watchdog.py` — so it now asserts, in the bootstrap-selected venv: `fastembed` importable, `FASTEMBED_CACHE_PATH` set, and the model directory present, on top of its existing `dotenv` + `mnemosyne` checks in the installer python and every managed venv.
|
||||||
|
- No gateway restart required.
|
||||||
|
|
||||||
|
## Verification (live, 2026-10-04)
|
||||||
|
```
|
||||||
|
Memories with embeddings: 1759 / 2194 (80.2%)
|
||||||
|
Working memory: 726 / 752 (96.5%)
|
||||||
|
Episodic memory: 1033 / 1039 (99.4%)
|
||||||
|
vec_episodes_rowids: 1035
|
||||||
|
vec_working_rowids: 739
|
||||||
|
Embeddings generated (24h): 13 (working: 7, episodic: 6)
|
||||||
|
```
|
||||||
|
`onnxruntime` threads now appear in `~/.hermes/logs/mcp-stderr.log` on each model load — direct evidence the embedding model is executing inside the gateway runtime. The dependency guard exits 0.
|
||||||
|
|
||||||
|
## Caveat — do not misread `mnemosyne doctor`
|
||||||
|
`mnemosyne doctor` and the recall CLI run under the **legacy** `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable. Doctor therefore prints `vec0: present_but_unloadable` and marks the working/episodic vector tiers `unavailable`, even while the gateway embeds normally. The managed Python 3.14 venvs cannot import `sqlite_vec` either when probed directly. **Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line.**
|
||||||
|
|
||||||
|
## Benign noise
|
||||||
|
`[E:onnxruntime:Default, env.cc:228 ThreadMain] pthread_setaffinity_np failed for thread: … mask: {0,}` — a container CPU-affinity warning emitted on every model load. Not an error; do not chase it.
|
||||||
|
|
||||||
|
## Follow-ups
|
||||||
|
- Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.
|
||||||
|
- Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/mnemosyne]]
|
||||||
|
- [[systems/memory-backend]]
|
||||||
|
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
|
||||||
|
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]]
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
# Anthony's Systems Wiki
|
# Anthony's Systems Wiki
|
||||||
> Operational knowledge for Anthony's agents, infrastructure,
|
> Operational knowledge for Anthony's agents, infrastructure,
|
||||||
> model routing and automation. Last updated: 2026-10-03
|
> model routing and automation. Last updated: 2026-10-04
|
||||||
|
|
||||||
## Start here
|
## Start here
|
||||||
- [[current-state]] — Current systems status
|
- [[current-state]] — Current systems status
|
||||||
@@ -28,10 +28,12 @@
|
|||||||
|
|
||||||
### Integrations
|
### Integrations
|
||||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
||||||
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; 6 failing as of 2026-10-03)
|
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; 6 failing as of 2026-10-04)
|
||||||
- [[systems/skills-index]] — Installed skills inventory (auto-regenerated daily; being refreshed by hand while the stocktake job is blocked)
|
- [[systems/skills-index]] — Installed skills inventory (auto-regenerated daily; being refreshed by hand while the stocktake job is blocked)
|
||||||
|
|
||||||
## Incidents
|
## Incidents
|
||||||
|
- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
|
||||||
|
- [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
|
||||||
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired; **2 of 3 jobs sharing the pin now failed**, the third untested (open 2026-10-01)
|
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired; **2 of 3 jobs sharing the pin now failed**, the third untested (open 2026-10-01)
|
||||||
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26)
|
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26)
|
||||||
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min
|
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ title: Known Hosts and Paths
|
|||||||
type: infrastructure
|
type: infrastructure
|
||||||
status: partial
|
status: partial
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-08-14
|
updated: 2026-10-04
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-07-22
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [infrastructure, hosts, containers, paths]
|
tags: [infrastructure, hosts, containers, paths]
|
||||||
@@ -21,7 +21,7 @@ sources: []
|
|||||||
| omni | OmniRoute proxy | `http://omni:20128/v1` | Yes (from config, 2026-08-13) |
|
| omni | OmniRoute proxy | `http://omni:20128/v1` | Yes (from config, 2026-08-13) |
|
||||||
| CT201 | Steel Browser, Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.166` (LAN) | Yes (live probe) |
|
| CT201 | Steel Browser, Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.166` (LAN) | Yes (live probe) |
|
||||||
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net` | Yes |
|
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net` | Yes |
|
||||||
| CT450 (hermesagent) | Hermes Agent v0.19.0, hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes |
|
| CT450 (hermesagent) | Hermes Agent **v0.21.5+6170.ga510d6c** (updated 2026-10-03 from v0.19.0), hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes |
|
||||||
|
|
||||||
## Project Paths
|
## Project Paths
|
||||||
|
|
||||||
|
|||||||
@@ -298,3 +298,27 @@
|
|||||||
- **`systems/skills-index.md` regenerated 341 → 344** — including a new skill, `fleet-wide-failure-triage` (devops). This was done **by hand**: job `1a8e90ea2271` is still blocked on HTTP 410 (retired `qwen3.5:397b`) and cannot self-recover. `wiki-maintain.sh` itself is healthy and was run manually, also writing `raw/stocktake/freshness-20261003.txt` — recorded so the next reader knows the stocktake is on manual refresh, not on schedule.
|
- **`systems/skills-index.md` regenerated 341 → 344** — including a new skill, `fleet-wide-failure-triage` (devops). This was done **by hand**: job `1a8e90ea2271` is still blocked on HTTP 410 (retired `qwen3.5:397b`) and cannot self-recover. `wiki-maintain.sh` itself is healthy and was run manually, also writing `raw/stocktake/freshness-20261003.txt` — recorded so the next reader knows the stocktake is on manual refresh, not on schedule.
|
||||||
- **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live under `wiki/`; `index.md` sits at the vault root.
|
- **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live under `wiki/`; `index.md` sits at the vault root.
|
||||||
- No secrets written; all credential-shaped values in config remain redacted.
|
- No secrets written; all credential-shaped values in config remain redacted.
|
||||||
|
|
||||||
|
## [2026-10-04] Nightly audit — CT450 update + Mnemosyne vector-recall repair
|
||||||
|
|
||||||
|
Reviewed window: 2026-10-03 00:12 → 2026-10-04 00:12 AWST.
|
||||||
|
|
||||||
|
**New pages**
|
||||||
|
- `incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md` — `mnemosyne_recall` had been scoring keyword-only: `fastembed` was absent from every uv-managed runtime, so the dense leg produced no candidates and recall fell back to FTS5 *without raising an error*. Repaired 2026-10-03; the runtime guard now asserts fastembed + `FASTEMBED_CACHE_PATH` + model dir every 15 min. Live 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%).
|
||||||
|
- `incidents/2026-10-03-ct450-hermes-update.md` — CT450 (`hermesagent`) updated v0.19.0 → v0.21.5, its first update since July. Records the no-SSH access path (`ssh root@100.96.100.82` → `pct exec 450 -- …`) and the four pre-flight checks (clang, stale locks, tree ownership, Caddy keyring perms).
|
||||||
|
|
||||||
|
**Updated**
|
||||||
|
- `current-state.md` — new Recent Changes (2026-10-04) block: the two findings above, the managed-runtime prune 10 → 3, cron re-verification, Mattermost unreachable, and the `mnemosyne doctor` / `sqlite_vec` misread gotcha.
|
||||||
|
- `systems/mnemosyne.md` — Database Stats refreshed to live 2026-10-04 numbers; the 2026-08-02 "vector coverage gap" marked RESOLVED (retired as a current issue); new Known Issues subsection for the FTS-only degradation plus the doctor caveat; History entry; embedding-coverage job `b63ed5817212` documented.
|
||||||
|
- `systems/hermes-agent.md` — version corrected to `v0.21.5+6169.gab86a20` (was `v0.21.5+3837.gbfda74c`), managed-runtime count corrected 10 → 3, guard description extended, profile line corrected (no `profiles/default/`; `ops` is the only named sub-profile), Known Issues bullet added.
|
||||||
|
- `systems/memory-backend.md` — Known Issues bullet for the FTS-only degradation.
|
||||||
|
- `systems/scheduled-tasks.md` — 2026-10-04 re-verification (30 jobs, failing set still 6, no new failures in the window), `79f9bb41ecf5` due 2026-10-04 18:00; bare-`telegram` delivery warnings confirmed live.
|
||||||
|
- `agents/hermes-production.md` — version, Python, profile, provider routing, cron count and messaging refreshed; stale `custom:litellm` / `freellmapi-auto` references replaced.
|
||||||
|
- `agents/hermes-chaos.md` — CT450 version bump recorded, no-SSH access path added, Mattermost failure noted.
|
||||||
|
- `infrastructure/hosts.md` — CT450 row updated to v0.21.5.
|
||||||
|
- `runbooks/update-hermes-safely.md` — new "Second instance — CT450" section (access path + pre-flight checks).
|
||||||
|
- `runbooks/provider-health.md` — stale v0.19.0 version reference corrected.
|
||||||
|
- `index.md` — last-updated bumped; both new incidents linked; cron-fleet line re-dated.
|
||||||
|
|
||||||
|
**Note** — `mnemosyne doctor` on this host prints `vec0: present_but_unloadable` and marks the vector tiers unavailable. That is a Python-3.11-legacy-`venv` artefact, not the gateway's state. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`.
|
||||||
|
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ title: Provider Health Check
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-04
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-07-22
|
||||||
last_tested: 2026-07-22
|
last_tested: 2026-07-22
|
||||||
confidence: high
|
confidence: high
|
||||||
@@ -84,7 +84,7 @@ Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or
|
|||||||
- Gateway journal shows no red alarm pattern.
|
- Gateway journal shows no red alarm pattern.
|
||||||
|
|
||||||
## Current install references
|
## Current install references
|
||||||
- Hermes: v0.19.0, git install: `/home/hermes/.hermes/hermes-agent`
|
- Hermes: v0.21.5+6169.gab86a20 (2026.9.24), git install: `/home/hermes/.hermes/hermes-agent`
|
||||||
- LiteLLM base URL: `http://litellm:4000/v1`
|
- LiteLLM base URL: `http://litellm:4000/v1`
|
||||||
- OmniRoute base URL: `http://omniroute:20128/v1`
|
- OmniRoute base URL: `http://omniroute:20128/v1`
|
||||||
- OmniRoute direct IP: `100.88.81.19:20128`
|
- OmniRoute direct IP: `100.88.81.19:20128`
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ title: Update Hermes safely
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-04
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-07-22
|
||||||
last_tested: 2026-07-22
|
last_tested: 2026-07-22
|
||||||
confidence: high
|
confidence: high
|
||||||
@@ -93,10 +93,30 @@ hermes dashboard restart
|
|||||||
- If update warns about config migration, revert the backup and re-run manually
|
- If update warns about config migration, revert the backup and re-run manually
|
||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
- Hermes version on this install: v0.19.0 (2026.7.20)
|
- Hermes version on this install: v0.21.5+6169.gab86a20 (2026.9.24); install tree rebuilt 2026-10-03 10:11 AWST
|
||||||
- Install directory: `/home/hermes/.hermes/hermes-agent`
|
- Install directory: `/home/hermes/.hermes/hermes-agent`
|
||||||
- Update preserves venv by default unless `--force-venv` is used
|
- Update preserves venv by default unless `--force-venv` is used
|
||||||
|
|
||||||
|
## Second instance — CT450 (`hermesagent`), updated 2026-10-03 (v0.19.0 → v0.21.5)
|
||||||
|
|
||||||
|
CT450 has **no SSH**. Reach it from the PVE host:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh root@100.96.100.82
|
||||||
|
pct exec 450 -- <command> # or: pct enter 450
|
||||||
|
pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"
|
||||||
|
```
|
||||||
|
|
||||||
|
`pct exec 450 -- hermes ...` fails (`Failed to exec "hermes"`): the binary is not on `lxc-attach`'s PATH. Use the absolute path or `su - hermes -c` (the dash gives a login shell, which sets `~/.local/bin`). A `hermes` login shell also prints a `hermes-setup` banner before output.
|
||||||
|
|
||||||
|
**Pre-flight before running the update** (each of these cost one failed run on 2026-10-03):
|
||||||
|
1. `clang` installed — source-compiled Python dependencies will not build without it (`pct exec 450 -- bash -lc "which clang"`).
|
||||||
|
2. No stale `.install.lock` / `.lazy-refresh-incomplete` in the Hermes home.
|
||||||
|
3. `~/.hermes/hermes-agent` owned by `hermes:hermes` (a root-owned tree from an earlier root-run update breaks the in-place build).
|
||||||
|
4. Host Caddy/`keyring` file permissions sane (`0600` keyring blocked a step mid-update).
|
||||||
|
|
||||||
|
**After:** `pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"`, confirm CT460 production is untouched (separate config/skills/cron/memory). Full record: [[incidents/2026-10-03-ct450-hermes-update]].
|
||||||
|
|
||||||
## Last tested
|
## Last tested
|
||||||
2026-07-22 on debian 13, git-based Hermes install
|
2026-07-22 on debian 13, git-based Hermes install
|
||||||
|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Hermes Agent
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-03
|
updated: 2026-10-04
|
||||||
verified_on: 2026-10-03
|
verified_on: 2026-10-04
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system, hermes, agent, llm]
|
tags: [system, hermes, agent, llm]
|
||||||
sources: [live-hermes-version, live-config, live-skill-inventory]
|
sources: [live-hermes-version, live-config, live-skill-inventory]
|
||||||
@@ -16,16 +16,16 @@ sources: [live-hermes-version, live-config, live-skill-inventory]
|
|||||||
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
|
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
|
||||||
|
|
||||||
## Current Status
|
## Current Status
|
||||||
- **Version:** v0.21.5+3837.gbfda74c (2026.9.24), upstream `27062c34` — verified 2026-09-30
|
- **Version:** v0.21.5+6169.gab86a20 (2026.9.24), upstream `ab86a20a` — verified live 2026-10-04. Same semver token as the earlier `v0.21.5+3837.gbfda74c` (`27062c34`), but a newer build: the install tree was rebuilt 2026-10-03 10:11 AWST.
|
||||||
- **Install method:** git
|
- **Install method:** git
|
||||||
- **Python:** 3.11.15
|
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
|
||||||
- **Upstream status:** Update available (`hermes update`)
|
- **Upstream status:** tree rebuilt 2026-10-03 from upstream `ab86a20a`; update availability not re-checked since.
|
||||||
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
|
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
|
||||||
- **Managed runtimes:** 10 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv`. Dependency guard `hermes-runtime-deps-guard.sh` verifies `dotenv` + `mnemosyne` in every venv every 15 min — see [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
- **Managed runtimes:** 3 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv` (was 10 — the 2026-10-03 rebuild pruned superseded environments and created two new ones at 09:59/10:00; the third survives from 2026-09-28). `hermes-runtime-deps-guard.sh` runs every 15 min via `mnemosyne-watchdog.py` and now asserts `dotenv` + `mnemosyne` in the installer python *and every managed venv*, **plus** `fastembed` + `FASTEMBED_CACHE_PATH` + the local model dir in the bootstrap-selected venv — see [[incidents/2026-09-28-hermes-managed-runtime-deps]] and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||||||
- **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions
|
- **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions
|
||||||
- **Binary:** `/home/hermes/.local/bin/hermes`
|
- **Binary:** `/home/hermes/.local/bin/hermes`
|
||||||
- **Config:** `~/.hermes/config.yaml`
|
- **Config:** `~/.hermes/config.yaml`
|
||||||
- **Profile:** `default` (`~/.hermes/profiles/default/`)
|
- **Profile:** `default` (top-level config `~/.hermes/config.yaml`; the only named sub-profile on this host is `ops` at `~/.hermes/profiles/ops/`)
|
||||||
|
|
||||||
## Location
|
## Location
|
||||||
- **Host:** Local Linux host (no containerized Hermes runtime confirmed)
|
- **Host:** Local Linux host (no containerized Hermes runtime confirmed)
|
||||||
@@ -131,6 +131,7 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
|
|||||||
|
|
||||||
## Known Issues
|
## Known Issues
|
||||||
- Managed-runtime venvs previously drifted on dependencies, breaking cron workers — now guarded every 15 min. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
- Managed-runtime venvs previously drifted on dependencies, breaking cron workers — now guarded every 15 min. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||||
|
- **Vector recall silently ran keyword-only until 2026-10-03 (repaired).** `fastembed` was missing from every uv-managed runtime and `FASTEMBED_CACHE_PATH` was unset, so the dense leg of `mnemosyne_recall` returned no candidates and recall degraded to FTS5 — with no error raised. Repaired 2026-10-03 and the guard extended (above). Live coverage 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%). See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||||||
- **Four cron jobs cannot self-recover — three pinned to models that no longer exist upstream, plus a credential fault:** `1a8e90ea2271` → `qwen3.5:397b` retired 2026-09-25 (HTTP 410); `dca8482e4f76` → `gpt-5.6-terra` plan-gated (403 `MODEL_NOT_IN_PLAN`); `fad5a86979d4` + `0e753e9720fd` → `deepseek-v4-flash` absent from the NVIDIA live catalog (HTTP 400); and `06a0a1c8704f` + `1b23c042e17d` → the `cliprox/terra` provider *connection* auth expired (HTTP 401), now confirmed on 2 of the 3 jobs sharing that pin. Repinning to the `smart-route` alias is the durable fix for the model faults; the cliprox pair needs the OmniRoute connection re-authenticated first. See [[systems/scheduled-tasks]], [[systems/model-providers]] and [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
- **Four cron jobs cannot self-recover — three pinned to models that no longer exist upstream, plus a credential fault:** `1a8e90ea2271` → `qwen3.5:397b` retired 2026-09-25 (HTTP 410); `dca8482e4f76` → `gpt-5.6-terra` plan-gated (403 `MODEL_NOT_IN_PLAN`); `fad5a86979d4` + `0e753e9720fd` → `deepseek-v4-flash` absent from the NVIDIA live catalog (HTTP 400); and `06a0a1c8704f` + `1b23c042e17d` → the `cliprox/terra` provider *connection* auth expired (HTTP 401), now confirmed on 2 of the 3 jobs sharing that pin. Repinning to the `smart-route` alias is the durable fix for the model faults; the cliprox pair needs the OmniRoute connection re-authenticated first. See [[systems/scheduled-tasks]], [[systems/model-providers]] and [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||||
- **The daily skills stock-take is not running.** Job `1a8e90ea2271` is blocked on the retired model above, so [[skills-index]] and the freshness audit are only refreshed when `wiki-maintain.sh` is run by hand (last done 2026-10-03). The script itself is healthy — only the scheduled wrapper is broken.
|
- **The daily skills stock-take is not running.** Job `1a8e90ea2271` is blocked on the retired model above, so [[skills-index]] and the freshness audit are only refreshed when `wiki-maintain.sh` is run by hand (last done 2026-10-03). The script itself is healthy — only the scheduled wrapper is broken.
|
||||||
- **Telegram thread delivery failures — root-caused and fixed 2026-09-30.** A bare `deliver: "telegram"` resolves to the home DM (`1793951355`), where group forum topic IDs cannot exist; the send hard-failed and the message was lost. Two jobs were retargeted to `telegram:-1004321904721:102` and verified. Three jobs still use bare `telegram` targets. See [[systems/messaging-integrations]].
|
- **Telegram thread delivery failures — root-caused and fixed 2026-09-30.** A bare `deliver: "telegram"` resolves to the home DM (`1793951355`), where group forum topic IDs cannot exist; the send hard-failed and the message was lost. Two jobs were retargeted to `telegram:-1004321904721:102` and verified. Three jobs still use bare `telegram` targets. See [[systems/messaging-integrations]].
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Memory Backend
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-08-01
|
updated: 2026-10-04
|
||||||
verified_on: 2026-08-01
|
verified_on: 2026-10-04
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [memory, mnemosyne, agent]
|
tags: [memory, mnemosyne, agent]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -43,6 +43,7 @@ Persist and recall agent memory across sessions.
|
|||||||
## Known Issues
|
## Known Issues
|
||||||
- Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods.
|
- Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods.
|
||||||
- The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
- The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
||||||
|
- **Vector recall degraded silently until 2026-10-03** — `fastembed` was missing from every uv-managed runtime, so `mnemosyne_recall` fell back to FTS5 keyword scoring with no error raised. Repaired 2026-10-03; the 15-min dependency guard now asserts `fastembed` and the model cache path. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||||||
|
|
||||||
**Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]].
|
**Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]].
|
||||||
|
|
||||||
|
|||||||
+25
-14
@@ -3,8 +3,8 @@ title: Mnemosyne
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-01
|
updated: 2026-10-04
|
||||||
verified_on: 2026-10-01
|
verified_on: 2026-10-04
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [memory, mnemosyne, backend]
|
tags: [memory, mnemosyne, backend]
|
||||||
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
||||||
@@ -33,25 +33,34 @@ Primary agent memory store for Hermes.
|
|||||||
- **Embedding model:** `BAAI/bge-small-en-v1.5` (384 dims)
|
- **Embedding model:** `BAAI/bge-small-en-v1.5` (384 dims)
|
||||||
- **Embeddings via API:** false (local fastembed)
|
- **Embeddings via API:** false (local fastembed)
|
||||||
|
|
||||||
## Database Stats (2026-08-02)
|
## Database Stats (live, 2026-10-04)
|
||||||
| Table | Rows | Notes |
|
| Table | Rows | Notes |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `working_memory` | 1,476 | Active memories |
|
| `working_memory` | 752 | Active memories |
|
||||||
| `episodic_memory` | 458 | Consolidated summaries |
|
| `episodic_memory` | 1,039 | Consolidated summaries |
|
||||||
| `memory_embeddings` | 158 | Only working memories — **0 episodic** |
|
| `memory_embeddings` | 739 | Working-tier embeddings |
|
||||||
| `memory` (legacy) | 2,181 | Old table, not migrated |
|
| `memory` (legacy) | 2,181 | Old table, not migrated |
|
||||||
| `vec_working` | 159 | Vector index rows |
|
| `vec_working` / `vec_working_rowids` | 739 | Working vector index |
|
||||||
|
| `vec_episodes` / `vec_episodes_rowids` | 1,035 | Episodic vector index |
|
||||||
|
| `vec_facts_rowids` | 0 | `memoria_facts` is intentionally left unvectorised |
|
||||||
| Shared surface | 0 | Empty — cross-agent sharing not functional |
|
| Shared surface | 0 | Empty — cross-agent sharing not functional |
|
||||||
|
|
||||||
|
**Embedding coverage (live, 2026-10-04):** 1,759/2,194 memories embedded (80.2%) — working **726/752 (96.5%)**, episodic **1,033/1,039 (99.4%)**; 13 embeddings generated in the previous 24h. Working coverage is above the 95% alert threshold enforced by the Embedding Coverage job (`b63ed5817212`, every 6h, Telegram topic 102).
|
||||||
|
|
||||||
## Known Issues
|
## Known Issues
|
||||||
|
|
||||||
### Vector embedding coverage gap
|
### Vector embedding coverage gap — RESOLVED (2026-08-02)
|
||||||
- **Only 159/1,476 working memories (10.8%)** have vector embeddings
|
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
|
||||||
- **0/458 episodic memories** have embeddings
|
- **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only.
|
||||||
- Semantic search (`mnemosyne_recall`) falls back to FTS5 keyword matching for ~90% of working and 100% of episodic memories
|
- Standard repair if coverage drops below 95% working: `~/.hermes/hermes-agent/venv/bin/mnemosyne reindex --yes` (the Embedding Coverage job `b63ed5817212` triggers the same path every 6h).
|
||||||
- This means recall quality is degraded — vector similarity can only search a tiny fraction of the memory store
|
|
||||||
- **Root cause:** The embedding pipeline (fastembed) never processed the bulk of memories. The `vec_working` table was rebuilt once (Jul 18) but only captured 159 rows.
|
### Vector recall silently ran FTS-only (2026-10-03 — repaired)
|
||||||
- **Fix needed:** Backfill embeddings for all working and episodic memories. This requires running the embedding pipeline across the full dataset.
|
- `fastembed` was absent from **every** uv-managed runtime, so the dense leg of `mnemosyne_recall` produced no candidates and recall fell back to FTS5/keyword scoring **without raising an error**. Sessions kept working, which is why it went unnoticed.
|
||||||
|
- Compounded by `FASTEMBED_CACHE_PATH` being unset in `~/.hermes/.env`, and by the embedding worker sometimes launching in isolated (`python -I`) mode, which does not inherit `PYTHONPATH`.
|
||||||
|
- **Repaired 2026-10-03:** `fastembed` installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min alongside its existing `dotenv`/`mnemosyne` checks.
|
||||||
|
- **CLI caveat — do not misread `mnemosyne doctor`.** Doctor and the recall CLI run under the legacy `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable; doctor therefore prints `vec0: present_but_unloadable` and marks the vector tiers `unavailable` even while the gateway embeds normally. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line.
|
||||||
|
- Benign noise: `pthread_setaffinity_np failed … mask: {0,}` from onnxruntime in `mcp-stderr.log` is a container affinity warning emitted on every model load — not an error.
|
||||||
|
- See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||||||
|
|
||||||
### Profile isolation chosen over shared surface
|
### Profile isolation chosen over shared surface
|
||||||
|- The ops profile was enabled with mnemosyne on 2026-08-02
|
|- The ops profile was enabled with mnemosyne on 2026-08-02
|
||||||
@@ -75,9 +84,11 @@ Primary agent memory store for Hermes.
|
|||||||
- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
|
- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
|
||||||
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
|
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
|
||||||
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception
|
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception
|
||||||
|
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
|
||||||
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
|
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
|
||||||
|
|
||||||
## History and Evidence
|
## History and Evidence
|
||||||
|
- **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||||||
- **2026-09-30 memory-hygiene pass:** 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (`hermes_cron_model_default` → `custom:omni-101`, job-radar model). No unresolved conflicts left in the working tier.
|
- **2026-09-30 memory-hygiene pass:** 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (`hermes_cron_model_default` → `custom:omni-101`, job-radar model). No unresolved conflicts left in the working tier.
|
||||||
- **2026-08-19 Mnemosyne Watchdog incident:** Watchdog job (`96eb95ce12f2`) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See [[systems/scheduled-tasks]] Known Issues section.
|
- **2026-08-19 Mnemosyne Watchdog incident:** Watchdog job (`96eb95ce12f2`) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See [[systems/scheduled-tasks]] Known Issues section.
|
||||||
- **2026-08-18 incident:** Working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See [[incidents/2026-08-18-mnemosyne-capture-pipeline]].
|
- **2026-08-18 incident:** Working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See [[incidents/2026-08-18-mnemosyne-capture-pipeline]].
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-03
|
updated: 2026-10-04
|
||||||
verified_on: 2026-10-03
|
verified_on: 2026-10-04
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, scheduled, automation]
|
tags: [cron, scheduled, automation]
|
||||||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||||
@@ -25,6 +25,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
|||||||
- **Resolved since the last entry:** 1 — `1b23c042e17d`'s stale `delivery_failed` flag cleared exactly as predicted. Its 2026-10-02 run delivered to topic `102` (`delivery_outcome: delivered`); it then failed on the model pin, above.
|
- **Resolved since the last entry:** 1 — `1b23c042e17d`'s stale `delivery_failed` flag cleared exactly as predicted. Its 2026-10-02 run delivered to topic `102` (`delivery_outcome: delivered`); it then failed on the model pin, above.
|
||||||
- **Shared model pin at risk:** 3 enabled jobs run `cliprox/terra` — `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`. **Two of the three have now failed on that pin.** The third (`79f9bb41ecf5`) last ran 2026-09-27 — before the credential lapsed — and is next due 2026-10-04 18:00. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
- **Shared model pin at risk:** 3 enabled jobs run `cliprox/terra` — `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`. **Two of the three have now failed on that pin.** The third (`79f9bb41ecf5`) last ran 2026-09-27 — before the credential lapsed — and is next due 2026-10-04 18:00. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||||
- **Streak recount (2026-10-03):** recounted directly from `~/.hermes/cron/executions.db`. `dca8482e4f76` 6 → 7 (one more daily run since), and `1a8e90ea2271` 8 → **7** — the ×8 carried in the 2026-10-02 entry was an overcount: that job has exactly 7 recorded runs and all 7 failed.
|
- **Streak recount (2026-10-03):** recounted directly from `~/.hermes/cron/executions.db`. `dca8482e4f76` 6 → 7 (one more daily run since), and `1a8e90ea2271` 8 → **7** — the ×8 carried in the 2026-10-02 entry was an overcount: that job has exactly 7 recorded runs and all 7 failed.
|
||||||
|
- **Re-verified 2026-10-04 00:12 AWST — no change:** still 30 jobs (27 enabled / 3 disabled) and the same **6** failing jobs; no new failure event landed inside the audit window. `79f9bb41ecf5` (Weekly Review, the last untested `cliprox/terra` job) drops at **2026-10-04 18:00 AWST** — its result settles whether that pin is 3/3 broken. `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found; delivered without thread_id`), confirmed live in `hermes cron list`.
|
||||||
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03)
|
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03)
|
||||||
|
|
||||||
## Job Table
|
## Job Table
|
||||||
@@ -87,7 +88,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
|||||||
|
|
||||||
### Open
|
### Open
|
||||||
- **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 10-02, streak 7):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change; it cannot self-recover. See [[runbooks/job-radar]].
|
- **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 10-02, streak 7):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change; it cannot self-recover. See [[runbooks/job-radar]].
|
||||||
- **`cliprox/terra` provider connection auth expired — 2 of 3 jobs confirmed (streak 1 each):** Jobs `06a0a1c8704f` (2026-10-01) and `1b23c042e17d` (2026-10-02) both failed with `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Failure notices reached their topics; no output was produced. This is a *credential/connection* problem at the proxy, not a missing model — it will not self-recover until the connection is re-authenticated. `79f9bb41ecf5` is the remaining untested job on this pin (next run 2026-10-04 18:00). Re-authenticate the OmniRoute connection before repinning anything. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
- **`cliprox/terra` provider connection auth expired — 2 of 3 jobs confirmed (streak 1 each):** Jobs `06a0a1c8704f` (2026-10-01) and `1b23c042e17d` (2026-10-02) both failed with `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Failure notices reached their topics; no output was produced. This is a *credential/connection* problem at the proxy, not a missing model — it will not self-recover until the connection is re-authenticated. `79f9bb41ecf5` is the remaining untested job on this pin (next run 2026-10-04 18:00 — re-check on 2026-10-05). Re-authenticate the OmniRoute connection before repinning anything. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||||
- **A delivered notice is not a successful job:** both `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output.
|
- **A delivered notice is not a successful job:** both `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output.
|
||||||
- **wiki-stocktake-maintenance — HTTP 410, model retired (streak 7):** Job `1a8e90ea2271` is pinned to `ollama-cloud / qwen3.5:397b`, which Ollama retired on 2026-09-25 (`qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT`). It cannot self-recover — it needs a new model pin. **The job's script is fine:** `wiki-maintain.sh` was run by hand on 2026-10-03 and completed cleanly, regenerating [[systems/skills-index]] (344 skills / 70 categories) and refreshing the freshness audit. Only the scheduled wrapper is broken — so every non-model step of the daily stocktake needs a manual run until it is repinned.
|
- **wiki-stocktake-maintenance — HTTP 410, model retired (streak 7):** Job `1a8e90ea2271` is pinned to `ollama-cloud / qwen3.5:397b`, which Ollama retired on 2026-09-25 (`qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT`). It cannot self-recover — it needs a new model pin. **The job's script is fine:** `wiki-maintain.sh` was run by hand on 2026-10-03 and completed cleanly, regenerating [[systems/skills-index]] (344 skills / 70 categories) and refreshing the freshness audit. Only the scheduled wrapper is broken — so every non-model step of the daily stocktake needs a manual run until it is repinned.
|
||||||
- **Facebook Birthday scrapes — HTTP 400, model missing from catalog (streak 3 / 1):** Job `fad5a86979d4` (weekly) is pinned to `nvidia/deepseek-ai/deepseek-v4-flash`; the provider reports the model is `not available in the active live catalog for provider 'nvidia'`. **Confirmed as a shared-pin issue on 2026-10-01**, when sibling job `0e753e9720fd` (monthly) failed on the same pin at its own 09:00 run. Safest durable fix is repinning both to a routing alias (`smart-route`) instead of a raw model name.
|
- **Facebook Birthday scrapes — HTTP 400, model missing from catalog (streak 3 / 1):** Job `fad5a86979d4` (weekly) is pinned to `nvidia/deepseek-ai/deepseek-v4-flash`; the provider reports the model is `not available in the active live catalog for provider 'nvidia'`. **Confirmed as a shared-pin issue on 2026-10-01**, when sibling job `0e753e9720fd` (monthly) failed on the same pin at its own 09:00 run. Safest durable fix is repinning both to a routing alias (`smart-route`) instead of a raw model name.
|
||||||
|
|||||||
Reference in New Issue
Block a user