wiki audit 2026-10-04: CT450 v0.21.5 update incident, mnemosyne vector-recall FTS-only repair + coverage refresh, managed runtimes 10->3, doctor vec0 caveat

This commit is contained in:
Tony0410
2026-10-04 00:15:35 +08:00
parent b9f69d3f75
commit 78048bdf0b
14 changed files with 241 additions and 49 deletions
+5 -2
View File
@@ -3,8 +3,8 @@ title: Hermes Chaos
type: system type: system
status: active status: active
created: 2026-08-15 created: 2026-08-15
updated: 2026-08-17 updated: 2026-10-04
verified_on: 2026-08-17 verified_on: 2026-10-04
confidence: high confidence: high
tags: [hermes, chaos, ct450, experimental, agent] tags: [hermes, chaos, ct450, experimental, agent]
sources: [live-probe, config-inspection] sources: [live-probe, config-inspection]
@@ -19,6 +19,7 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod
## Location ## Location
- **Container:** LXC CT450 (`hermesagent`) - **Container:** LXC CT450 (`hermesagent`)
- **Access:** **no SSH** — go through the PVE host: `ssh root@100.96.100.82`, then `pct exec 450 -- <cmd>` (or `pct enter 450`). `hermes` is not on `pct exec`'s default PATH; use `su - hermes -c "/home/hermes/.local/bin/hermes …"`.
- **Tailscale IP:** `100.98.205.112` - **Tailscale IP:** `100.98.205.112`
- **Hostname:** `hermesagent` - **Hostname:** `hermesagent`
- **Software:** hermes-ui (Vite web app for browser use, port 5174) - **Software:** hermes-ui (Vite web app for browser use, port 5174)
@@ -27,6 +28,8 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod
## Current State ## Current State
- **Status:** Active (experimental) - **Status:** Active (experimental)
- **Hermes version:** `v0.21.5+6170.ga510d6c` (2026.9.24), upstream `a510d6c6` — **updated from v0.19.0 on 2026-10-03**, first update since July; 3 commits behind upstream. Python 3.14.7. See [[incidents/2026-10-03-ct450-hermes-update]].
- **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]].
- **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01) - **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01)
- **Duplicate cron jobs:** 11 paused (no post-pause execution records since 2026-08-01) - **Duplicate cron jobs:** 11 paused (no post-pause execution records since 2026-08-01)
- **Production counterparts:** CT460 has 11 enabled jobs with successful latest runs - **Production counterparts:** CT460 has 11 enabled jobs with successful latest runs
+12 -10
View File
@@ -3,8 +3,8 @@ title: Hermes Production
type: system type: system
status: active status: active
created: 2026-08-15 created: 2026-08-15
updated: 2026-08-15 updated: 2026-10-04
verified_on: 2026-08-15 verified_on: 2026-10-04
confidence: high confidence: high
tags: [hermes, production, ct460, agent] tags: [hermes, production, ct460, agent]
sources: [live-probe, config-inspection] sources: [live-probe, config-inspection]
@@ -27,11 +27,11 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
## Current State ## Current State
- **Status:** Active - **Status:** Active
- **Hermes version:** v0.19.0 - **Hermes version:** `v0.21.5+6169.gab86a20` (2026.9.24), upstream `ab86a20a`; install tree rebuilt 2026-10-03 10:11 AWST — verified live 2026-10-04
- **Python:** 3.11.15 - **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
- **Install directory:** `/home/hermes/.hermes/hermes-agent` - **Install directory:** `/home/hermes/.hermes/hermes-agent`
- **Config:** `~/.hermes/config.yaml` - **Config:** `~/.hermes/config.yaml`
- **Profile:** `default` (`~/.hermes/profiles/default/`) - **Profile:** `default` — top-level config; **no `profiles/default/` directory exists** (`ops` at `~/.hermes/profiles/ops/` is the only named sub-profile)
- **Soul:** Rhino — trickster-mentor in a green jacket 🎩 - **Soul:** Rhino — trickster-mentor in a green jacket 🎩
## Profiles ## Profiles
@@ -43,9 +43,10 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
## Providers & Routing ## Providers & Routing
- **Default provider:** `custom:litellm` → `http://litellm:4000/v1` - **Default provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — as attested 2026-09-30; the `custom:omni` provider key no longer exists
- **Fallback provider:** `custom:omniroute` → `http://omniroute:20128/v1` - **Default model:** `smart-route` (previously `freellmapi-auto`)
- **Default model:** `freellmapi-auto` - **Fallback providers:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`, `custom / litellm:4000`
- **Auxiliary curator model:** `openai-codex / gpt-5.6-terra`
- See [[systems/model-providers]] for full breakdown. - See [[systems/model-providers]] for full breakdown.
## Memory ## Memory
@@ -57,7 +58,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
## Scheduled Tasks ## Scheduled Tasks
- **Cron jobs:** 25+ active jobs - **Cron jobs:** 30 (27 enabled / 3 disabled; **6 failing** as of 2026-10-04)
- Delivery targets: Telegram (primary), Discord, origin, local - Delivery targets: Telegram (primary), Discord, origin, local
- Notable: Birthday checker, job radar, newsletter digests, mnemosyne health/watchdog/backup - Notable: Birthday checker, job radar, newsletter digests, mnemosyne health/watchdog/backup
- See [[systems/scheduled-tasks]] for full table. - See [[systems/scheduled-tasks]] for full table.
@@ -65,6 +66,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
## Messaging ## Messaging
- Telegram, Discord, Slack, Email, Mattermost, ntfy - Telegram, Discord, Slack, Email, Mattermost, ntfy
- **Mattermost is currently unreachable** — the gateway logs `mattermost connect timed out after 30s, next retry in 300s` (live, 2026-10-04) and the Mattermost host does not answer. Treat Mattermost delivery as down until probed.
- See [[systems/messaging-integrations]] for details. - See [[systems/messaging-integrations]] for details.
## Key Differences from Chaos ## Key Differences from Chaos
@@ -72,7 +74,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
| Aspect | Production (CT460) | Chaos (CT450) | | Aspect | Production (CT460) | Chaos (CT450) |
|--------|-------------------|---------------| |--------|-------------------|---------------|
| Purpose | Stable daily operations | Experimental/testing | | Purpose | Stable daily operations | Experimental/testing |
| Cron jobs | 25+ enabled | 11 paused duplicates | | Cron jobs | 30 (27 enabled / 3 disabled) | 11 paused duplicates |
| Config changes | Requires verification | Safe to experiment | | Config changes | Requires verification | Safe to experiment |
| Memory | Production Mnemosyne | Separate state | | Memory | Production Mnemosyne | Separate state |
+11 -3
View File
@@ -3,8 +3,8 @@ title: Current State
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-03 updated: 2026-10-04
verified_on: 2026-10-03 verified_on: 2026-10-04
review_after: 2026-10-31 review_after: 2026-10-31
confidence: high confidence: high
tags: [system] tags: [system]
@@ -12,7 +12,15 @@ sources: [live-config, live-cron-list, live-hermes-version]
--- ---
# Current State # Current State
Updated: 2026-10-03 Updated: 2026-10-04
## Recent Changes (2026-10-04)
- **Both Hermes instances updated on 2026-10-03, from opposite directions.** CT460 (production) was rebuilt to `v0.21.5+6169.gab86a20` (upstream `ab86a20a`, tree mtime 10:11 AWST) and CT450 (chaos) came up from `v0.19.0` to `v0.21.5+6170.ga510d6c` (upstream `a510d6c6`, tree mtime 10:22 AWST). Same semver token, different builds, separate state. CT450 now reports `Update available: 3 commits behind`. CT450 has no SSH — it is reached via `ssh root@100.96.100.82` + `pct exec 450`, and `hermes` is not on the default `pct exec` PATH. See [[incidents/2026-10-03-ct450-hermes-update]].
- **Vector recall had been silently running keyword-only — found and fixed 2026-10-03.** `fastembed` was missing from every uv-managed runtime, so the dense leg of `mnemosyne_recall` contributed no candidates and recall fell back to FTS5 **without erroring**. Fixed the same day: fastembed 0.8.1 installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min. **Live coverage 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%)**, 13 embeddings generated in the previous 24h. The previous "coverage gap" entry (159/1,476) is retired — it was resolved by the 2026-08-02 reindex. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] and [[systems/mnemosyne]].
- **Managed runtimes pruned 10 → 3.** Only 3 uv-managed venvs remain under `~/.hermes/installs/019d0d114d22b7e6/environments/` (two created 2026-10-03 09:59/10:00, one surviving from 2026-09-28). Any note that says "ten managed venvs" is stale.
- **Cron fleet re-verified 2026-10-04 00:12 AWST:** 30 jobs (27 enabled / 3 disabled), failing set unchanged at 6, no new failure inside the window. `79f9bb41ecf5` — the last untested `cliprox/terra` job — is due 2026-10-04 18:00 AWST. See [[systems/scheduled-tasks]].
- **Mattermost is unreachable from CT460 right now** (`mattermost connect timed out after 30s, next retry in 300s`, every 5 min, live 2026-10-04). Host-side condition; Mattermost delivery should be treated as down until probed.
- **Wiki maintenance gotcha worth remembering:** `mnemosyne doctor` runs under the legacy Python 3.11 `venv/`, where `sqlite_vec` is not importable, so it prints `vec0: present_but_unloadable` and marks vector tiers `unavailable` *even when the gateway is embedding fine*. Judge vector health with `check_embedding_coverage.py`, never with doctor's `vec0` line.
## Recent Changes (2026-10-03) ## Recent Changes (2026-10-03)
- **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set stays at **6**, but the composition changed: `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) failed on 2026-10-02 with the *identical* HTTP 401 the cliprox incident predicted — `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. That makes **2 of the 3** `cliprox/terra` jobs confirmed broken, so the provider-connection auth expiry is a real credential fault, not a one-job fluke. `79f9bb41ecf5` (Weekly Review, Sun 18:00) is the last untested one — its only run (2026-09-27) predates the expiry and succeeded; next due **2026-10-04 18:00**. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. - **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set stays at **6**, but the composition changed: `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) failed on 2026-10-02 with the *identical* HTTP 401 the cliprox incident predicted — `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. That makes **2 of the 3** `cliprox/terra` jobs confirmed broken, so the provider-connection auth expiry is a real credential fault, not a one-job fluke. `79f9bb41ecf5` (Weekly Review, Sun 18:00) is the last untested one — its only run (2026-09-27) predates the expiry and succeeded; next due **2026-10-04 18:00**. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
@@ -0,0 +1,59 @@
---
title: CT450 Hermes Update to v0.21.5
type: incident
status: resolved
created: 2026-10-04
updated: 2026-10-04
verified_on: 2026-10-04
confidence: high
tags: [hermes, ct450, chaos, update, incident]
sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
---
# CT450 Hermes Update to v0.21.5
## Summary
The Chaos instance (LXC CT450, `hermesagent`) was updated **v0.19.0 → v0.21.5** on 2026-10-03 — its first update since July. CT460 (production) was rebuilt the same morning, so both instances now carry the same semver token from separate builds.
## Result (live-verified 2026-10-04)
| | CT460 (production) | CT450 (chaos) |
|---|---|---|
| Version | `v0.21.5+6169.gab86a20` (2026.9.24) | `v0.21.5+6170.ga510d6c` (2026.9.24) |
| Upstream | `ab86a20a` | `a510d6c6` — 3 commits behind, `hermes update` available |
| Python | managed runtime 3.14 (legacy `venv/` 3.11.15) | 3.14.7 |
| Install tree mtime | 2026-10-03 10:11 AWST | 2026-10-03 10:22 AWST |
## Access
**CT450 has no SSH.** Reach it through the PVE host:
```bash
ssh root@100.96.100.82
pct exec 450 -- <command> # or: pct enter 450
pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"
```
Two gotchas that cost time:
- `pct exec 450 -- hermes ...` **fails** with `Failed to exec "hermes"` — the binary is not on `lxc-attach`'s default PATH. Use the absolute path, or `su - hermes -c` (the dash matters: only a login shell puts `~/.local/bin` on PATH).
- A `hermes` login shell prints a `hermes-setup` banner ahead of command output; parse with `| tail -n` rather than expecting clean stdout.
## Pre-flight checks (each cost one failed run on 2026-10-03)
1. **Build toolchain.** `clang` must be present *before* updating — source-compiled Python dependencies will not build without it. Verified present on CT450 (`/usr/bin/clang`, 2026-10-04).
2. **Stale update locks.** A leftover `.install.lock` / `.lazy-refresh-incomplete` in the Hermes home makes the updater refuse or silently skip the refresh step. Neither exists on CT450 now.
3. **Install-tree ownership.** `~/.hermes/hermes-agent` must stay owned by `hermes:hermes` (uid 1000); a root-owned tree left by an earlier root-run update fails the in-place build.
4. **Caddy keyring permissions.** Host-level Caddy/`keyring` file permissions (`0600`) blocked a step mid-update — check them before starting, not during.
## Verification
- `hermes --version` on CT450 → `Hermes Agent v0.21.5+6170.ga510d6c (2026.9.24) · upstream a510d6c6`, Python 3.14.7.
- Install tree mtime `2026-10-03 10:22` matches the update window.
- `clang` present; no stale locks present.
- CT460 production untouched — separate config, skills, cron and memory state.
## Follow-ups
- CT450 reports `Update available: 3 commits behind`. Harmless; re-run `hermes update` if the chaos box should track upstream exactly.
- CT450's Mattermost connection is failing (`mattermost connect timed out after 30s`, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See [[agents/hermes-chaos]].
## Related
- [[agents/hermes-chaos]]
- [[agents/hermes-production]]
- [[runbooks/update-hermes-safely]]
- [[infrastructure/hosts]]
@@ -0,0 +1,60 @@
---
title: Mnemosyne Vector Recall Ran FTS-Only
type: incident
status: resolved
created: 2026-10-04
updated: 2026-10-04
verified_on: 2026-10-04
confidence: high
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
---
# Mnemosyne Vector Recall Ran FTS-Only
## Summary
Until 2026-10-03, `mnemosyne_recall` was scoring **keyword-only**. `fastembed` was missing from every UV-managed runtime, so the dense (vector) leg of recall contributed no candidates. Nothing errored — recall still returned results, just worse ones. Detected and repaired 2026-10-03; re-verified live 2026-10-04.
## Impact
- Semantic similarity search silently degraded to keyword/`FTS5` matching.
- No user-visible error, no failed cron job, no alert. The failure mode of a missing embedding backend is **quiet recall degradation**, not a crash — which is why it survived unnoticed.
- Nothing was lost: the residue is *un-embedded* rows, not missing rows (26 working / 6 episodic as of 2026-10-04). New writes embed normally.
## Root Cause
Three overlapping conditions:
1. **`fastembed` absent from the managed runtimes.** The gateway runs from a uv-managed venv under `~/.hermes/installs/<install>/environments/<hash>/venv`, and none of them had `fastembed`. The *legacy* `/home/hermes/.hermes/hermes-agent/venv` had it (0.8.0), which is why CLI-side checks looked healthy while the gateway could not embed.
2. **`FASTEMBED_CACHE_PATH` unset** in `~/.hermes/.env`, so the local model directory was resolved implicitly rather than pinned.
3. **Isolated-mode launches.** The embedding worker can be launched with `python -I`, which ignores `PYTHONPATH`/`PYTHONHOME` — any path-based injection of the dependency does not survive that launch mode.
## Fix (2026-10-03)
- Installed `fastembed` (0.8.1) into the runtime the bootstrap actually selects.
- Pinned `FASTEMBED_CACHE_PATH` (models cached under `~/.hermes/cache/fastembed`, `BAAI/bge-small-en-v1.5`).
- Extended `~/.hermes/scripts/hermes-runtime-deps-guard.sh` — run every 15 min by `mnemosyne-watchdog.py` — so it now asserts, in the bootstrap-selected venv: `fastembed` importable, `FASTEMBED_CACHE_PATH` set, and the model directory present, on top of its existing `dotenv` + `mnemosyne` checks in the installer python and every managed venv.
- No gateway restart required.
## Verification (live, 2026-10-04)
```
Memories with embeddings: 1759 / 2194 (80.2%)
Working memory: 726 / 752 (96.5%)
Episodic memory: 1033 / 1039 (99.4%)
vec_episodes_rowids: 1035
vec_working_rowids: 739
Embeddings generated (24h): 13 (working: 7, episodic: 6)
```
`onnxruntime` threads now appear in `~/.hermes/logs/mcp-stderr.log` on each model load — direct evidence the embedding model is executing inside the gateway runtime. The dependency guard exits 0.
## Caveat — do not misread `mnemosyne doctor`
`mnemosyne doctor` and the recall CLI run under the **legacy** `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable. Doctor therefore prints `vec0: present_but_unloadable` and marks the working/episodic vector tiers `unavailable`, even while the gateway embeds normally. The managed Python 3.14 venvs cannot import `sqlite_vec` either when probed directly. **Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line.**
## Benign noise
`[E:onnxruntime:Default, env.cc:228 ThreadMain] pthread_setaffinity_np failed for thread: … mask: {0,}` — a container CPU-affinity warning emitted on every model load. Not an error; do not chase it.
## Follow-ups
- Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.
- Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.
## Related
- [[systems/mnemosyne]]
- [[systems/memory-backend]]
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]]
+4 -2
View File
@@ -1,6 +1,6 @@
# Anthony's Systems Wiki # Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure, > Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-10-03 > model routing and automation. Last updated: 2026-10-04
## Start here ## Start here
- [[current-state]] — Current systems status - [[current-state]] — Current systems status
@@ -28,10 +28,12 @@
### Integrations ### Integrations
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; 6 failing as of 2026-10-03) - [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; 6 failing as of 2026-10-04)
- [[systems/skills-index]] — Installed skills inventory (auto-regenerated daily; being refreshed by hand while the stocktake job is blocked) - [[systems/skills-index]] — Installed skills inventory (auto-regenerated daily; being refreshed by hand while the stocktake job is blocked)
## Incidents ## Incidents
- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
- [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired; **2 of 3 jobs sharing the pin now failed**, the third untested (open 2026-10-01) - [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired; **2 of 3 jobs sharing the pin now failed**, the third untested (open 2026-10-01)
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26) - [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26)
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min - [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min
+2 -2
View File
@@ -3,7 +3,7 @@ title: Known Hosts and Paths
type: infrastructure type: infrastructure
status: partial status: partial
created: 2026-07-22 created: 2026-07-22
updated: 2026-08-14 updated: 2026-10-04
verified_on: 2026-07-22 verified_on: 2026-07-22
confidence: medium confidence: medium
tags: [infrastructure, hosts, containers, paths] tags: [infrastructure, hosts, containers, paths]
@@ -21,7 +21,7 @@ sources: []
| omni | OmniRoute proxy | `http://omni:20128/v1` | Yes (from config, 2026-08-13) | | omni | OmniRoute proxy | `http://omni:20128/v1` | Yes (from config, 2026-08-13) |
| CT201 | Steel Browser, Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.166` (LAN) | Yes (live probe) | | CT201 | Steel Browser, Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.166` (LAN) | Yes (live probe) |
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net` | Yes | | CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net` | Yes |
| CT450 (hermesagent) | Hermes Agent v0.19.0, hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes | | CT450 (hermesagent) | Hermes Agent **v0.21.5+6170.ga510d6c** (updated 2026-10-03 from v0.19.0), hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes |
## Project Paths ## Project Paths
+24
View File
@@ -298,3 +298,27 @@
- **`systems/skills-index.md` regenerated 341 → 344** — including a new skill, `fleet-wide-failure-triage` (devops). This was done **by hand**: job `1a8e90ea2271` is still blocked on HTTP 410 (retired `qwen3.5:397b`) and cannot self-recover. `wiki-maintain.sh` itself is healthy and was run manually, also writing `raw/stocktake/freshness-20261003.txt` — recorded so the next reader knows the stocktake is on manual refresh, not on schedule. - **`systems/skills-index.md` regenerated 341 → 344** — including a new skill, `fleet-wide-failure-triage` (devops). This was done **by hand**: job `1a8e90ea2271` is still blocked on HTTP 410 (retired `qwen3.5:397b`) and cannot self-recover. `wiki-maintain.sh` itself is healthy and was run manually, also writing `raw/stocktake/freshness-20261003.txt` — recorded so the next reader knows the stocktake is on manual refresh, not on schedule.
- **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live under `wiki/`; `index.md` sits at the vault root. - **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live under `wiki/`; `index.md` sits at the vault root.
- No secrets written; all credential-shaped values in config remain redacted. - No secrets written; all credential-shaped values in config remain redacted.
## [2026-10-04] Nightly audit — CT450 update + Mnemosyne vector-recall repair
Reviewed window: 2026-10-03 00:12 → 2026-10-04 00:12 AWST.
**New pages**
- `incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md` — `mnemosyne_recall` had been scoring keyword-only: `fastembed` was absent from every uv-managed runtime, so the dense leg produced no candidates and recall fell back to FTS5 *without raising an error*. Repaired 2026-10-03; the runtime guard now asserts fastembed + `FASTEMBED_CACHE_PATH` + model dir every 15 min. Live 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%).
- `incidents/2026-10-03-ct450-hermes-update.md` — CT450 (`hermesagent`) updated v0.19.0 → v0.21.5, its first update since July. Records the no-SSH access path (`ssh root@100.96.100.82` → `pct exec 450 -- …`) and the four pre-flight checks (clang, stale locks, tree ownership, Caddy keyring perms).
**Updated**
- `current-state.md` — new Recent Changes (2026-10-04) block: the two findings above, the managed-runtime prune 10 → 3, cron re-verification, Mattermost unreachable, and the `mnemosyne doctor` / `sqlite_vec` misread gotcha.
- `systems/mnemosyne.md` — Database Stats refreshed to live 2026-10-04 numbers; the 2026-08-02 "vector coverage gap" marked RESOLVED (retired as a current issue); new Known Issues subsection for the FTS-only degradation plus the doctor caveat; History entry; embedding-coverage job `b63ed5817212` documented.
- `systems/hermes-agent.md` — version corrected to `v0.21.5+6169.gab86a20` (was `v0.21.5+3837.gbfda74c`), managed-runtime count corrected 10 → 3, guard description extended, profile line corrected (no `profiles/default/`; `ops` is the only named sub-profile), Known Issues bullet added.
- `systems/memory-backend.md` — Known Issues bullet for the FTS-only degradation.
- `systems/scheduled-tasks.md` — 2026-10-04 re-verification (30 jobs, failing set still 6, no new failures in the window), `79f9bb41ecf5` due 2026-10-04 18:00; bare-`telegram` delivery warnings confirmed live.
- `agents/hermes-production.md` — version, Python, profile, provider routing, cron count and messaging refreshed; stale `custom:litellm` / `freellmapi-auto` references replaced.
- `agents/hermes-chaos.md` — CT450 version bump recorded, no-SSH access path added, Mattermost failure noted.
- `infrastructure/hosts.md` — CT450 row updated to v0.21.5.
- `runbooks/update-hermes-safely.md` — new "Second instance — CT450" section (access path + pre-flight checks).
- `runbooks/provider-health.md` — stale v0.19.0 version reference corrected.
- `index.md` — last-updated bumped; both new incidents linked; cron-fleet line re-dated.
**Note** — `mnemosyne doctor` on this host prints `vec0: present_but_unloadable` and marks the vector tiers unavailable. That is a Python-3.11-legacy-`venv` artefact, not the gateway's state. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`.
+2 -2
View File
@@ -3,7 +3,7 @@ title: Provider Health Check
type: runbook type: runbook
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-04
verified_on: 2026-07-22 verified_on: 2026-07-22
last_tested: 2026-07-22 last_tested: 2026-07-22
confidence: high confidence: high
@@ -84,7 +84,7 @@ Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or
- Gateway journal shows no red alarm pattern. - Gateway journal shows no red alarm pattern.
## Current install references ## Current install references
- Hermes: v0.19.0, git install: `/home/hermes/.hermes/hermes-agent` - Hermes: v0.21.5+6169.gab86a20 (2026.9.24), git install: `/home/hermes/.hermes/hermes-agent`
- LiteLLM base URL: `http://litellm:4000/v1` - LiteLLM base URL: `http://litellm:4000/v1`
- OmniRoute base URL: `http://omniroute:20128/v1` - OmniRoute base URL: `http://omniroute:20128/v1`
- OmniRoute direct IP: `100.88.81.19:20128` - OmniRoute direct IP: `100.88.81.19:20128`
+22 -2
View File
@@ -3,7 +3,7 @@ title: Update Hermes safely
type: runbook type: runbook
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-04
verified_on: 2026-07-22 verified_on: 2026-07-22
last_tested: 2026-07-22 last_tested: 2026-07-22
confidence: high confidence: high
@@ -93,10 +93,30 @@ hermes dashboard restart
- If update warns about config migration, revert the backup and re-run manually - If update warns about config migration, revert the backup and re-run manually
## Notes ## Notes
- Hermes version on this install: v0.19.0 (2026.7.20) - Hermes version on this install: v0.21.5+6169.gab86a20 (2026.9.24); install tree rebuilt 2026-10-03 10:11 AWST
- Install directory: `/home/hermes/.hermes/hermes-agent` - Install directory: `/home/hermes/.hermes/hermes-agent`
- Update preserves venv by default unless `--force-venv` is used - Update preserves venv by default unless `--force-venv` is used
## Second instance — CT450 (`hermesagent`), updated 2026-10-03 (v0.19.0 → v0.21.5)
CT450 has **no SSH**. Reach it from the PVE host:
```bash
ssh root@100.96.100.82
pct exec 450 -- <command> # or: pct enter 450
pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"
```
`pct exec 450 -- hermes ...` fails (`Failed to exec "hermes"`): the binary is not on `lxc-attach`'s PATH. Use the absolute path or `su - hermes -c` (the dash gives a login shell, which sets `~/.local/bin`). A `hermes` login shell also prints a `hermes-setup` banner before output.
**Pre-flight before running the update** (each of these cost one failed run on 2026-10-03):
1. `clang` installed — source-compiled Python dependencies will not build without it (`pct exec 450 -- bash -lc "which clang"`).
2. No stale `.install.lock` / `.lazy-refresh-incomplete` in the Hermes home.
3. `~/.hermes/hermes-agent` owned by `hermes:hermes` (a root-owned tree from an earlier root-run update breaks the in-place build).
4. Host Caddy/`keyring` file permissions sane (`0600` keyring blocked a step mid-update).
**After:** `pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"`, confirm CT460 production is untouched (separate config/skills/cron/memory). Full record: [[incidents/2026-10-03-ct450-hermes-update]].
## Last tested ## Last tested
2026-07-22 on debian 13, git-based Hermes install 2026-07-22 on debian 13, git-based Hermes install
+8 -7
View File
@@ -3,8 +3,8 @@ title: Hermes Agent
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-03 updated: 2026-10-04
verified_on: 2026-10-03 verified_on: 2026-10-04
confidence: high confidence: high
tags: [system, hermes, agent, llm] tags: [system, hermes, agent, llm]
sources: [live-hermes-version, live-config, live-skill-inventory] sources: [live-hermes-version, live-config, live-skill-inventory]
@@ -16,16 +16,16 @@ sources: [live-hermes-version, live-config, live-skill-inventory]
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks. Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
## Current Status ## Current Status
- **Version:** v0.21.5+3837.gbfda74c (2026.9.24), upstream `27062c34` — verified 2026-09-30 - **Version:** v0.21.5+6169.gab86a20 (2026.9.24), upstream `ab86a20a` — verified live 2026-10-04. Same semver token as the earlier `v0.21.5+3837.gbfda74c` (`27062c34`), but a newer build: the install tree was rebuilt 2026-10-03 10:11 AWST.
- **Install method:** git - **Install method:** git
- **Python:** 3.11.15 - **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
- **Upstream status:** Update available (`hermes update`) - **Upstream status:** tree rebuilt 2026-10-03 from upstream `ab86a20a`; update availability not re-checked since.
- **Install directory:** `/home/hermes/.hermes/hermes-agent` - **Install directory:** `/home/hermes/.hermes/hermes-agent`
- **Managed runtimes:** 10 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv`. Dependency guard `hermes-runtime-deps-guard.sh` verifies `dotenv` + `mnemosyne` in every venv every 15 min — see [[incidents/2026-09-28-hermes-managed-runtime-deps]]. - **Managed runtimes:** 3 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv` (was 10 — the 2026-10-03 rebuild pruned superseded environments and created two new ones at 09:59/10:00; the third survives from 2026-09-28). `hermes-runtime-deps-guard.sh` runs every 15 min via `mnemosyne-watchdog.py` and now asserts `dotenv` + `mnemosyne` in the installer python *and every managed venv*, **plus** `fastembed` + `FASTEMBED_CACHE_PATH` + the local model dir in the bootstrap-selected venv — see [[incidents/2026-09-28-hermes-managed-runtime-deps]] and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
- **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions - **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions
- **Binary:** `/home/hermes/.local/bin/hermes` - **Binary:** `/home/hermes/.local/bin/hermes`
- **Config:** `~/.hermes/config.yaml` - **Config:** `~/.hermes/config.yaml`
- **Profile:** `default` (`~/.hermes/profiles/default/`) - **Profile:** `default` (top-level config `~/.hermes/config.yaml`; the only named sub-profile on this host is `ops` at `~/.hermes/profiles/ops/`)
## Location ## Location
- **Host:** Local Linux host (no containerized Hermes runtime confirmed) - **Host:** Local Linux host (no containerized Hermes runtime confirmed)
@@ -131,6 +131,7 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
## Known Issues ## Known Issues
- Managed-runtime venvs previously drifted on dependencies, breaking cron workers — now guarded every 15 min. See [[incidents/2026-09-28-hermes-managed-runtime-deps]]. - Managed-runtime venvs previously drifted on dependencies, breaking cron workers — now guarded every 15 min. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **Vector recall silently ran keyword-only until 2026-10-03 (repaired).** `fastembed` was missing from every uv-managed runtime and `FASTEMBED_CACHE_PATH` was unset, so the dense leg of `mnemosyne_recall` returned no candidates and recall degraded to FTS5 — with no error raised. Repaired 2026-10-03 and the guard extended (above). Live coverage 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%). See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
- **Four cron jobs cannot self-recover — three pinned to models that no longer exist upstream, plus a credential fault:** `1a8e90ea2271` → `qwen3.5:397b` retired 2026-09-25 (HTTP 410); `dca8482e4f76` → `gpt-5.6-terra` plan-gated (403 `MODEL_NOT_IN_PLAN`); `fad5a86979d4` + `0e753e9720fd` → `deepseek-v4-flash` absent from the NVIDIA live catalog (HTTP 400); and `06a0a1c8704f` + `1b23c042e17d` → the `cliprox/terra` provider *connection* auth expired (HTTP 401), now confirmed on 2 of the 3 jobs sharing that pin. Repinning to the `smart-route` alias is the durable fix for the model faults; the cliprox pair needs the OmniRoute connection re-authenticated first. See [[systems/scheduled-tasks]], [[systems/model-providers]] and [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. - **Four cron jobs cannot self-recover — three pinned to models that no longer exist upstream, plus a credential fault:** `1a8e90ea2271` → `qwen3.5:397b` retired 2026-09-25 (HTTP 410); `dca8482e4f76` → `gpt-5.6-terra` plan-gated (403 `MODEL_NOT_IN_PLAN`); `fad5a86979d4` + `0e753e9720fd` → `deepseek-v4-flash` absent from the NVIDIA live catalog (HTTP 400); and `06a0a1c8704f` + `1b23c042e17d` → the `cliprox/terra` provider *connection* auth expired (HTTP 401), now confirmed on 2 of the 3 jobs sharing that pin. Repinning to the `smart-route` alias is the durable fix for the model faults; the cliprox pair needs the OmniRoute connection re-authenticated first. See [[systems/scheduled-tasks]], [[systems/model-providers]] and [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- **The daily skills stock-take is not running.** Job `1a8e90ea2271` is blocked on the retired model above, so [[skills-index]] and the freshness audit are only refreshed when `wiki-maintain.sh` is run by hand (last done 2026-10-03). The script itself is healthy — only the scheduled wrapper is broken. - **The daily skills stock-take is not running.** Job `1a8e90ea2271` is blocked on the retired model above, so [[skills-index]] and the freshness audit are only refreshed when `wiki-maintain.sh` is run by hand (last done 2026-10-03). The script itself is healthy — only the scheduled wrapper is broken.
- **Telegram thread delivery failures — root-caused and fixed 2026-09-30.** A bare `deliver: "telegram"` resolves to the home DM (`1793951355`), where group forum topic IDs cannot exist; the send hard-failed and the message was lost. Two jobs were retargeted to `telegram:-1004321904721:102` and verified. Three jobs still use bare `telegram` targets. See [[systems/messaging-integrations]]. - **Telegram thread delivery failures — root-caused and fixed 2026-09-30.** A bare `deliver: "telegram"` resolves to the home DM (`1793951355`), where group forum topic IDs cannot exist; the send hard-failed and the message was lost. Two jobs were retargeted to `telegram:-1004321904721:102` and verified. Three jobs still use bare `telegram` targets. See [[systems/messaging-integrations]].
+3 -2
View File
@@ -3,8 +3,8 @@ title: Memory Backend
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-08-01 updated: 2026-10-04
verified_on: 2026-08-01 verified_on: 2026-10-04
confidence: high confidence: high
tags: [memory, mnemosyne, agent] tags: [memory, mnemosyne, agent]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -43,6 +43,7 @@ Persist and recall agent memory across sessions.
## Known Issues ## Known Issues
- Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods. - Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods.
- The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]]. - The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]].
- **Vector recall degraded silently until 2026-10-03** — `fastembed` was missing from every uv-managed runtime, so `mnemosyne_recall` fell back to FTS5 keyword scoring with no error raised. Repaired 2026-10-03; the 15-min dependency guard now asserts `fastembed` and the model cache path. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
**Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]]. **Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]].
+25 -14
View File
@@ -3,8 +3,8 @@ title: Mnemosyne
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-01 updated: 2026-10-04
verified_on: 2026-10-01 verified_on: 2026-10-04
confidence: high confidence: high
tags: [memory, mnemosyne, backend] tags: [memory, mnemosyne, backend]
sources: [live-probe, mnemosyne-db, config, ops-profile-config] sources: [live-probe, mnemosyne-db, config, ops-profile-config]
@@ -33,25 +33,34 @@ Primary agent memory store for Hermes.
- **Embedding model:** `BAAI/bge-small-en-v1.5` (384 dims) - **Embedding model:** `BAAI/bge-small-en-v1.5` (384 dims)
- **Embeddings via API:** false (local fastembed) - **Embeddings via API:** false (local fastembed)
## Database Stats (2026-08-02) ## Database Stats (live, 2026-10-04)
| Table | Rows | Notes | | Table | Rows | Notes |
|---|---|---| |---|---|---|
| `working_memory` | 1,476 | Active memories | | `working_memory` | 752 | Active memories |
| `episodic_memory` | 458 | Consolidated summaries | | `episodic_memory` | 1,039 | Consolidated summaries |
| `memory_embeddings` | 158 | Only working memories — **0 episodic** | | `memory_embeddings` | 739 | Working-tier embeddings |
| `memory` (legacy) | 2,181 | Old table, not migrated | | `memory` (legacy) | 2,181 | Old table, not migrated |
| `vec_working` | 159 | Vector index rows | | `vec_working` / `vec_working_rowids` | 739 | Working vector index |
| `vec_episodes` / `vec_episodes_rowids` | 1,035 | Episodic vector index |
| `vec_facts_rowids` | 0 | `memoria_facts` is intentionally left unvectorised |
| Shared surface | 0 | Empty — cross-agent sharing not functional | | Shared surface | 0 | Empty — cross-agent sharing not functional |
**Embedding coverage (live, 2026-10-04):** 1,759/2,194 memories embedded (80.2%) — working **726/752 (96.5%)**, episodic **1,033/1,039 (99.4%)**; 13 embeddings generated in the previous 24h. Working coverage is above the 95% alert threshold enforced by the Embedding Coverage job (`b63ed5817212`, every 6h, Telegram topic 102).
## Known Issues ## Known Issues
### Vector embedding coverage gap ### Vector embedding coverage gap — RESOLVED (2026-08-02)
- **Only 159/1,476 working memories (10.8%)** have vector embeddings - Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
- **0/458 episodic memories** have embeddings - **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only.
- Semantic search (`mnemosyne_recall`) falls back to FTS5 keyword matching for ~90% of working and 100% of episodic memories - Standard repair if coverage drops below 95% working: `~/.hermes/hermes-agent/venv/bin/mnemosyne reindex --yes` (the Embedding Coverage job `b63ed5817212` triggers the same path every 6h).
- This means recall quality is degraded — vector similarity can only search a tiny fraction of the memory store
- **Root cause:** The embedding pipeline (fastembed) never processed the bulk of memories. The `vec_working` table was rebuilt once (Jul 18) but only captured 159 rows. ### Vector recall silently ran FTS-only (2026-10-03 — repaired)
- **Fix needed:** Backfill embeddings for all working and episodic memories. This requires running the embedding pipeline across the full dataset. - `fastembed` was absent from **every** uv-managed runtime, so the dense leg of `mnemosyne_recall` produced no candidates and recall fell back to FTS5/keyword scoring **without raising an error**. Sessions kept working, which is why it went unnoticed.
- Compounded by `FASTEMBED_CACHE_PATH` being unset in `~/.hermes/.env`, and by the embedding worker sometimes launching in isolated (`python -I`) mode, which does not inherit `PYTHONPATH`.
- **Repaired 2026-10-03:** `fastembed` installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min alongside its existing `dotenv`/`mnemosyne` checks.
- **CLI caveat — do not misread `mnemosyne doctor`.** Doctor and the recall CLI run under the legacy `venv/` (Python 3.11), where the `sqlite_vec` extension is not importable; doctor therefore prints `vec0: present_but_unloadable` and marks the vector tiers `unavailable` even while the gateway embeds normally. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`, not with doctor's `vec0` line.
- Benign noise: `pthread_setaffinity_np failed … mask: {0,}` from onnxruntime in `mcp-stderr.log` is a container affinity warning emitted on every model load — not an error.
- See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
### Profile isolation chosen over shared surface ### Profile isolation chosen over shared surface
|- The ops profile was enabled with mnemosyne on 2026-08-02 |- The ops profile was enabled with mnemosyne on 2026-08-02
@@ -75,9 +84,11 @@ Primary agent memory store for Hermes.
- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent - Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent - Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception - Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent - Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
## History and Evidence ## History and Evidence
- **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
- **2026-09-30 memory-hygiene pass:** 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (`hermes_cron_model_default` → `custom:omni-101`, job-radar model). No unresolved conflicts left in the working tier. - **2026-09-30 memory-hygiene pass:** 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (`hermes_cron_model_default` → `custom:omni-101`, job-radar model). No unresolved conflicts left in the working tier.
- **2026-08-19 Mnemosyne Watchdog incident:** Watchdog job (`96eb95ce12f2`) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See [[systems/scheduled-tasks]] Known Issues section. - **2026-08-19 Mnemosyne Watchdog incident:** Watchdog job (`96eb95ce12f2`) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See [[systems/scheduled-tasks]] Known Issues section.
- **2026-08-18 incident:** Working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See [[incidents/2026-08-18-mnemosyne-capture-pipeline]]. - **2026-08-18 incident:** Working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See [[incidents/2026-08-18-mnemosyne-capture-pipeline]].
+4 -3
View File
@@ -3,8 +3,8 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-03 updated: 2026-10-04
verified_on: 2026-10-03 verified_on: 2026-10-04
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db] sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -25,6 +25,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
- **Resolved since the last entry:** 1 — `1b23c042e17d`'s stale `delivery_failed` flag cleared exactly as predicted. Its 2026-10-02 run delivered to topic `102` (`delivery_outcome: delivered`); it then failed on the model pin, above. - **Resolved since the last entry:** 1 — `1b23c042e17d`'s stale `delivery_failed` flag cleared exactly as predicted. Its 2026-10-02 run delivered to topic `102` (`delivery_outcome: delivered`); it then failed on the model pin, above.
- **Shared model pin at risk:** 3 enabled jobs run `cliprox/terra` — `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`. **Two of the three have now failed on that pin.** The third (`79f9bb41ecf5`) last ran 2026-09-27 — before the credential lapsed — and is next due 2026-10-04 18:00. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. - **Shared model pin at risk:** 3 enabled jobs run `cliprox/terra` — `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`. **Two of the three have now failed on that pin.** The third (`79f9bb41ecf5`) last ran 2026-09-27 — before the credential lapsed — and is next due 2026-10-04 18:00. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- **Streak recount (2026-10-03):** recounted directly from `~/.hermes/cron/executions.db`. `dca8482e4f76` 6 → 7 (one more daily run since), and `1a8e90ea2271` 8 → **7** — the ×8 carried in the 2026-10-02 entry was an overcount: that job has exactly 7 recorded runs and all 7 failed. - **Streak recount (2026-10-03):** recounted directly from `~/.hermes/cron/executions.db`. `dca8482e4f76` 6 → 7 (one more daily run since), and `1a8e90ea2271` 8 → **7** — the ×8 carried in the 2026-10-02 entry was an overcount: that job has exactly 7 recorded runs and all 7 failed.
- **Re-verified 2026-10-04 00:12 AWST — no change:** still 30 jobs (27 enabled / 3 disabled) and the same **6** failing jobs; no new failure event landed inside the audit window. `79f9bb41ecf5` (Weekly Review, the last untested `cliprox/terra` job) drops at **2026-10-04 18:00 AWST** — its result settles whether that pin is 3/3 broken. `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found; delivered without thread_id`), confirmed live in `hermes cron list`.
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) - Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03)
## Job Table ## Job Table
@@ -87,7 +88,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
### Open ### Open
- **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 10-02, streak 7):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change; it cannot self-recover. See [[runbooks/job-radar]]. - **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 10-02, streak 7):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change; it cannot self-recover. See [[runbooks/job-radar]].
- **`cliprox/terra` provider connection auth expired — 2 of 3 jobs confirmed (streak 1 each):** Jobs `06a0a1c8704f` (2026-10-01) and `1b23c042e17d` (2026-10-02) both failed with `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Failure notices reached their topics; no output was produced. This is a *credential/connection* problem at the proxy, not a missing model — it will not self-recover until the connection is re-authenticated. `79f9bb41ecf5` is the remaining untested job on this pin (next run 2026-10-04 18:00). Re-authenticate the OmniRoute connection before repinning anything. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. - **`cliprox/terra` provider connection auth expired — 2 of 3 jobs confirmed (streak 1 each):** Jobs `06a0a1c8704f` (2026-10-01) and `1b23c042e17d` (2026-10-02) both failed with `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Failure notices reached their topics; no output was produced. This is a *credential/connection* problem at the proxy, not a missing model — it will not self-recover until the connection is re-authenticated. `79f9bb41ecf5` is the remaining untested job on this pin (next run 2026-10-04 18:00 — re-check on 2026-10-05). Re-authenticate the OmniRoute connection before repinning anything. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- **A delivered notice is not a successful job:** both `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output. - **A delivered notice is not a successful job:** both `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output.
- **wiki-stocktake-maintenance — HTTP 410, model retired (streak 7):** Job `1a8e90ea2271` is pinned to `ollama-cloud / qwen3.5:397b`, which Ollama retired on 2026-09-25 (`qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT`). It cannot self-recover — it needs a new model pin. **The job's script is fine:** `wiki-maintain.sh` was run by hand on 2026-10-03 and completed cleanly, regenerating [[systems/skills-index]] (344 skills / 70 categories) and refreshing the freshness audit. Only the scheduled wrapper is broken — so every non-model step of the daily stocktake needs a manual run until it is repinned. - **wiki-stocktake-maintenance — HTTP 410, model retired (streak 7):** Job `1a8e90ea2271` is pinned to `ollama-cloud / qwen3.5:397b`, which Ollama retired on 2026-09-25 (`qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT`). It cannot self-recover — it needs a new model pin. **The job's script is fine:** `wiki-maintain.sh` was run by hand on 2026-10-03 and completed cleanly, regenerating [[systems/skills-index]] (344 skills / 70 categories) and refreshing the freshness audit. Only the scheduled wrapper is broken — so every non-model step of the daily stocktake needs a manual run until it is repinned.
- **Facebook Birthday scrapes — HTTP 400, model missing from catalog (streak 3 / 1):** Job `fad5a86979d4` (weekly) is pinned to `nvidia/deepseek-ai/deepseek-v4-flash`; the provider reports the model is `not available in the active live catalog for provider 'nvidia'`. **Confirmed as a shared-pin issue on 2026-10-01**, when sibling job `0e753e9720fd` (monthly) failed on the same pin at its own 09:00 run. Safest durable fix is repinning both to a routing alias (`smart-route`) instead of a raw model name. - **Facebook Birthday scrapes — HTTP 400, model missing from catalog (streak 3 / 1):** Job `fad5a86979d4` (weekly) is pinned to `nvidia/deepseek-ai/deepseek-v4-flash`; the provider reports the model is `not available in the active live catalog for provider 'nvidia'`. **Confirmed as a shared-pin issue on 2026-10-01**, when sibling job `0e753e9720fd` (monthly) failed on the same pin at its own 09:00 run. Safest durable fix is repinning both to a routing alias (`smart-route`) instead of a raw model name.