stocktake(2026-11-01): fix the Quartz build (YAML title), close the "unfixed memory ranking" incident, and correct live drift in tailscale/docker/proxmox

BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic.
Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an
unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a
mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)".
A scan of all 117 pages found exactly one such line; it is now properly quoted.
Quartz now builds cleanly: 690 files emitted and deployed.

RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page).
The page claimed the recall-ranking fix "was not present in live config" and that
conflict 152 was "the only open conflict". Both claims were false:
- The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5,
  fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 —
  not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the
  2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks.
- The original regression test now passes: "mnemosyne recall 'Are you up to date'"
  returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale
  Uber Eats episode no longer outranks anything.
- Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise.
All three open items closed; page moved to status: resolved.

LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01):
- infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping
  times out); m93p peer record reads false for the 4th consecutive time while the
  host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages
  refreshed; no IP changes, no missing peers.
- infrastructure/docker-services.md: CT201's Tailscale IP was wrong
  (100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale
  worldmonitor-seeder-fast entry removed from the CT200 list, which is now
  regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to
  thirsty_payne.
- infrastructure/proxmox.md: storage percentages refreshed on both hosts from live
  pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an
  elevated load average recorded.
- systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md:
  re-probed and verified_on bumped.
- systems/skills-index.md: regenerated — 301 unique skills in 68 categories,
  46 archived. (301 is the unique (category, skill_name) count; find returns 302
  files due to the known nested obsidian-git-sync alias duplicate.)

Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
This commit is contained in:
Tony0410
2026-11-01 04:48:30 +08:00
parent c69681dc16
commit a0b3734c95
14 changed files with 596 additions and 111 deletions
+29 -2
View File
@@ -3,8 +3,8 @@ title: Messaging Integrations
type: system
status: active
created: 2026-07-22
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-01
verified_on: 2026-11-01
confidence: high
tags: [messaging, telegram, discord, email, integrations]
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
@@ -125,3 +125,30 @@ one of those happens — restarting the adapter while the host is stopped achiev
process's** attempts, not the outage's length — do not quote it as downtime.
- **Impact unchanged:** messaging path only; no cron job delivers via Mattermost.
- **Not attempted:** `pct start 255` — starting a service on the PVE host is Anthony's call.
## 2026-10-31 06:2x AWST — ✅ Mattermost RECOVERED (outage over)
**The ~40-day Mattermost outage is over.** All three live checks pass, and the gateway log confirms
the adapter stayed connected since.
| Check | Live result (2026-10-31) |
|---|---|
| `pct list` on `pve` | **CT255 (`mattermost`) → `running`** (was `stopped`) |
| `pct exec 255 -- systemctl is-active mattermost` | **`active`** |
| `pct exec 255 -- ss -ltn` | `LISTEN *:8065` |
| `tailscale ping 100.98.116.14` | **`pong from mattermost ... in 0s`** |
| `curl http://100.98.116.14:8065/api/v4/system/ping` | **HTTP 200** |
| Tailscale peer record | **`Online: true`** — first time since the outage began |
- **Gateway reconnection (CT460's own `gateway.log`):** `Mattermost: authenticated as @rhino-bot` at
**2026-10-30 07:19:37**, with `✓ mattermost reconnected successfully` and a successful WebSocket
handshake at the same second. A second connect cycle at **2026-10-30 16:36:07** produced
`✓ mattermost connected` at **16:36:17**.
- **No errors since.** Grepping every `mattermost` line in `gateway.log` after `2026-10-30 16:36:17`
returns **zero** `error` / `timeout` / `disconnect` / `fail` lines — the 5-minute reconnect loop that
produced 264–268 timeouts a day is **no longer running**.
- **The prior "cause: CT255 is stopped" finding is superseded.** CT255 is running; the gateway can
reach `http://100.98.116.14:8065/` and authenticate. → [[infrastructure/proxmox]],
[[infrastructure/tailscale]]
- **Open item now closed:** the platform is no longer an outage. The home channel startup notification
was delivered successfully on reconnect (`Sent home-channel startup notification to mattermost:…`).
+5 -3
View File
@@ -3,8 +3,8 @@ title: Mnemosyne
type: system
status: active
created: 2026-07-22
updated: 2026-10-31
verified_on: 2026-10-31
updated: 2026-11-01
verified_on: 2026-11-01
review_after: 2026-11-07
confidence: high
tags: [memory, mnemosyne, backend]
@@ -52,7 +52,9 @@ Primary agent memory store for Hermes.
**Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below.
**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. Ranking relevance is the live issue, not coverage — see [[incidents/2026-10-09-memory-recall-relevance]].
**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. ~~Ranking relevance is the live issue, not coverage~~ — see [[incidents/2026-10-09-memory-recall-relevance]].
**Tier counts (live, 2026-11-01 03:0x — the 2026-10-09 ranking incident is now RESOLVED):** working `941`, episodic `1,124`, `canonical_facts` `90`, `conflicts` `175` (**11 open** — ids **165–175**, all `contradiction`, all created in one batch at 2026-10-31 19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature), working embeddings `927/941 (98.5%)`, episodic embeddings `1,124/1,124 (100%)` — both tiers above the 95% floor, no `reindex` needed. **The recall weights are live in `~/.hermes/mnemosyne/config.yaml`** (`vec_weight: 0.5`, `fts_weight: 0.3`, `importance_weight: 0.3`, plus `MNEMOSYNE_IMPORTANCE_WEIGHT=0.3` in the environment) — **not** in `~/.hermes/config.yaml`'s `memory.mnemosyne` block, which is why the 2026-10-10 probe wrongly concluded the fix was never applied. The original regression query now returns only substantive current-state facts; the stale Uber Eats episode no longer outranks them → [[incidents/2026-10-09-memory-recall-relevance]] (**resolved**).
**Tier counts (live, 2026-10-12 12:02 — health job `ea79a2107e36`, green):** working **760/786 embedded (96.7%)**, episodic **1050/1056 (99.4%)** — both above the 95% floor, no `reindex` needed (`b63ed5817212` continues to backfill). Orphan embeddings **0**. FTS5 exactly aligned: `fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876. **0 unconsolidated** working rows; **1 unresolved conflict** (threshold 10); last consolidation 0.3 h earlier; 2,252 memories total (786 working / 1,056 episodic / 1 triple); disk 15 GB free (81% used). Interpretation: coverage is *not* the live issue — ranking relevance is (see [[incidents/2026-10-09-memory-recall-relevance]]).
+18 -17
View File
@@ -3,8 +3,8 @@ title: OmniRoute
type: system
status: active
created: 2026-07-22
updated: 2026-10-22
verified_on: 2026-10-22
updated: 2026-11-01
verified_on: 2026-11-01
confidence: high
tags: [routing, provider, model, inference]
sources: [raw/configs/hermes-config-sanitized.txt]
@@ -54,30 +54,31 @@ The CT600 provider key `custom:omni` (`http://omniroute:20128/v1`) remains auxil
→ [[systems/current-profile]]
## Current Configuration
- **Hermes provider name:** `custom:omni` (maps to provider `omniroute:20128`)
- **Base URL:** `http://omniroute:20128/v1`
- **Hermes default provider:** `custom:omni-101` → base URL `http://100.78.63.20:20128` (CT101, `m93p`), model default **`smart-route`** — **re-verified live 2026-10-31** from `~/.hermes/config.yaml` (`model.default: smart-route`, `model.provider: custom:omni-101`, `model.base_url: http://100.78.63.20:20128`)
- **Secondary provider entry:** `custom:Omniroute:20128` → base URL `http://omniroute:20128/v1` (CT600, `pve`), referenced by the `moa` reference-model set (`vertex/gemini-3.6-flash`, `kilo-gateway/kilo-auto/free`, `nvidia/deepseek-ai/deepseek-v4-pro`, `freeapi/nemotron-3-ultra-550b`) — auxiliary/MoA use, **not** the main chat default
- **Host reachability:** bare name `omniroute` resolves to `100.88.81.19`; direct HTTP on `:20128` answers `401` (auth-required, service live)
- **Runs on:** Proxmox CT600 (`omniroute`, Tailscale `100.88.81.19`, onboot=1) — verified running 2026-10-11
- **Runs on:** two live nodes — CT600 (`omniroute`, Tailscale `100.88.81.19`, on `pve`) and CT101 (`omniroute`, Tailscale `100.78.63.20`, on `m93p`) — **both verified running and listening on `0.0.0.0:20128` on 2026-10-31**
- **DB path:** `/opt/omniroute/storage.sqlite` (was `/var/lib/omniroute/storage.sqlite`)
- **Backup:** pre-migration config saved at `~/.hermes/config.yaml.bak.omniroute-removal`
- **Correction (2026-10-26):** the earlier *"Hermes default provider path does not use OmniRoute for main chat"* note is **stale** — the default is OmniRoute again (`custom:omni-101`, model `smart-route`). The **CT600** key `custom:omni` is still auxiliary-only.
- **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp`
- **Correction (2026-10-26):** the earlier *"Hermes default provider path does not use OmniRoute for main chat"* note is **stale** — the default is OmniRoute again (`custom:omni-101`, model `smart-route`). The **CT600** key `custom:Omniroute:20128` is still auxiliary-only (MoA reference models).
- **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp` — all on `custom:omni-101` with model `no-think/freeapi/auto` (live-verified 2026-10-31)
- **Key prefix visible in config:** `sk-5c67fbdc0...` — full value not stored in wiki
## CT600 / CT601 / omniroute-1 — actual state (2026-10-11)
## CT600 / CT601 / omniroute-1 — actual state (re-verified 2026-10-31)
The wiki previously recorded CT600 as retired and CT601 as the live successor. **That is no longer true.** Live PVE and Tailscale checks return:
The wiki previously recorded CT600 as retired and CT601 as the live successor, then CT601 as stopped.
**All three framings are now superseded.** Live `pct list` on **both** hypervisors and live HTTP probes return:
| Node | Status (live 2026-10-11) | Tailscale IP | Notes |
| Node | Status (live 2026-10-31) | Tailscale IP | Notes |
|---|---|---|---|
| CT600 `omniroute` | **running**, onboot=1 | `100.88.81.19` | OmniRoute listening on `0.0.0.0:20128`; `/v1/models` returns `401` (alive, auth required) |
| CT601 `omni` | **stopped**, onboot=0 | `100.93.204.4` | Tailscale peer last seen 54d ago |
| `omniroute-1` | **active** (not in PVE `pct list`) | `100.78.63.20` | Third OmniRoute node; `/v1/models` returns `401`. Registered in Hermes config as custom provider `omni-101`, which carries `skills_hub` (model `free-stack`, 5384 models, default `smart-route`) |
| CT600 `omniroute` | **running** | `100.88.81.19` | On `pve`. OmniRoute listening on `0.0.0.0:20128`; `/v1/models` and `/api/combos` both return `401` (alive, auth required) |
| CT101 `omniroute-1` | **running** | `100.78.63.20` | On `m93p`. Live `pct config 101` → `hostname: omniroute`; `omniroute` listening on `0.0.0.0:20128`; `/v1/models` and `/api/combos` both return `401`. Registered in Hermes config as `custom:omni-101` — **the live default provider** |
| CT601 `omni` | **does not exist** | — | `/etc/pve/lxc/601.conf` absent on **both** `pve` and `m93p`. The `omni` peer (`100.93.204.4`) has **expired out of the tailnet entirely** — live `tailscale status --json` has no peer named `omni` and no peer at that IP |
- Hermes config resolving to the omniroute hostname (`custom:omni` → `omniroute:20128`) therefore points at **CT600**, not CT601.
- CT600's combo API is reachable with its local key and returns combos (e.g. `go-pro`), confirming it is the serving instance.
- **Do not treat CT601 as the production OmniRoute node.** It is stopped with `onboot=0`.
- Treat `omniroute-1` (`100.78.63.20`) as a second live instance used by the `omni-101` provider. It is not visible in `pct list`, so it is likely a VM or off-PVE host; its identity was not confirmed.
- Hermes config resolving to the omniroute hostname (`custom:Omniroute:20128` → `omniroute:20128`) therefore points at **CT600**; the live default `custom:omni-101` points at **CT101**.
- **CT601 was deleted, not merely stopped** — the earlier *"stopped, onboot=0"* entry was wrong.
- **The `omniroute-1` identity is now confirmed** — it is **CT101 on `m93p`**, not "a VM or off-PVE host". The 2026-10-11 note that it was "not visible in `pct list`" was an artifact of probing only `pve`.
- **Two live OmniRoute nodes remain** (CT600, CT101), both serving `:20128` with separate credentials — a pin's usability must be probed on the node that actually serves it.
## Catalog and Routing Facts
+4 -8
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system
status: active
created: 2026-07-22
updated: 2026-10-30
verified_on: 2026-10-30
updated: 2026-11-01
verified_on: 2026-11-01
confidence: high
tags: [skills, hermes]
sources: []
@@ -12,9 +12,9 @@ sources: []
# Installed Skills
**Total installed: 302 skills** in `~/.hermes/skills/`.
**Total installed: 301 skills** in `~/.hermes/skills/`.
- **45 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
- **46 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
- The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory.
This page is auto-regenerated from the live skill inventory. Do not hand-edit.
@@ -308,10 +308,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `hermes-mcp-troubleshooting` — # Hermes MCP Troubleshooting
## Hermes Runtime Worker Diagnostics (1)
- `hermes-runtime-worker-diagnostics` — # Hermes Runtime / Cron Worker Diagnostics
## Hermes Themes (1)
- `hermes-themes` — # Hermes Themes Skill