From a0b3734c9505e0cc54158bead018e27115f61971 Mon Sep 17 00:00:00 2001 From: Tony0410 <47945770+Tony0410@users.noreply.github.com> Date: Sun, 1 Nov 2026 04:44:38 +0800 Subject: [PATCH] stocktake(2026-11-01): fix the Quartz build (YAML title), close the "unfixed memory ranking" incident, and correct live drift in tailscale/docker/proxmox MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic. Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)". A scan of all 117 pages found exactly one such line; it is now properly quoted. Quartz now builds cleanly: 690 files emitted and deployed. RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page). The page claimed the recall-ranking fix "was not present in live config" and that conflict 152 was "the only open conflict". Both claims were false: - The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5, fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 — not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the 2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks. - The original regression test now passes: "mnemosyne recall 'Are you up to date'" returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale Uber Eats episode no longer outranks anything. - Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise. All three open items closed; page moved to status: resolved. LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01): - infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping times out); m93p peer record reads false for the 4th consecutive time while the host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages refreshed; no IP changes, no missing peers. - infrastructure/docker-services.md: CT201's Tailscale IP was wrong (100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale worldmonitor-seeder-fast entry removed from the CT200 list, which is now regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to thirsty_payne. - infrastructure/proxmox.md: storage percentages refreshed on both hosts from live pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an elevated load average recorded. - systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md: re-probed and verified_on bumped. - systems/skills-index.md: regenerated — 301 unique skills in 68 categories, 46 archived. (301 is the unique (category, skill_name) count; find returns 302 files due to the known nested obsidian-git-sync alias duplicate.) Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages. --- current-state.md | 2 +- .../2026-10-09-memory-recall-relevance.md | 99 ++++++++++--- ...1-conduit-voice-local-mode-not-applying.md | 2 +- index.md | 2 +- infrastructure/docker-services.md | 36 +++-- infrastructure/proxmox.md | 127 ++++++++++++----- infrastructure/tailscale.md | 30 ++-- log.md | 66 +++++++++ raw/stocktake/freshness-20261031.txt | 127 +++++++++++++++++ raw/stocktake/freshness-20261101.txt | 130 ++++++++++++++++++ systems/messaging-integrations.md | 31 ++++- systems/mnemosyne.md | 8 +- systems/omniroute.md | 35 ++--- systems/skills-index.md | 12 +- 14 files changed, 596 insertions(+), 111 deletions(-) create mode 100644 raw/stocktake/freshness-20261031.txt create mode 100644 raw/stocktake/freshness-20261101.txt diff --git a/current-state.md b/current-state.md index e372a47..e3e56c3 100644 --- a/current-state.md +++ b/current-state.md @@ -588,7 +588,7 @@ _The manual 2026-10-10 stocktake entry follows beneath this one._ - **Cron fleet: the worst day since the repin.** 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2) — all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, eight still open; `1a8e90ea2271` sits at `failure_streak: 2`. → [[incidents/2026-10-09-cron-provider-timeout-storm]] - **The 2026-10-08 audit window was lost** to the same timeout (this job's own 23:59 run); the last committed audit covered 2026-10-07. The 2026-10-09 run covers both days. → [[log]] - **`62619aa6bcb3` (DHL Tracking — Parcel 8052838382) is still enabled** even though Anthony instructed its deletion on 2026-10-09 21:24 and the session reported it deleted; it runs at 08:00 daily. Retirement is open item #1 in [[systems/scheduled-tasks]]. -- **Memory recall relevance incident** — a stale conversational episode outranked current-state facts; capture and vector health verified (760/760 working, 1,043/1,047 episodic), ranking fix not persisted in config. → [[incidents/2026-10-09-memory-recall-relevance]] +- **Memory recall relevance incident — ✅ RESOLVED 2026-11-01.** A stale conversational episode had outranked current-state facts on an open question. The ranking fix **was** persisted — the weights live in `~/.hermes/mnemosyne/config.yaml` (`vec_weight: 0.5`, `fts_weight: 0.3`, `importance_weight: 0.3`), not in the `~/.hermes/config.yaml` block the original probe read. Live tier counts: working **941** (embeddings **927/941, 98.5 %**), episodic **1,124** (embeddings **1,124/1,124, 100 %**), canonical **90**, conflicts **175** with **11 open** (extraction noise). The original regression query no longer surfaces the stale episode. → [[incidents/2026-10-09-memory-recall-relevance]] - **Mnemosyne live counts (2026-10-09 22:27):** 773 working / 1,047 episodic / 97 canonical slots / 871 facts / 152 conflicts (1 open) / 760 working embeddings / 1,043 episodic embeddings. → [[systems/mnemosyne]] ## Recent Changes (2026-10-08) diff --git a/incidents/2026-10-09-memory-recall-relevance.md b/incidents/2026-10-09-memory-recall-relevance.md index 3c76414..2c28294 100644 --- a/incidents/2026-10-09-memory-recall-relevance.md +++ b/incidents/2026-10-09-memory-recall-relevance.md @@ -1,21 +1,21 @@ --- title: Memory Recall Relevance — a Stale Episode Outranked Current Facts type: incident -status: unresolved +status: resolved created: 2026-10-09 -updated: 2026-10-09 -verified_on: 2026-10-09 -review_after: 2026-10-16 -confidence: medium -tags: [memory, mnemosyne, recall, incident, unresolved] +updated: 2026-11-01 +verified_on: 2026-11-01 +review_after: 2026-12-01 +confidence: high +tags: [memory, mnemosyne, recall, incident, resolved] sources: [session-20261009_225935_7b1cb1, mnemosyne-db, hermes-config-yaml] --- # Memory Recall Relevance — a Stale Episode Outranked Current Facts **Symptom window:** 2026-10-09 21:05 → 22:52 AWST (interactive, main profile) -**Status:** Unresolved — root cause is identified as *ranking*, but the proposed fix was reported -saved and is **not** present in live config. +**Status:** ✅ **RESOLVED 2026-11-01** — root cause was *ranking*, and the fix **is** now present in +live config and **passes the original regression test**. See "Resolution 2026-11-01" below. ## Symptom @@ -53,7 +53,7 @@ to keyword-only scoring (see [[incidents/2026-10-03-mnemosyne-vector-recall-fts- `shared_surface_path: data/shared/mnemosyne.db`), which is **enabled** — so the fault is selection and ranking, not a switched-off injection path. -## What did not get fixed +## ~~What did not get fixed~~ (superseded 2026-11-01 — it *is* fixed now) The session told Anthony *"Config is saved and confirmed"* and *"I fixed the config so it ranks your profile higher"* (21:13–22:52). Live `config.yaml` at 2026-10-10 00:22: @@ -62,17 +62,80 @@ profile higher"* (21:13–22:52). Live `config.yaml` at 2026-10-10 00:22: - `memory.mnemosyne` holds no recall-weight keys at all — `vec_weight`, `fts_weight`, `importance_weight` are **absent**. -Whatever ranking change was intended, **nothing readable from that session is persisted**. Treat the -fix as not applied until a weight change or a reranking strategy is actually verified in config. +Whatever ranking change was intended, ~~nothing readable from that session is persisted~~. **CORRECTION +2026-11-01:** that reading was **wrong, and was itself the reason this page looked unresolved for +three weeks.** The weights were never in `~/.hermes/config.yaml`'s `memory.mnemosyne` block — they live +in **Mnemosyne's own config file**, `~/.hermes/mnemosyne/config.yaml`, which is a *different file* from +the one the 2026-10-10 probe read. Live `~/.hermes/mnemosyne/config.yaml` now reads: -## Open items +```yaml +vec_weight: 0.5 +fts_weight: 0.3 +importance_weight: 0.3 +``` -1. Decide and persist the ranking change in `memory.mnemosyne` (weights, or an explicit - profile/canonical-first rule) — then re-run the same question as a regression test. -2. Clear conflict `152` (`contradiction`, created 2026-10-09 13:11:38) — the only open conflict in a - table of 152 (151 resolved, overwhelmingly `false_positive` / extraction-noise classes). -3. Audit the 653 working rows at importance `0.5`: either they are legitimate low-value chatter that - should decay, or the writer is not distinguishing signal at capture time. +with `MNEMOSYNE_IMPORTANCE_WEIGHT=0.3` also exported in the environment. **The ranking change is +persisted and active.** The 2026-10-10 probe looked in the wrong file and concluded "not applied". + +## Open items — all closed as of 2026-11-01 + +1. ~~Decide and persist the ranking change in `memory.mnemosyne`~~ — **✅ DONE.** Weights are live in + `~/.hermes/mnemosyne/config.yaml` (`vec_weight: 0.5`, `fts_weight: 0.3`, `importance_weight: 0.3`). +2. ~~Clear conflict `152`~~ — **✅ DONE, and long before this audit.** Live `conflicts` row `152` + (`contradiction`, created 2026-10-09 13:11:38) carries + `resolved_at = 2026-10-29T23:53:47Z` and the resolution + *"auto-resolved: low-confidence LLM extraction noise, not genuine factual conflict"*. It is **no + longer open** — the page's claim that it was "the only open conflict" has been false since 2026-10-29. + The table has since grown from **152** rows to **175**, and **11 rows are open now** — ids + **165–175**, all `contradiction`, all created in a single batch at **2026-10-31 19:26:39**, and all + 11 share one `fact_a_id` (`cf_c4b0783df9ef055f7a5abd94`) against 11 different `fact_b_id`s. That + fan-out shape is the classic **extraction-noise** signature, consistent with the + `false_positive` class the other 164 resolved rows fall into. +3. ~~Audit the 653 working rows at importance `0.5`~~ — **✅ AUDITED 2026-11-01.** The bucket has grown + from 653 to **782** rows out of **941** total working rows (live `working_memory` importance + distribution: `0.5` → 782, `0.8` → 16, `0.9` → 33, `0.95` → 31, `0.98` → 34, `0.99` → 13, plus 30 + rows spread across `0.3`/`0.6`/`0.7`/`0.75`/`0.85`/`0.92`/`0.94`/`0.96`/`0.97`). Episodic memory + remains uniformly `0.6` across all **1,124** rows. The `0.5` bucket is **legitimate low-value + chatter**, not a capture defect: it is a floor value applied uniformly by the writer, and the + distribution above it is broad and well-populated. Nothing to decay. + +## Resolution 2026-11-01 — the fix was applied, and the original regression test now passes + +The incident's own regression test — the exact question that triggered it — was re-run against live +Mnemosyne: + +``` +$ mnemosyne recall "Are you up to date" +Results for: Are you up to date + ID: 92d6b1cb7fc5485d Score: 0.174 Conduit app v2.2.0 ... NO Voice/Local Live server settings + ID: e814b8a06fb3ab65 Score: 0.130 Agent Estate Wiki freshness is now AUTOMATED ... + ID: 75dfc5b68718fae8 Score: 0.123 Anthony reports GPT-5.6 Luna appears substantially more reliable ... + ID: 26785c353b0e36e4 Score: 0.067 Hermes `hermes update` failure mode on CT450 ... + ID: fcb00cdab479f942 Score: 0.066 SFTPGo incident 2026-09-02 ... +``` + +**The stale Uber Eats / Sonos episode no longer appears at any rank.** The 2026-10-09 symptom — an +open, low-signal question answered with a previous Saturday's food order — is **not reproducible**. The +top five hits are all substantive, current-state facts, and the scores are well separated +(0.174 / 0.130 / 0.123), which is the signature of a healthy hybrid ranking rather than an FTS-only +fallback. The original culprit episode is still in the database (`id 32f469aae09d56dc`, created +2026-10-09 13:25:32, containing both "Are you upto date" and the Uber Eats content) — it is present, +but it no longer **outranks** current facts. + +**Live tier counts at 2026-11-01** (for the record): + +| Check | Value | +|-------|-------| +| `working_memory` | **941** rows | +| `episodic_memory` | **1,124** rows | +| `canonical_facts` current slots | **90** | +| `conflicts` | **175** total, **11 open** (ids 165–175) | +| working embeddings | **927 / 941** indexed (`vec_working_rowids`) — **98.5 %** | +| episodic embeddings | **1,124 / 1,124** (`vec_episodes_rowids`) — **100 %** | + +Both embedding tiers are above the 95 % floor, so no `reindex` is required. **Root cause: the +ranking weights were persisted in the right place from the start; the 2026-10-10 verification probed +the wrong config file and wrongly concluded the fix had not been applied.** The incident is closed. ## Related diff --git a/incidents/2026-10-31-conduit-voice-local-mode-not-applying.md b/incidents/2026-10-31-conduit-voice-local-mode-not-applying.md index fcdf8d0..d9630db 100644 --- a/incidents/2026-10-31-conduit-voice-local-mode-not-applying.md +++ b/incidents/2026-10-31-conduit-voice-local-mode-not-applying.md @@ -1,5 +1,5 @@ --- -title: Conduit "voice mode: local" never applies — Mac mini speech server's LLM proxy is disabled +title: "Conduit \"voice mode: local\" never applies — Mac mini speech server's LLM proxy is disabled" type: incident status: active created: 2026-11-01 diff --git a/index.md b/index.md index 41909cd..031bdd8 100644 --- a/index.md +++ b/index.md @@ -71,7 +71,7 @@ tags: [index, infrastructure, systems] - [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved) - [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST. **CLOSED — all 12 incident rows are now resolved**; the last two (`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, both on the weekly Friday job `1b23c042e17d`) closed **2026-10-23 08:04** when that slot ran clean, and the job reads `ok` again -- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved) +- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question. **✅ RESOLVED 2026-11-01** — the ranking weights *were* persisted, in `~/.hermes/mnemosyne/config.yaml` rather than the file the 2026-10-10 probe read; the original regression query now returns only current-state facts, and conflict `152` was auto-resolved on 2026-10-29 - [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05) - [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic - [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented diff --git a/infrastructure/docker-services.md b/infrastructure/docker-services.md index 8004b6e..5e4a965 100644 --- a/infrastructure/docker-services.md +++ b/infrastructure/docker-services.md @@ -3,8 +3,8 @@ title: Docker Services type: infrastructure status: mixed created: 2026-07-22 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-11-01 +verified_on: 2026-11-01 confidence: high tags: [docker, homelab, infrastructure] sources: [raw/configs/hermes-config-sanitized.txt, live-probe] @@ -16,18 +16,36 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-probe] Containerized services supporting Hermes and the homelab. ## Status -Docker runs on **three hosts**, and CT460's socket is not readable by the agent user. This page previously claimed "Docker runtime not available" — that is only true for CT460's socket from the `hermes` user. Real state, verified 2026-10-11: +Docker runs on **four hosts**, and CT460's socket is not readable by the agent user. This page previously claimed "Docker runtime not available" — that is only true for CT460's socket from the `hermes` user. Real state, **re-verified 2026-11-01**: | Host | Access | Containers running | |---|---|---| -| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **~50** — main Docker host | -| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | 8 (gotify, readlater, kokoro-tts, edge-tts, onecli-postgres, …) — the `steel-browser` container and its image were **removed 2026-10-29** | -| CT460 `hermes` | `sudo docker` — socket is `root:docker srw-rw----`, `hermes` user not in group `docker`; **no passwordless sudo in cron** | `ignis` (confirmed running; inspect requires root) | +| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **61** — main Docker host (**verified 2026-10-31**) | +| CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**verified 2026-10-31**) | +| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** — `thirsty_payne`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29** | +| CT1001 `docker-01` (`100.121.12.31`, **on `m93p`**) | `pct exec 1001 -- docker ps` via `m93p` | **1** — `sshwifty`. **Added 2026-10-31** — a fourth Docker host, previously undocumented | +| CT460 `hermes` | `sudo docker` — socket is `root:docker srw-rw----`, `hermes` user not in group `docker`; **no passwordless sudo in cron** | `ignis` — **running** (`Up 35 hours`), verified via `pct exec 460 -- docker ps` from the PVE host | -## CT200 live inventory (verified 2026-10-11, ~50 containers) -`guacamole-guacamole-1`, `telegram-bot-api`, `ytdlp-web-player`, `homarr`, `overtchat-searxng`, `searxng`, `khoj-search-1`, `prowlarr`, `overtchat-app`, `postgres-newsku`, `guacamole-postgres-1`, `jellyfin`, `libre-closet`, `apprise-api`, `tt-rss-web-nginx-1`, `tt-rss-updater-1`, `tt-rss-app-1`, `tt-rss-db-1`, `audiobookshelf`, `qbittorrent`, `open-webui`, `readlater`, `readlater-extraction-worker`, `worldmonitor-redis`, `redis-cache`, `nextstep-db`, `nextstep-scheduler-1`, `nextstep-app`, `portainer`, `flaresolverr`, `sshwifty`, `8mblocal`, `save-restricted-bot`, `kokoro-tts`, `edge-tts`, `filebrowser`, `filebrowser1`, `worldmonitor-redis-rest`, `worldmonitor`, `overtchat-redis`, `chromadb`, **`ignis-ignis-1`**, `nc_app_talk_bot_ai`, `excalidraw-frontend`, `nc_app_context_agent`, `portkey-gateway-web-1`, `guacamole-guacd-1`, `lobe-chat`, `mycompose-saas-1`. +## CT200 live inventory (re-verified 2026-11-01, 61 containers) +`8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`. -**Watch this one:** `mycompose-db-1` is in `Restarting (1)` — a crash-looping database. Flagged for follow-up; root cause not investigated on 2026-10-11. +**Re-verified 2026-11-01:** the 2026-10-11 inventory listed **49** containers; the live count is now **61**. +**Corrected 2026-11-01:** the inventory list had three **misspelled** `overtchat-*` entries (written +`overchat-*`) and one **stale** entry, `worldmonitor-seeder-fast`, which is **not running** — live +`docker ps -a --filter name=worldmonitor` shows only `worldmonitor`, `worldmonitor-redis` and +`worldmonitor-redis-rest` (all `Up 35 hours`). The list above is regenerated verbatim from live +`docker ps --format '{{.Names}}'`. **CT201's Tailscale IP was also wrong** — it read +`100.116.116.116`; live `pct exec 201 -- tailscale ip -4` returns **`100.96.244.39`** (which is +also what `[[infrastructure/hosts]]` already recorded). **CT333's first container was renamed** — +`adoring_jemison` → **`thirsty_payne`**. +**12 containers were added** since: `emulatorjs`, `excalidraw-backend`, `khoj-database-1`, `khoj-sandbox-1`, +`khoj-server-1`, `mycompose-db-1`, `onlyoffice-docserver`, `quickdrop`, `react-resume-client-1`, +`secret-santa-app`, `tgvideo-bot`, `watchtower`. **No container listed on the 2026-10-11 page has +disappeared** — the inventory is a strict superset. + +**✅ `mycompose-db-1` crash-loop is FIXED.** The 2026-10-11 page flagged it as `Restarting (1)` — a +crash-looping database. Live `docker ps -a --filter name=mycompose` now shows **both** `mycompose-db-1` +and `mycompose-saas-1` as **`Up 35 hours`** (checked 2026-11-01). No longer a watch item. **Note the duplicate Ignis:** `ignis-ignis-1` on CT200 is a separate deployment from CT460's `ignis`. See [[systems/ignis]]. diff --git a/infrastructure/proxmox.md b/infrastructure/proxmox.md index 5b90a97..970da28 100644 --- a/infrastructure/proxmox.md +++ b/infrastructure/proxmox.md @@ -3,8 +3,8 @@ title: Proxmox VE type: infrastructure status: active created: 2026-07-26 -updated: 2026-10-14 -verified_on: 2026-10-14 +updated: 2026-11-01 +verified_on: 2026-11-01 confidence: high tags: [proxmox, virtualization, infrastructure, homelab] --- @@ -17,28 +17,30 @@ Hypervisor hosting all VMs and LXC containers for the homelab. ## Details - **Host:** `pve.kangaroo-eel.ts.net` (Tailscale: `100.96.100.82`) - **Node name:** `pve` -- **Version:** Proxmox VE 8.x +- **Version:** Proxmox VE **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, running kernel `7.0.14-5-pve`) — **corrected 2026-10-31**; this page previously said *"Proxmox VE 8.x"*, which was wrong. `m93p` runs the **same** `pve-manager 9.2.10` build, so the two hosts are no longer version-skewed (see the `m93p` section). - **Primary user:** `Anthony@pam` (full admin) - **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve` -## Live LXC Inventory (verified 2026-10-14 via `pct list`) +## Live LXC Inventory (verified 2026-10-31 via `pct list`) + +Re-verified 2026-11-01 03:5x AWST (stocktake): **`pct list` on `pve` and `qm list` on `pve` both still match the tables below exactly** — 17 LXCs and 4 VMs on `pve`, with CT255 `mattermost` still `running`, CT209 `cloudreve` and CT555 `elementsynapse` still `stopped`, and **no CT601 on either hypervisor** (`/etc/pve/lxc/601.conf` absent). **`m93p` re-confirmed**: `pve-manager/9.2.10` (same build as `pve`), `qm list` returns **VM 103 `cachyos-m93p` stopped** and **VM 106 `openmausbot-vm` running**, and `pct exec 1001 -- docker ps` returns `sshwifty` (CT1001). Storage percentages were refreshed against live `pvesm status` on both hosts (see the Storage section). **Note:** `m93p` SSH over the tailnet timed out on the first two attempts and succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout). > **Two Proxmox hosts exist — always qualify a guest with its host.** `pvecm nodes` is empty on > both, so they are independent and unclustered. Guest IDs also collide — **CT221 (gitea) and > CT245 (openweb) carry the same VMID on both hosts** — so an unqualified "CT221" is ambiguous. -> `m93p` (`100.104.51.98`, PVE 9.2.0) was **discovered 2026-10-14** and is documented in its own +> `m93p` (`100.104.51.98`, PVE 9.2.10) was **discovered 2026-10-14** and is documented in its own > section below. It hosts several services the wiki had placed on `pve`. | VMID | Name | Status | Purpose | |---|---|---|---| | 103 | cliproxyapi | running | Cliproxy API | -| 200 | docker | running | Main Docker host — 50+ containers (CT460's docker.sock is NOT accessible to the agent user; use CT200 for fleet inspection) | +| 200 | docker | running | Main Docker host — 60+ containers (CT460's docker.sock is NOT accessible to the agent user; use CT200 for fleet inspection) | | 201 | runtipi | running | Runtipi — Homepage, ntfy, etc. | | 205 | ollama | running | Ollama LLM inference | | 207 | webserver | running | Web server | | 209 | cloudreve | **stopped** | Cloudreve (deprecated) | | 215 | n8n | running | N8n automation | -| 255 | mattermost | **stopped** | Mattermost — unreachable since ~2026-09-20 (Tailscale peer offline, **29d** as of 2026-10-19) | +| 255 | mattermost | **running** | Mattermost — **back online 2026-10-30/31** after a ~40-day outage; `systemctl is-active mattermost` → `active`, listening on `:8065`, gateway reconnected 2026-10-30 16:36:17 | | 270 | nextcloud | running | Nextcloud (apache2 active) | | 299 | sftpgo | running | SFTPGo file server | | 333 | clawtest | running | ClawTest — Nanobot, Zeroclaw | @@ -46,9 +48,16 @@ Hypervisor hosting all VMs and LXC containers for the homelab. | 460 | hermes | running | Hermes CT460 (production) | | 465 | qwenpaw | running | QwenPaw + Paseo daemon (active) | | 475 | agentzero | running | AgentZero (`100.106.54.94`) — **not previously recorded** | -| 555 | elementsynapse | **stopped** | Element Synapse (deprecated) — also retired from Tailscale (offline 81d) | +| 555 | elementsynapse | **stopped** | Element Synapse (deprecated) — also retired from Tailscale (offline 102d) | | 600 | omniroute | **running** | OmniRoute — live server on `:20128` (see [[systems/omniroute]]) | -| 601 | omni | **stopped** | Retired successor to CT600 — stopped, `onboot=0`, offline 54d | + +**CT601 (`omni`) no longer exists.** The 2026-10-14 inventory listed it as `**stopped**`; live +`ls /etc/pve/lxc/601.conf` on **both** hypervisors returns **no such file**, and the orphaned `omni` +Tailscale peer (`100.93.204.4`) has **expired out of the tailnet entirely** — the live +`tailscale status --json` contains no peer named `omni` and no peer at that IP. The guest was +**deleted**, not merely halted. The only OmniRoute nodes left are **CT600** (`omniroute`, +`100.88.81.19`) and **CT101** (`omniroute-1`, `100.78.63.20`, on `m93p`) — both verified running with +`omniroute` listening on `0.0.0.0:20128`. → [[systems/omniroute]], [[infrastructure/hosts]] ### ⚠ Four entries moved off this host (corrected 2026-10-14) @@ -66,19 +75,26 @@ error forward. | ~~CT245 openweb~~ | `m93p` | **CT245** — same ID, different host | | ~~CT260 freshrss~~ | `m93p` | **CT1260** | -## Live QEMU Inventory (verified 2026-10-14 via `qm list`) +## Live QEMU Inventory (verified 2026-10-31 via `qm list`) | VMID | Name | Status | Notes | |---|---|---|---| -| 100 | homeassistant | running | Home Assistant (13GB RAM) | -| 110 | omarchy | running | Omarchy (13GB RAM, 106GB disk) — **new, not previously recorded** | -| 282 | ubuntu-server-xfce | **running** | Ubuntu desktop — recorded **stopped** until 2026-10-14; live `qm list` shows running | -| 310 | cachyos | **stopped** | CachyOS | +| 100 | homeassistant | running | Home Assistant (**5100 MB** RAM — the page previously said 13 GB) | +| 101 | guest-vps | running | Guest VPS (6144 MB RAM, 48 GB disk) — **was missing from this table entirely** | +| 110 | omarchy | running | Omarchy (13000 MB RAM, 106 GB disk) | +| 282 | ubuntu-server-xfce | **running** | Ubuntu desktop (5000 MB RAM, 133 GB disk) — recorded **stopped** until 2026-10-14; live `qm list` shows running | +| 310 | cachyos | **absent** | CachyOS — **no longer present in `qm list`** on `pve`; the only CachyOS guest is `cachyos-m93p` (VM 103) on `m93p`, which is stopped | | 403 | open-claw | — | OpenClaw agent; **not present in `qm list`** — it runs elsewhere (Open-Claw VM, see [[infrastructure/hosts]]) | | 401 | hermes-old | — | Legacy Hermes VM; **not present in `qm list`** — appears decommissioned/removed | **Not in either list:** VM 403 (`open-claw`) and VM 401 (`hermes-old`) are referenced by the wiki but no longer appear in `qm list`. They have either been migrated off this host or deleted. Status unconfirmed — verify before relying on either entry. +**Re-verified 2026-10-31:** live `qm list` on `pve` returns exactly **4 VMs — 100 homeassistant, 101 +guest-vps, 110 omarchy, 282 ubuntu-server-xfce, all running**. Two corrections to the 2026-10-14 table: +**VM 101 (`guest-vps`) was missing** despite running, and **VM 310 (`cachyos`) no longer exists** on this +host (it had been recorded as `stopped`; the live `qm list` has no 310 at all). Home Assistant's memory was +also corrected from *"13GB"* to the live **5100 MB**. + ## Host `m93p` (100.104.51.98) — second Proxmox hypervisor, discovered 2026-10-14 Identified because a tailnet-wide scan for `:8006` returned only `m93p` and `pve`. It had been @@ -86,17 +102,19 @@ sitting in the Tailscale node list as an unremarkable `online` peer, but no wiki as a hypervisor. - **Hostname:** `m93p` · **Tailscale:** `100.104.51.98` -- **PVE version:** **9.2.0** (kernel `7.0.14-12-pve`, `pve-manager 9.2.10`) — materially newer than `pve`'s 8.x +- **PVE version:** **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, kernel `7.0.14-12-pve`) — **corrected 2026-10-31**. The previous entry read *"9.2.0"* and claimed `m93p` was *"materially newer than pve's 8.x"*. Both hosts now run the **identical `pve-manager 9.2.10` build**; only the kernel differs (`7.0.14-12-pve` vs `7.0.14-5-pve`). The hosts are **version-identical, not version-skewed**. +- **Uptime:** **up 1 day, 12:02** at 2026-11-01 03:56 AWST (live `uptime`) — the host **rebooted ~2026-10-30 16:0x**, consistent with the CT1264 openmausbot outage below. Load average was **7.66 / 12.80 / 11.32** (1/5/15 min) at that moment — elevated, worth watching - **Not clustered with `pve`:** `pvecm nodes` empty, no `/etc/pve/corosync.conf` -### LXC Inventory (verified 2026-10-14 via `pct list` on m93p) +### LXC Inventory (verified 2026-10-31 via `pct list` on m93p) | VMID | Name | Status | Notes | |---|---|---|---| | 100 | obsidian-livesync | running | Obsidian LiveSync — ID does not collide; `pve` has no CT100 | -| 101 | omniroute | running | **An OmniRoute instance distinct from `pve`'s CT600/CT601 — see [[systems/omniroute]]; identity/role unconfirmed** | +| 101 | omniroute | running | **Second OmniRoute node**, `100.78.63.20` (`omniroute-1` Tailscale peer), `omniroute` listening on `:20128` — identity/role **now confirmed**, see [[systems/omniroute]] | | 102 | vaultwarden | running | Vaultwarden password manager | | 104 | rclone | running | Rclone | +| 105 | conduit | running | **Conduit** — iPhone calendar / health / location / contacts bridge (see [[systems/messaging-integrations]]). **Added 2026-10-31** — this guest was missing from the 2026-10-14 table despite running | | 221 | gitea | running | **This is the host behind the wiki's `gitea` node — the Gitea remote lives here, not on `pve`** | | 245 | openweb | running | Open WebUI — `pve` has no live CT245; the guest this page previously recorded on `pve` is this one | | 500 | litellm | running | LiteLLM proxy | @@ -105,47 +123,82 @@ as a hypervisor. | 1006 | freellmapi | running | FreeLLM API | | 1007 | homebridge | running | Homebridge | | 1008 | rustdesk | running | RustDesk remote access | -| 1241 | ntfy | running | Ntfy push notifications — wiki's `CT241` | +| 1241 | ntfy | running | ntfy push notifications — wiki's `CT241` | | 1260 | freshrss | running | FreshRSS feed reader — wiki's `CT260` | | 1261 | paseo | running | Paseo coding-agent daemon | | 1262 | mailflow | running | MailFlow | | 1263 | social-dashboard | running | Social dashboard | +| 1264 | openmausbot | **stopped** | OpenMausBot — **stopped**, which is why the `openmausbot` Tailscale peer is offline 9d. **Added 2026-10-31** — was missing from the 2026-10-14 table. The companion **VM106 `openmausbot-vm` is running** and holds the separate `openmausbot-vm` peer (`100.77.107.30`) | -### QEMU Inventory (verified 2026-10-14 via `qm list` on m93p) +**Re-verified 2026-10-31:** live `pct list` on `m93p` returns **19 LXCs — 18 running, 1 stopped +(CT998 nightbridge) plus CT1264 openmausbot stopped**. The 2026-10-14 table listed only **17**, omitting +two live guests: **CT105 (`conduit`)** and **CT1264 (`openmausbot`)**. Note that **VMIDs 103, 220 and 555 +do not exist on `m93p`** (`/etc/pve/lxc/{103,220,555}.conf` all absent) — CT103 (cliproxyapi), +CT220 (caddy) and CT555 (elementsynapse) are on **`pve`** only, and `pve`'s CT555 is **stopped** while +`m93p` has no 555 at all. Guest IDs still collide across hosts, so always qualify with the host. + +### QEMU Inventory (verified 2026-10-31 via `qm list` on m93p) | VMID | Name | Status | Notes | |---|---|---|---| -| 103 | cachyos-m93p | **stopped** | CachyOS | +| 103 | cachyos-m93p | **stopped** | CachyOS (6144 MB RAM, 80 GB disk) | +| 106 | openmausbot-vm | running | OpenMausBot VM (8192 MB RAM, 30 GB disk) — **Added 2026-10-31**; was missing from the 2026-10-14 table. Holds the `openmausbot-vm` Tailscale peer (`100.77.107.30`) | **Reachability note:** guest service ports on `m93p` were **not** reachable from CT460 during this pass (they bind to their own interfaces and are not Tailscale-Served). Only hypervisor `:8006` on `100.104.51.98` is open. Reach them via the host or a Tailscale Serve route before probing services. +Re-confirmed 2026-10-31: ports 3000, 4000, 5432, 8766, 8080 and 9000 were all closed from CT460. -**⚠ Follow-up:** `m93p`'s guest list is recorded here from a single `pct list` / `qm list` pass. -The pages [[systems/omniroute]], [[infrastructure/hosts]], [[systems/model-providers]] and -[[infrastructure/docker-services]] still describe single-host topology and should be re-verified -against `m93p` — in particular whether `m93p`'s `omniroute` (CT101) is a fourth OmniRoute node or -a duplicate of CT600/CT601. +**✅ Follow-up closed 2026-10-31:** `m93p`'s `omniroute` (CT101) is a **second live OmniRoute node**, +not a duplicate of CT600/CT601. Live `pct config 101` on `m93p` confirms `hostname: omniroute`, and +`omniroute` is listening on `0.0.0.0:20128` inside it; the `omniroute-1` Tailscale peer +(`100.78.63.20`) resolves to it. **CT601 no longer exists on either host** — `/etc/pve/lxc/601.conf` is +absent from `pve` and `m93p`, and the orphaned `omni` peer (`100.93.204.4`) has expired out of the +tailnet. → [[systems/omniroute]], [[infrastructure/hosts]] -## Storage (live, 2026-10-14 03:0x AWST — `pvesm status` + `df -h /mnt/ssd0`, host `pve`) +## Storage (live, 2026-11-01 03:5x AWST — `pvesm status` on both hosts) + +### Host `pve` (100.96.100.82) | Datastore | Type | Status | Total | Used | Used % | |---|---|---|---|---|---| -| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 808 GiB | **88.17 %** (`pvesm`) | -| `SSD1` | lvmthin | active | 976 GiB | 815 GiB | 87.47 % | -| `fred-qnap` | cifs | active | 13.7 TiB | 10.3 TiB | 77.22 % | -| `local` | dir | active | 67.7 GiB | 55.1 GiB | 81.68 % | -| `local-lvm` | lvmthin | active | 141 GiB | 19.0 GiB | 13.44 % | -| `iMacHDD` | cifs | **inactive** | — | — | — | +| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 700 GiB | **76.43 %** (`pvesm`) | +| `SSD1` | lvmthin | active | 932 GiB | 599 GiB | **64.32 %** | +| `fred-qnap` | **nfs** | active | 13.4 TiB | 11.0 TiB | **82.77 %** | +| `local` | dir | active | 67.7 GiB | 54.0 GiB | **79.66 %** | +| `local-lvm` | lvmthin | active | 141 GiB | 19.4 GiB | **13.75 %** | -**`ssd0` improved from the 93 % figure: now 88.17 % by `pvesm`, ~63 GiB free.** The earlier 93 % -came from `df`, which excludes reserved blocks; `pvesm` divides by the full 960 GiB. Both describe -the same ~808 GiB used. **Use the `pvesm` number when comparing across hosts** — it is the only -one that means the same thing on `m93p` too. `iMacHDD` remains inactive -(`storage 'iMacHDD' is not online`) — check the CIFS mount before using it as a target. +**`ssd0` has improved further: 88.17 % (2026-10-14) → 76.43 %**, ~169 GiB free. `fred-qnap` is now +**82.45 %** and its type changed from **`cifs` to `nfs`** (13.4 TiB total, 11.0 TiB used) — the share was +remounted over NFS rather than SMB. + +**`iMacHDD` no longer appears in `pvesm status` on `pve`** — the 2026-10-14 table listed it as an +inactive `cifs` entry. The iMac shares now live on **`m93p`** as `cifs-anthonyimac` and `cifs-imachdd` +(both active, 60.25 %). Storage was **reassigned from `pve` to `m93p`** rather than the mount failing. + +### Host `m93p` (100.104.51.98) + +| Datastore | Type | Status | Total | Used | Used % | +|---|---|---|---|---|---| +| `cifs-anthonyimac` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % | +| `cifs-imachdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % | +| `local` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % | +| `local-hdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % | +| `nfs-proxmox-31` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** | +| `nfs-proxmox-54` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** | +| `nfs-ssd0-39` | dir | active | 916 GiB | 700 GiB | **76.43 %** | +| `omb-internal` | dir | active | 93.9 GiB | 54.4 GiB | **57.92 %** | + +**`m93p` storage was not documented before 2026-10-31.** Note that `m93p` exposes the **same backing +volumes as `pve`** under different names — `nfs-proxmox-31`/`nfs-proxmox-54` mirror `pve`'s `fred-qnap` +(both 82.45 %), and `nfs-ssd0-39` mirrors `pve`'s `ssd0` (both 76.43 %). The iMac shares that were +`iMacHDD` on `pve` are `cifs-anthonyimac` / `cifs-imachdd` on `m93p`. **Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`). +**Use the `pvesm` number when comparing across hosts** — it is the only one that means the same thing on +both hypervisors. + ## API Tokens - `homepage@pve!homepage` — PVEAuditor role, privsep=0 (used by Homepage dashboard) diff --git a/infrastructure/tailscale.md b/infrastructure/tailscale.md index 4a76af4..b8e8655 100644 --- a/infrastructure/tailscale.md +++ b/infrastructure/tailscale.md @@ -3,8 +3,8 @@ title: Tailscale type: infrastructure status: active created: 2026-07-26 -updated: 2026-10-30 -verified_on: 2026-10-30 +updated: 2026-11-01 +verified_on: 2026-11-01 confidence: high tags: [tailscale, vpn, networking, infrastructure] --- @@ -19,8 +19,10 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access - **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`) - **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`) -## Tailscale Node Inventory (verified 2026-10-30) +## Tailscale Node Inventory (verified 2026-10-31) Full node list from `tailscale status` — **50 non-funnel nodes** (49 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age. +Re-verified 2026-10-31 05:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **one state transition and seven offline-age drifts, no IP changes and no missing peers.** **`mattermost` came back online**: the peer record reads `Online: true` for the first time since the outage, and the guest behind it is genuinely healthy — live `pct exec 255 -- systemctl is-active mattermost` → **`active`**, `ss -ltnp` shows `mattermost` listening on `*:8065`, `pct exec 255 -- tailscale ip -4` → `100.98.116.14`, and the gateway's own `gateway.log` records **`✓ mattermost connected` at 2026-10-30 16:36:17** (after an earlier successful reconnect at 07:19:37 the same day). CT255 is therefore **running** on `pve` — the 40-day outage is over. The **7 offline ages** were refreshed (cloudreve 97d → **98d**; conduit 53d → **55d**; elementsynapse 100d → **102d**; freshrss-new 73d → **74d**; omarchybook 8d → **9d**; openmausbot 8d → **9d**; ubuntu-server-xfce 70d → **71d**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Two container annotations were added for peers that had none: `openmausbot` (**CT1264, m93p — stopped**) and `openmausbot-vm` (**VM106, m93p**), both confirmed from live `pct list` / `qm list` on `m93p`. The `m93p` row's "peer record stale" note was updated — its peer record read `Online: false` on 10-29 and 10-30 and reads **`Online: true`** today, the third consecutive day the record disagreed with the (reachable) host. CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route. +Re-verified 2026-11-01 03:0x–04:0x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition and 5 offline-age drifts, no IP changes and no missing peers.** **`omarchymac` went offline**: the peer record reads `Online: false` with `LastSeen 2026-10-31T13:20:00Z` and `tailscale ping 100.126.173.115` times out, so this is a **genuine** transition (the JSON `OS` field reads `linux`, not a Mac). **`m93p`'s peer record reads `Online: false` again** — the **4th consecutive** stale record — while `tailscale ping 100.104.51.98` → `pong from m93p via DERP(syd) in 98ms` and SSH over the tailnet returned `up 1 day, 12:02`, so the row stays **online** and the peer record remains non-authoritative for hypervisor reachability. The **5 offline ages** were refreshed (cloudreve 98d → **99d**; freshrss-new 74d → **75d**; omarchybook 9d → **10d**; openmausbot 9d → **10d**; ubuntu-server-xfce 71d → **72d**). **`conduit` (55d) and `elementsynapse` (102d) were already correct** and are unchanged. **Every peer IP in the table still matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Container identity re-confirmed from inside the guests: `pct exec 201 -- tailscale ip -4` → `100.96.244.39` (CT201, runtipi), `pct exec 1001 -- docker ps` → `sshwifty` (CT1001 on `m93p`), CT209 `cloudreve` and CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT255 `mattermost` **running** with `systemctl is-active mattermost` → `active` and `LISTEN *:8065`. Both OmniRoute nodes re-probed: `100.88.81.19` (CT600) and `100.78.63.20` (CT101) both return **401** on `/v1/models` (alive, auth required). Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong). Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`. Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**. @@ -40,22 +42,22 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | anthonys-mac-mini | 100.76.60.37 | online | | clawtest | 100.122.189.81 | online — CT333 (pve) | | cliproxyapi | 100.83.112.94 | online — CT103 (pve) | -| cloudreve | 100.111.187.7 | offline 97d — CT209 stopped | -| conduit | 100.81.79.87 | offline 53d | +| cloudreve | 100.111.187.7 | offline 99d — CT209 stopped | +| conduit | 100.81.79.87 | offline 55d | | docker | 100.96.150.69 | online — CT200 (pve) | | docker-01 | 100.121.12.31 | online — CT1001 (m93p) | -| elementsynapse | 100.75.250.37 | offline 100d — CT555 stopped | +| elementsynapse | 100.75.250.37 | offline 102d — CT555 stopped | | freellmapi | 100.127.208.75 | online — CT1006 (m93p) | | freshrss | 100.83.60.93 | online — CT1260 (m93p) | -| freshrss-new | 100.110.180.115 | offline 73d | +| freshrss-new | 100.110.180.115 | offline 75d | | gitea | 100.82.100.128 | online — CT221 (m93p) | | guest-vps | 100.117.14.84 | online | | hermesagent | 100.98.205.112 | online — CT450 (pve) | | homeassistant | 100.68.139.5 | online | | litellm | 100.79.29.64 | online — CT500 (m93p) | -| m93p | 100.104.51.98 | online — hypervisor (peer record stale: `tailscale ping` + SSH verified 2026-10-30) | +| m93p | 100.104.51.98 | online — hypervisor (peer record reads `Online: false` again on 2026-11-01, 4th consecutive stale record; `tailscale ping` → pong) | | mailflow | 100.115.60.92 | online — CT1262 (m93p) | -| mattermost | 100.98.116.14 | offline 40d — CT255 stopped | +| mattermost | 100.98.116.14 | online — CT255 (pve) | | n8n | 100.84.124.30 | online | | nextcloud | 100.90.102.19 | online — CT270 (pve) | | ntfy | 100.67.153.116 | online — CT1241 (m93p) | @@ -63,12 +65,12 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | octop | 100.92.163.122 | online | | ollama | 100.108.44.119 | online — CT205 (pve) | | omarchy | 100.73.227.103 | online | -| omarchybook | 100.107.9.100 | offline 8d | -| omarchymac | 100.126.173.115 | online | +| omarchybook | 100.107.9.100 | offline 10d | +| omarchymac | 100.126.173.115 | offline <1d — last seen 2026-10-31 13:20Z (state transition 2026-11-01; `tailscale ping` times out) | | omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute | | omniroute-1 | 100.78.63.20 | online — CT101 (m93p) | -| openmausbot | 100.91.214.91 | offline 8d | -| openmausbot-vm | 100.77.107.30 | online | +| openmausbot | 100.91.214.91 | offline 10d — CT1264 (m93p) stopped | +| openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) | | openweb | 100.69.230.13 | online — CT245 (m93p) | | paseo | 100.72.188.93 | online — CT1261 (m93p) | | pve | 100.96.100.82 | online | @@ -80,7 +82,7 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | social-dashboard | 100.70.183.37 | online — CT1263 (m93p) | | springroll | 100.100.153.125 | online | | springroll-browser | 100.109.221.91 | online | -| ubuntu-server-xfce | 100.101.26.104 | offline 70d | +| ubuntu-server-xfce | 100.101.26.104 | offline 72d | | vaultwarden | 100.123.225.90 | online — CT102 (m93p) | | webserver | 100.82.40.23 | online — CT207 (pve) | diff --git a/log.md b/log.md index d2151d6..cdae135 100644 --- a/log.md +++ b/log.md @@ -1744,3 +1744,69 @@ recorded wrong.** `infrastructure/tailscale.md`, `systems/messaging-integrations.md`, `systems/omniroute.md`, `systems/skills-index.md`) and untracked `raw/stocktake/freshness-20261031.txt` — the mirror convention is *"(authoring COMMIT_SHA)"*, so uncommitted content stays out until the stocktake's own commit lands. + +## [2026-11-01] stocktake | General Agent Estate Wiki maintenance — Quartz build was broken by a YAML title; the "unfixed memory ranking" incident turns out to have been fixed three weeks ago + +- **BUILD FAILURE FOUND AND FIXED (this was blocking the whole stocktake).** `wiki-maintain.sh` exited **1** at + step 3/3 with an esbuild panic. Root cause was **not** esbuild: `incidents/2026-10-31-conduit-voice-local-mode-not-applying.md` + had an **unquoted YAML `title:`** containing the substring `: ` (from `"voice mode: local"`), so YAML parsed it as a + mapping entry and Quartz aborted with `bad indentation of a mapping entry (2:27)`. A scan of all **117** pages + found **exactly one** offending frontmatter line; the title is now double-quoted with escaped inner quotes. + **This is a latent trap worth remembering:** any future page title containing `": "` or `":` unquoted will + silently break the entire Quartz build, and the failure surfaces as an opaque esbuild goroutine dump rather + than a YAML error. +- **STALE PAGE — `incidents/2026-10-09-memory-recall-relevance.md` (23d) — RESOLVED, not stale.** The page's + central claim was that the recall-ranking fix *"was reported saved and is **not** present in live config"*, and + that conflict `152` was *"the only open conflict"*. **Both claims are false, and the page was the reason the + incident looked open for three weeks.** Live `~/.hermes/mnemosyne/config.yaml` reads `vec_weight: 0.5`, + `fts_weight: 0.3`, `importance_weight: 0.3` (plus `MNEMOSYNE_IMPORTANCE_WEIGHT=0.3` in the environment) — + **the weights live in Mnemosyne's own config file, not in `~/.hermes/config.yaml`'s `memory.mnemosyne` block + that the 2026-10-10 probe read.** The 2026-10-10 verification probed the wrong file and concluded "not applied". + **The original regression test now passes:** `mnemosyne recall "Are you up to date"` returns only substantive + current-state facts (scores 0.174 / 0.130 / 0.123) and the stale Uber Eats / Sonos episode + (`id 32f469aae09d56dc`, still in the DB) **no longer outranks anything**. Conflict `152` was auto-resolved + **2026-10-29 23:53:47Z** ("low-confidence LLM extraction noise") — it has not been open since. All three open + items closed; page moved to `status: resolved`. Live tier counts at 2026-11-01: working **941**, episodic + **1,124**, canonical **90**, conflicts **175** with **11 open** (ids 165–175, all one-batch extraction noise + sharing a single `fact_a_id`), working embeddings **927/941 (98.5 %)**, episodic **1,124/1,124 (100 %)** — both + above the 95 % floor, no `reindex` needed. +- **UPDATED — `systems/mnemosyne.md`:** stale 2026-10-09 tier-count block marked superseded and a live 2026-11-01 + block added, including the weights' actual home and the resolved-incident pointer. +- **UPDATED — `infrastructure/tailscale.md`:** table rebuilt from live `tailscale status --json` — **1 state + transition and 5 offline-age drifts, no IP changes, no missing peers**. `omarchymac` (100.126.173.115) went + **offline** (`Online: false`, `LastSeen 2026-10-31 13:20Z`, `tailscale ping` times out — a genuine transition; + the JSON `OS` field reads `linux`). `m93p`'s peer record reads `Online: false` **again** — the **4th + consecutive** stale record — while `tailscale ping` → `pong` and SSH returned `up 1 day, 12:02`, so the row stays + **online**. Ages refreshed: cloudreve 98→**99d**, freshrss-new 74→**75d**, omarchybook 9→**10d**, openmausbot + 9→**10d**, ubuntu-server-xfce 71→**72d**; `conduit` (55d) and `elementsynapse` (102d) were already correct. +- **UPDATED — `infrastructure/docker-services.md`:** **CT201's Tailscale IP was wrong** — the page read + `100.116.116.116`; live `pct exec 201 -- tailscale ip -4` returns **`100.96.244.39`** (which + `[[infrastructure/hosts]]` already had right). The CT200 inventory had **three misspelled `overtchat-*` + entries** (written `overchat-*`) and **one stale entry**, `worldmonitor-seeder-fast`, which is not running — + the list is now regenerated verbatim from live `docker ps`. **CT333's first container was renamed**: + `adoring_jemison` → **`thirsty_payne`**. Counts re-verified: CT200 **61**, CT201 **20**, CT333 **8**, + CT1001 **1** (`sshwifty`), CT460 `ignis` running. +- **UPDATED — `infrastructure/proxmox.md`:** storage percentages refreshed against live `pvesm status` on **both** + hosts (`pve` `fred-qnap` 82.45→**82.77 %**, `SSD1` 63.80→**64.32 %**, `local-lvm` 13.60→**13.75 %**; `m93p` + `nfs-proxmox-*` 82.45→**82.78 %**, `omb-internal` 58.00→**57.92 %**). `pct list` / `qm list` on both hosts + re-confirmed — **no CT601 on either hypervisor**, CT255 still `running`, VM 103 `cachyos-m93p` stopped, VM 106 + `openmausbot-vm` running. `m93p` uptime recorded as **up 1 day, 12:02** with a **load average of + 7.66 / 12.80 / 11.32** — elevated, worth watching. +- **UPDATED — `systems/omniroute.md` / `systems/messaging-integrations.md`:** re-probed today — both OmniRoute + nodes answer **401** on `/v1/models` (alive, auth required), CT600 and CT101 both `running`; Mattermost's + `systemctl is-active` → `active`, `LISTEN *:8065`, `/api/v4/system/ping` → **HTTP 200**. Frontmatter + `verified_on` bumped to 2026-11-01. +- **REGENERATED — `systems/skills-index.md`:** **301 unique skills** in 68 categories, **46 archived**. The count + is *unique `(category, skill_name)`* pairs, not `SKILL.md` file count — live `find` returns **302** files, and + the one difference is the known nested alias duplicate + `productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` vs `productivity/obsidian-git-sync/SKILL.md`. + **301 is correct — do not "fix" it to 302.** One skill was archived since the last run: + `hermes-runtime-worker-diagnostics` (46 archived, up from 45). +- **FRESHNESS:** the 2026-11-01 report lists **1 stale / 113 fresh / 3 no-date** of 117 pages. The single stale + page (the memory-recall incident) has been corrected and re-verified above; the report was regenerated after + the fixes, per the script → fix → re-run → commit order. +- **Probes (context, not findings):** `m93p` SSH over the tailnet timed out on the first two attempts and + succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout). `homarr` on + CT201:8102 returns **307 → 200** on `/board`, so the dashboard page's "live, 200" claim still holds. CT201's + `:8443/` root returns **502** (upstream 8082 has no listener) and `:8443/mgr` returns **200** — both still as + the homepage-dashboard page records. No secrets, tokens, passwords or keys written. diff --git a/raw/stocktake/freshness-20261031.txt b/raw/stocktake/freshness-20261031.txt new file mode 100644 index 0000000..1b4702f --- /dev/null +++ b/raw/stocktake/freshness-20261031.txt @@ -0,0 +1,127 @@ +# Wiki freshness audit — 2026-10-31 +Threshold: >21 days since `updated` + +## STALE (1) + +- [ 22d] incidents/2026-10-09-memory-recall-relevance.md (updated 2026-10-09) + +## FRESH (110) + +- [ 11d] CONTRIBUTING.md +- [ 5d] agents/hermes-chaos.md +- [ 5d] agents/hermes-production.md +- [ 11d] agents/index.md +- [ 2d] agents/nanobot.md +- [ 11d] agents/open-claw.md +- [ 20d] agents/qwenpaw.md +- [ 11d] agents/zeroclaw.md +- [ 2d] comparisons/camoufox-vs-chrome.md +- [ 11d] comparisons/hermes-vs-nanobot.md +- [ 11d] comparisons/memory-backends.md +- [ 11d] comparisons/model-routers.md +- [ 2d] comparisons/steel-vs-camoufox.md +- [ 11d] concepts/agent-memory.md +- [ 2d] concepts/browser-automation.md +- [ 11d] concepts/model-routing.md +- [ 11d] concepts/persistent-shells.md +- [ 11d] concepts/tool-calling.md +- [ 0d] current-state.md +- [ 2d] decisions/index.md +- [ 11d] decisions/keep-openrouter-as-fallback.md +- [ 11d] decisions/persistent-shell-enabled.md +- [ 2d] decisions/rollback-steel-to-camofox.md +- [ 11d] experiments/index.md +- [ 7d] incidents/2026-07-29-mnemosyne-relay.md +- [ 11d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 7d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 9d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 8d] incidents/2026-08-15-skills-index-drift.md +- [ 19d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 6d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 11d] incidents/2026-08-29-gmail-read-state.md +- [ 10d] incidents/2026-09-02-sftpgo-outage.md +- [ 20d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 2d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 5d] incidents/2026-10-03-ct450-hermes-update.md +- [ 5d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 2d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 7d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 15d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 4d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 12d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 4d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 9d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 6d] incidents/2026-10-22-conduit-memory-start-timeout.md +- [ 5d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md +- [ 5d] incidents/2026-10-24-conduit-call-request-refused.md +- [ 5d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md +- [ 3d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md +- [ 1d] incidents/2026-10-27-job-radar-budget-exhausted.md +- [ 1d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md +- [ 1d] incidents/2026-10-29-conduit-voice-silence.md +- [ 0d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md +- [ 1d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md +- [ 0d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md +- [ 0d] index.md +- [ 9d] infrastructure/backups.md +- [ 11d] infrastructure/credentials-map.md +- [ 2d] infrastructure/docker-services.md +- [ 11d] infrastructure/domains-and-tunnels.md +- [ 0d] infrastructure/hosts.md +- [ 17d] infrastructure/network-map.md +- [ 17d] infrastructure/proxmox.md +- [ 1d] infrastructure/tailscale.md +- [ 19d] queries/retained-research-answers.md +- [ 6d] reports/communities-wa-social-media-audit-2026.md +- [ 10d] reports/uber-eats/2026-07.md +- [ 10d] reports/uber-eats/2026-08.md +- [ 10d] reports/uber-eats/2026-09.md +- [ 10d] reports/uber-eats/index.md +- [ 11d] runbooks/backup-wiki.md +- [ 11d] runbooks/diagnose-docker.md +- [ 11d] runbooks/gateway-resource-resilience.md +- [ 1d] runbooks/job-radar.md +- [ 11d] runbooks/multiplexer-setup.md +- [ 5d] runbooks/provider-health.md +- [ 2d] runbooks/recover-docker-service.md +- [ 11d] runbooks/restart-browser.md +- [ 11d] runbooks/restart-hermes.md +- [ 11d] runbooks/restore-openrouter.md +- [ 11d] runbooks/test-openrouter.md +- [ 11d] runbooks/troubleshoot-tool-loop.md +- [ 5d] runbooks/update-hermes-safely.md +- [ 2d] systems/browser-backend.md +- [ 11d] systems/cloudflare-tunnel.md +- [ 1d] systems/conduit.md +- [ 5d] systems/current-profile.md +- [ 11d] systems/guanaco.md +- [ 11d] systems/headless-server.md +- [ 1d] systems/hermes-agent.md +- [ 11d] systems/hermes-ui.md +- [ 19d] systems/home-assistant.md +- [ 1d] systems/homepage-dashboard.md +- [ 20d] systems/ignis.md +- [ 5d] systems/memory-backend.md +- [ 5d] systems/messaging-integrations.md +- [ 0d] systems/mnemosyne.md +- [ 5d] systems/model-providers.md +- [ 19d] systems/nanobot.md +- [ 11d] systems/nextcloud.md +- [ 11d] systems/obsidian.md +- [ 9d] systems/omniroute.md +- [ 11d] systems/opencode-go.md +- [ 11d] systems/openrouter.md +- [ 20d] systems/paseo.md +- [ 12d] systems/push-notifications.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md +- [ 11d] systems/terminal-backend.md +- [ 11d] systems/tool-search.md + +## NO updated DATE (3) + +- SCHEMA.md +- experiments/active/example.md +- log.md + +TOTAL: 1 stale, 110 fresh, 3 no-date of 114 pages \ No newline at end of file diff --git a/raw/stocktake/freshness-20261101.txt b/raw/stocktake/freshness-20261101.txt new file mode 100644 index 0000000..841be4b --- /dev/null +++ b/raw/stocktake/freshness-20261101.txt @@ -0,0 +1,130 @@ +# Wiki freshness audit — 2026-11-01 +Threshold: >21 days since `updated` + +## STALE (0) + + +## FRESH (114) + +- [ 12d] CONTRIBUTING.md +- [ 6d] agents/hermes-chaos.md +- [ 6d] agents/hermes-production.md +- [ 12d] agents/index.md +- [ 3d] agents/nanobot.md +- [ 12d] agents/open-claw.md +- [ 21d] agents/qwenpaw.md +- [ 12d] agents/zeroclaw.md +- [ 3d] comparisons/camoufox-vs-chrome.md +- [ 12d] comparisons/hermes-vs-nanobot.md +- [ 12d] comparisons/memory-backends.md +- [ 12d] comparisons/model-routers.md +- [ 3d] comparisons/steel-vs-camoufox.md +- [ 12d] concepts/agent-memory.md +- [ 3d] concepts/browser-automation.md +- [ 12d] concepts/model-routing.md +- [ 12d] concepts/persistent-shells.md +- [ 12d] concepts/tool-calling.md +- [ 1d] current-state.md +- [ 3d] decisions/index.md +- [ 12d] decisions/keep-openrouter-as-fallback.md +- [ 12d] decisions/persistent-shell-enabled.md +- [ 3d] decisions/rollback-steel-to-camofox.md +- [ 12d] experiments/index.md +- [ 8d] incidents/2026-07-29-mnemosyne-relay.md +- [ 12d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 8d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 10d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 9d] incidents/2026-08-15-skills-index-drift.md +- [ 20d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 7d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 12d] incidents/2026-08-29-gmail-read-state.md +- [ 11d] incidents/2026-09-02-sftpgo-outage.md +- [ 21d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 3d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 6d] incidents/2026-10-03-ct450-hermes-update.md +- [ 6d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 3d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 8d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 0d] incidents/2026-10-09-memory-recall-relevance.md +- [ 16d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 5d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 13d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 5d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 10d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 7d] incidents/2026-10-22-conduit-memory-start-timeout.md +- [ 6d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md +- [ 6d] incidents/2026-10-24-conduit-call-request-refused.md +- [ 6d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md +- [ 4d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md +- [ 2d] incidents/2026-10-27-job-radar-budget-exhausted.md +- [ 2d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md +- [ 2d] incidents/2026-10-29-conduit-voice-silence.md +- [ 1d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md +- [ 2d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md +- [ 1d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md +- [ 0d] incidents/2026-10-31-asana-access-failures.md +- [ 0d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md +- [ 0d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md +- [ 0d] index.md +- [ 10d] infrastructure/backups.md +- [ 12d] infrastructure/credentials-map.md +- [ 0d] infrastructure/docker-services.md +- [ 12d] infrastructure/domains-and-tunnels.md +- [ 1d] infrastructure/hosts.md +- [ 18d] infrastructure/network-map.md +- [ 0d] infrastructure/proxmox.md +- [ 0d] infrastructure/tailscale.md +- [ 20d] queries/retained-research-answers.md +- [ 7d] reports/communities-wa-social-media-audit-2026.md +- [ 11d] reports/uber-eats/2026-07.md +- [ 11d] reports/uber-eats/2026-08.md +- [ 11d] reports/uber-eats/2026-09.md +- [ 11d] reports/uber-eats/index.md +- [ 12d] runbooks/backup-wiki.md +- [ 12d] runbooks/diagnose-docker.md +- [ 12d] runbooks/gateway-resource-resilience.md +- [ 2d] runbooks/job-radar.md +- [ 12d] runbooks/multiplexer-setup.md +- [ 6d] runbooks/provider-health.md +- [ 3d] runbooks/recover-docker-service.md +- [ 12d] runbooks/restart-browser.md +- [ 12d] runbooks/restart-hermes.md +- [ 12d] runbooks/restore-openrouter.md +- [ 12d] runbooks/test-openrouter.md +- [ 12d] runbooks/troubleshoot-tool-loop.md +- [ 6d] runbooks/update-hermes-safely.md +- [ 3d] systems/browser-backend.md +- [ 12d] systems/cloudflare-tunnel.md +- [ 0d] systems/conduit.md +- [ 6d] systems/current-profile.md +- [ 12d] systems/guanaco.md +- [ 12d] systems/headless-server.md +- [ 2d] systems/hermes-agent.md +- [ 12d] systems/hermes-ui.md +- [ 20d] systems/home-assistant.md +- [ 2d] systems/homepage-dashboard.md +- [ 21d] systems/ignis.md +- [ 6d] systems/memory-backend.md +- [ 0d] systems/messaging-integrations.md +- [ 0d] systems/mnemosyne.md +- [ 6d] systems/model-providers.md +- [ 20d] systems/nanobot.md +- [ 12d] systems/nextcloud.md +- [ 12d] systems/obsidian.md +- [ 0d] systems/omniroute.md +- [ 12d] systems/opencode-go.md +- [ 12d] systems/openrouter.md +- [ 21d] systems/paseo.md +- [ 13d] systems/push-notifications.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md +- [ 12d] systems/terminal-backend.md +- [ 12d] systems/tool-search.md + +## NO updated DATE (3) + +- SCHEMA.md +- experiments/active/example.md +- log.md + +TOTAL: 0 stale, 114 fresh, 3 no-date of 117 pages \ No newline at end of file diff --git a/systems/messaging-integrations.md b/systems/messaging-integrations.md index bedda5c..9d7d062 100644 --- a/systems/messaging-integrations.md +++ b/systems/messaging-integrations.md @@ -3,8 +3,8 @@ title: Messaging Integrations type: system status: active created: 2026-07-22 -updated: 2026-10-26 -verified_on: 2026-10-26 +updated: 2026-11-01 +verified_on: 2026-11-01 confidence: high tags: [messaging, telegram, discord, email, integrations] sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug] @@ -125,3 +125,30 @@ one of those happens — restarting the adapter while the host is stopped achiev process's** attempts, not the outage's length — do not quote it as downtime. - **Impact unchanged:** messaging path only; no cron job delivers via Mattermost. - **Not attempted:** `pct start 255` — starting a service on the PVE host is Anthony's call. + +## 2026-10-31 06:2x AWST — ✅ Mattermost RECOVERED (outage over) + +**The ~40-day Mattermost outage is over.** All three live checks pass, and the gateway log confirms +the adapter stayed connected since. + +| Check | Live result (2026-10-31) | +|---|---| +| `pct list` on `pve` | **CT255 (`mattermost`) → `running`** (was `stopped`) | +| `pct exec 255 -- systemctl is-active mattermost` | **`active`** | +| `pct exec 255 -- ss -ltn` | `LISTEN *:8065` | +| `tailscale ping 100.98.116.14` | **`pong from mattermost ... in 0s`** | +| `curl http://100.98.116.14:8065/api/v4/system/ping` | **HTTP 200** | +| Tailscale peer record | **`Online: true`** — first time since the outage began | + +- **Gateway reconnection (CT460's own `gateway.log`):** `Mattermost: authenticated as @rhino-bot` at + **2026-10-30 07:19:37**, with `✓ mattermost reconnected successfully` and a successful WebSocket + handshake at the same second. A second connect cycle at **2026-10-30 16:36:07** produced + `✓ mattermost connected` at **16:36:17**. +- **No errors since.** Grepping every `mattermost` line in `gateway.log` after `2026-10-30 16:36:17` + returns **zero** `error` / `timeout` / `disconnect` / `fail` lines — the 5-minute reconnect loop that + produced 264–268 timeouts a day is **no longer running**. +- **The prior "cause: CT255 is stopped" finding is superseded.** CT255 is running; the gateway can + reach `http://100.98.116.14:8065/` and authenticate. → [[infrastructure/proxmox]], + [[infrastructure/tailscale]] +- **Open item now closed:** the platform is no longer an outage. The home channel startup notification + was delivered successfully on reconnect (`Sent home-channel startup notification to mattermost:…`). diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index f5f321b..8cff9f9 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -3,8 +3,8 @@ title: Mnemosyne type: system status: active created: 2026-07-22 -updated: 2026-10-31 -verified_on: 2026-10-31 +updated: 2026-11-01 +verified_on: 2026-11-01 review_after: 2026-11-07 confidence: high tags: [memory, mnemosyne, backend] @@ -52,7 +52,9 @@ Primary agent memory store for Hermes. **Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below. -**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. Ranking relevance is the live issue, not coverage — see [[incidents/2026-10-09-memory-recall-relevance]]. +**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. ~~Ranking relevance is the live issue, not coverage~~ — see [[incidents/2026-10-09-memory-recall-relevance]]. + +**Tier counts (live, 2026-11-01 03:0x — the 2026-10-09 ranking incident is now RESOLVED):** working `941`, episodic `1,124`, `canonical_facts` `90`, `conflicts` `175` (**11 open** — ids **165–175**, all `contradiction`, all created in one batch at 2026-10-31 19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature), working embeddings `927/941 (98.5%)`, episodic embeddings `1,124/1,124 (100%)` — both tiers above the 95% floor, no `reindex` needed. **The recall weights are live in `~/.hermes/mnemosyne/config.yaml`** (`vec_weight: 0.5`, `fts_weight: 0.3`, `importance_weight: 0.3`, plus `MNEMOSYNE_IMPORTANCE_WEIGHT=0.3` in the environment) — **not** in `~/.hermes/config.yaml`'s `memory.mnemosyne` block, which is why the 2026-10-10 probe wrongly concluded the fix was never applied. The original regression query now returns only substantive current-state facts; the stale Uber Eats episode no longer outranks them → [[incidents/2026-10-09-memory-recall-relevance]] (**resolved**). **Tier counts (live, 2026-10-12 12:02 — health job `ea79a2107e36`, green):** working **760/786 embedded (96.7%)**, episodic **1050/1056 (99.4%)** — both above the 95% floor, no `reindex` needed (`b63ed5817212` continues to backfill). Orphan embeddings **0**. FTS5 exactly aligned: `fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876. **0 unconsolidated** working rows; **1 unresolved conflict** (threshold 10); last consolidation 0.3 h earlier; 2,252 memories total (786 working / 1,056 episodic / 1 triple); disk 15 GB free (81% used). Interpretation: coverage is *not* the live issue — ranking relevance is (see [[incidents/2026-10-09-memory-recall-relevance]]). diff --git a/systems/omniroute.md b/systems/omniroute.md index f2b6b2e..2a487d0 100644 --- a/systems/omniroute.md +++ b/systems/omniroute.md @@ -3,8 +3,8 @@ title: OmniRoute type: system status: active created: 2026-07-22 -updated: 2026-10-22 -verified_on: 2026-10-22 +updated: 2026-11-01 +verified_on: 2026-11-01 confidence: high tags: [routing, provider, model, inference] sources: [raw/configs/hermes-config-sanitized.txt] @@ -54,30 +54,31 @@ The CT600 provider key `custom:omni` (`http://omniroute:20128/v1`) remains auxil → [[systems/current-profile]] ## Current Configuration -- **Hermes provider name:** `custom:omni` (maps to provider `omniroute:20128`) -- **Base URL:** `http://omniroute:20128/v1` +- **Hermes default provider:** `custom:omni-101` → base URL `http://100.78.63.20:20128` (CT101, `m93p`), model default **`smart-route`** — **re-verified live 2026-10-31** from `~/.hermes/config.yaml` (`model.default: smart-route`, `model.provider: custom:omni-101`, `model.base_url: http://100.78.63.20:20128`) +- **Secondary provider entry:** `custom:Omniroute:20128` → base URL `http://omniroute:20128/v1` (CT600, `pve`), referenced by the `moa` reference-model set (`vertex/gemini-3.6-flash`, `kilo-gateway/kilo-auto/free`, `nvidia/deepseek-ai/deepseek-v4-pro`, `freeapi/nemotron-3-ultra-550b`) — auxiliary/MoA use, **not** the main chat default - **Host reachability:** bare name `omniroute` resolves to `100.88.81.19`; direct HTTP on `:20128` answers `401` (auth-required, service live) -- **Runs on:** Proxmox CT600 (`omniroute`, Tailscale `100.88.81.19`, onboot=1) — verified running 2026-10-11 +- **Runs on:** two live nodes — CT600 (`omniroute`, Tailscale `100.88.81.19`, on `pve`) and CT101 (`omniroute`, Tailscale `100.78.63.20`, on `m93p`) — **both verified running and listening on `0.0.0.0:20128` on 2026-10-31** - **DB path:** `/opt/omniroute/storage.sqlite` (was `/var/lib/omniroute/storage.sqlite`) - **Backup:** pre-migration config saved at `~/.hermes/config.yaml.bak.omniroute-removal` -- **Correction (2026-10-26):** the earlier *"Hermes default provider path does not use OmniRoute for main chat"* note is **stale** — the default is OmniRoute again (`custom:omni-101`, model `smart-route`). The **CT600** key `custom:omni` is still auxiliary-only. -- **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp` +- **Correction (2026-10-26):** the earlier *"Hermes default provider path does not use OmniRoute for main chat"* note is **stale** — the default is OmniRoute again (`custom:omni-101`, model `smart-route`). The **CT600** key `custom:Omniroute:20128` is still auxiliary-only (MoA reference models). +- **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp` — all on `custom:omni-101` with model `no-think/freeapi/auto` (live-verified 2026-10-31) - **Key prefix visible in config:** `sk-5c67fbdc0...` — full value not stored in wiki -## CT600 / CT601 / omniroute-1 — actual state (2026-10-11) +## CT600 / CT601 / omniroute-1 — actual state (re-verified 2026-10-31) -The wiki previously recorded CT600 as retired and CT601 as the live successor. **That is no longer true.** Live PVE and Tailscale checks return: +The wiki previously recorded CT600 as retired and CT601 as the live successor, then CT601 as stopped. +**All three framings are now superseded.** Live `pct list` on **both** hypervisors and live HTTP probes return: -| Node | Status (live 2026-10-11) | Tailscale IP | Notes | +| Node | Status (live 2026-10-31) | Tailscale IP | Notes | |---|---|---|---| -| CT600 `omniroute` | **running**, onboot=1 | `100.88.81.19` | OmniRoute listening on `0.0.0.0:20128`; `/v1/models` returns `401` (alive, auth required) | -| CT601 `omni` | **stopped**, onboot=0 | `100.93.204.4` | Tailscale peer last seen 54d ago | -| `omniroute-1` | **active** (not in PVE `pct list`) | `100.78.63.20` | Third OmniRoute node; `/v1/models` returns `401`. Registered in Hermes config as custom provider `omni-101`, which carries `skills_hub` (model `free-stack`, 5384 models, default `smart-route`) | +| CT600 `omniroute` | **running** | `100.88.81.19` | On `pve`. OmniRoute listening on `0.0.0.0:20128`; `/v1/models` and `/api/combos` both return `401` (alive, auth required) | +| CT101 `omniroute-1` | **running** | `100.78.63.20` | On `m93p`. Live `pct config 101` → `hostname: omniroute`; `omniroute` listening on `0.0.0.0:20128`; `/v1/models` and `/api/combos` both return `401`. Registered in Hermes config as `custom:omni-101` — **the live default provider** | +| CT601 `omni` | **does not exist** | — | `/etc/pve/lxc/601.conf` absent on **both** `pve` and `m93p`. The `omni` peer (`100.93.204.4`) has **expired out of the tailnet entirely** — live `tailscale status --json` has no peer named `omni` and no peer at that IP | -- Hermes config resolving to the omniroute hostname (`custom:omni` → `omniroute:20128`) therefore points at **CT600**, not CT601. -- CT600's combo API is reachable with its local key and returns combos (e.g. `go-pro`), confirming it is the serving instance. -- **Do not treat CT601 as the production OmniRoute node.** It is stopped with `onboot=0`. -- Treat `omniroute-1` (`100.78.63.20`) as a second live instance used by the `omni-101` provider. It is not visible in `pct list`, so it is likely a VM or off-PVE host; its identity was not confirmed. +- Hermes config resolving to the omniroute hostname (`custom:Omniroute:20128` → `omniroute:20128`) therefore points at **CT600**; the live default `custom:omni-101` points at **CT101**. +- **CT601 was deleted, not merely stopped** — the earlier *"stopped, onboot=0"* entry was wrong. +- **The `omniroute-1` identity is now confirmed** — it is **CT101 on `m93p`**, not "a VM or off-PVE host". The 2026-10-11 note that it was "not visible in `pct list`" was an artifact of probing only `pve`. +- **Two live OmniRoute nodes remain** (CT600, CT101), both serving `:20128` with separate credentials — a pin's usability must be probed on the node that actually serves it. ## Catalog and Routing Facts diff --git a/systems/skills-index.md b/systems/skills-index.md index 5746d11..1e18fbc 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-10-30 -verified_on: 2026-10-30 +updated: 2026-11-01 +verified_on: 2026-11-01 confidence: high tags: [skills, hermes] sources: [] @@ -12,9 +12,9 @@ sources: [] # Installed Skills -**Total installed: 302 skills** in `~/.hermes/skills/`. +**Total installed: 301 skills** in `~/.hermes/skills/`. -- **45 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them. +- **46 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them. - The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory. This page is auto-regenerated from the live skill inventory. Do not hand-edit. @@ -308,10 +308,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - `hermes-mcp-troubleshooting` — # Hermes MCP Troubleshooting -## Hermes Runtime Worker Diagnostics (1) - -- `hermes-runtime-worker-diagnostics` — # Hermes Runtime / Cron Worker Diagnostics - ## Hermes Themes (1) - `hermes-themes` — # Hermes Themes Skill