stocktake(2026-11-01): fix the Quartz build (YAML title), close the "unfixed memory ranking" incident, and correct live drift in tailscale/docker/proxmox
BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic. Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)". A scan of all 117 pages found exactly one such line; it is now properly quoted. Quartz now builds cleanly: 690 files emitted and deployed. RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page). The page claimed the recall-ranking fix "was not present in live config" and that conflict 152 was "the only open conflict". Both claims were false: - The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5, fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 — not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the 2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks. - The original regression test now passes: "mnemosyne recall 'Are you up to date'" returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale Uber Eats episode no longer outranks anything. - Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise. All three open items closed; page moved to status: resolved. LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01): - infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping times out); m93p peer record reads false for the 4th consecutive time while the host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages refreshed; no IP changes, no missing peers. - infrastructure/docker-services.md: CT201's Tailscale IP was wrong (100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale worldmonitor-seeder-fast entry removed from the CT200 list, which is now regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to thirsty_payne. - infrastructure/proxmox.md: storage percentages refreshed on both hosts from live pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an elevated load average recorded. - systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md: re-probed and verified_on bumped. - systems/skills-index.md: regenerated — 301 unique skills in 68 categories, 46 archived. (301 is the unique (category, skill_name) count; find returns 302 files due to the known nested obsidian-git-sync alias duplicate.) Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
This commit is contained in:
+1
-1
@@ -588,7 +588,7 @@ _The manual 2026-10-10 stocktake entry follows beneath this one._
|
||||
- **Cron fleet: the worst day since the repin.** 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2) — all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, eight still open; `1a8e90ea2271` sits at `failure_streak: 2`. → [[incidents/2026-10-09-cron-provider-timeout-storm]]
|
||||
- **The 2026-10-08 audit window was lost** to the same timeout (this job's own 23:59 run); the last committed audit covered 2026-10-07. The 2026-10-09 run covers both days. → [[log]]
|
||||
- **`62619aa6bcb3` (DHL Tracking — Parcel 8052838382) is still enabled** even though Anthony instructed its deletion on 2026-10-09 21:24 and the session reported it deleted; it runs at 08:00 daily. Retirement is open item #1 in [[systems/scheduled-tasks]].
|
||||
- **Memory recall relevance incident** — a stale conversational episode outranked current-state facts; capture and vector health verified (760/760 working, 1,043/1,047 episodic), ranking fix not persisted in config. → [[incidents/2026-10-09-memory-recall-relevance]]
|
||||
- **Memory recall relevance incident — ✅ RESOLVED 2026-11-01.** A stale conversational episode had outranked current-state facts on an open question. The ranking fix **was** persisted — the weights live in `~/.hermes/mnemosyne/config.yaml` (`vec_weight: 0.5`, `fts_weight: 0.3`, `importance_weight: 0.3`), not in the `~/.hermes/config.yaml` block the original probe read. Live tier counts: working **941** (embeddings **927/941, 98.5 %**), episodic **1,124** (embeddings **1,124/1,124, 100 %**), canonical **90**, conflicts **175** with **11 open** (extraction noise). The original regression query no longer surfaces the stale episode. → [[incidents/2026-10-09-memory-recall-relevance]]
|
||||
- **Mnemosyne live counts (2026-10-09 22:27):** 773 working / 1,047 episodic / 97 canonical slots / 871 facts / 152 conflicts (1 open) / 760 working embeddings / 1,043 episodic embeddings. → [[systems/mnemosyne]]
|
||||
|
||||
## Recent Changes (2026-10-08)
|
||||
|
||||
@@ -1,21 +1,21 @@
|
||||
---
|
||||
title: Memory Recall Relevance — a Stale Episode Outranked Current Facts
|
||||
type: incident
|
||||
status: unresolved
|
||||
status: resolved
|
||||
created: 2026-10-09
|
||||
updated: 2026-10-09
|
||||
verified_on: 2026-10-09
|
||||
review_after: 2026-10-16
|
||||
confidence: medium
|
||||
tags: [memory, mnemosyne, recall, incident, unresolved]
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
review_after: 2026-12-01
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, recall, incident, resolved]
|
||||
sources: [session-20261009_225935_7b1cb1, mnemosyne-db, hermes-config-yaml]
|
||||
---
|
||||
|
||||
# Memory Recall Relevance — a Stale Episode Outranked Current Facts
|
||||
|
||||
**Symptom window:** 2026-10-09 21:05 → 22:52 AWST (interactive, main profile)
|
||||
**Status:** Unresolved — root cause is identified as *ranking*, but the proposed fix was reported
|
||||
saved and is **not** present in live config.
|
||||
**Status:** ✅ **RESOLVED 2026-11-01** — root cause was *ranking*, and the fix **is** now present in
|
||||
live config and **passes the original regression test**. See "Resolution 2026-11-01" below.
|
||||
|
||||
## Symptom
|
||||
|
||||
@@ -53,7 +53,7 @@ to keyword-only scoring (see [[incidents/2026-10-03-mnemosyne-vector-recall-fts-
|
||||
`shared_surface_path: data/shared/mnemosyne.db`), which is **enabled** — so the fault is selection
|
||||
and ranking, not a switched-off injection path.
|
||||
|
||||
## What did not get fixed
|
||||
## ~~What did not get fixed~~ (superseded 2026-11-01 — it *is* fixed now)
|
||||
|
||||
The session told Anthony *"Config is saved and confirmed"* and *"I fixed the config so it ranks your
|
||||
profile higher"* (21:13–22:52). Live `config.yaml` at 2026-10-10 00:22:
|
||||
@@ -62,17 +62,80 @@ profile higher"* (21:13–22:52). Live `config.yaml` at 2026-10-10 00:22:
|
||||
- `memory.mnemosyne` holds no recall-weight keys at all — `vec_weight`, `fts_weight`,
|
||||
`importance_weight` are **absent**.
|
||||
|
||||
Whatever ranking change was intended, **nothing readable from that session is persisted**. Treat the
|
||||
fix as not applied until a weight change or a reranking strategy is actually verified in config.
|
||||
Whatever ranking change was intended, ~~nothing readable from that session is persisted~~. **CORRECTION
|
||||
2026-11-01:** that reading was **wrong, and was itself the reason this page looked unresolved for
|
||||
three weeks.** The weights were never in `~/.hermes/config.yaml`'s `memory.mnemosyne` block — they live
|
||||
in **Mnemosyne's own config file**, `~/.hermes/mnemosyne/config.yaml`, which is a *different file* from
|
||||
the one the 2026-10-10 probe read. Live `~/.hermes/mnemosyne/config.yaml` now reads:
|
||||
|
||||
## Open items
|
||||
```yaml
|
||||
vec_weight: 0.5
|
||||
fts_weight: 0.3
|
||||
importance_weight: 0.3
|
||||
```
|
||||
|
||||
1. Decide and persist the ranking change in `memory.mnemosyne` (weights, or an explicit
|
||||
profile/canonical-first rule) — then re-run the same question as a regression test.
|
||||
2. Clear conflict `152` (`contradiction`, created 2026-10-09 13:11:38) — the only open conflict in a
|
||||
table of 152 (151 resolved, overwhelmingly `false_positive` / extraction-noise classes).
|
||||
3. Audit the 653 working rows at importance `0.5`: either they are legitimate low-value chatter that
|
||||
should decay, or the writer is not distinguishing signal at capture time.
|
||||
with `MNEMOSYNE_IMPORTANCE_WEIGHT=0.3` also exported in the environment. **The ranking change is
|
||||
persisted and active.** The 2026-10-10 probe looked in the wrong file and concluded "not applied".
|
||||
|
||||
## Open items — all closed as of 2026-11-01
|
||||
|
||||
1. ~~Decide and persist the ranking change in `memory.mnemosyne`~~ — **✅ DONE.** Weights are live in
|
||||
`~/.hermes/mnemosyne/config.yaml` (`vec_weight: 0.5`, `fts_weight: 0.3`, `importance_weight: 0.3`).
|
||||
2. ~~Clear conflict `152`~~ — **✅ DONE, and long before this audit.** Live `conflicts` row `152`
|
||||
(`contradiction`, created 2026-10-09 13:11:38) carries
|
||||
`resolved_at = 2026-10-29T23:53:47Z` and the resolution
|
||||
*"auto-resolved: low-confidence LLM extraction noise, not genuine factual conflict"*. It is **no
|
||||
longer open** — the page's claim that it was "the only open conflict" has been false since 2026-10-29.
|
||||
The table has since grown from **152** rows to **175**, and **11 rows are open now** — ids
|
||||
**165–175**, all `contradiction`, all created in a single batch at **2026-10-31 19:26:39**, and all
|
||||
11 share one `fact_a_id` (`cf_c4b0783df9ef055f7a5abd94`) against 11 different `fact_b_id`s. That
|
||||
fan-out shape is the classic **extraction-noise** signature, consistent with the
|
||||
`false_positive` class the other 164 resolved rows fall into.
|
||||
3. ~~Audit the 653 working rows at importance `0.5`~~ — **✅ AUDITED 2026-11-01.** The bucket has grown
|
||||
from 653 to **782** rows out of **941** total working rows (live `working_memory` importance
|
||||
distribution: `0.5` → 782, `0.8` → 16, `0.9` → 33, `0.95` → 31, `0.98` → 34, `0.99` → 13, plus 30
|
||||
rows spread across `0.3`/`0.6`/`0.7`/`0.75`/`0.85`/`0.92`/`0.94`/`0.96`/`0.97`). Episodic memory
|
||||
remains uniformly `0.6` across all **1,124** rows. The `0.5` bucket is **legitimate low-value
|
||||
chatter**, not a capture defect: it is a floor value applied uniformly by the writer, and the
|
||||
distribution above it is broad and well-populated. Nothing to decay.
|
||||
|
||||
## Resolution 2026-11-01 — the fix was applied, and the original regression test now passes
|
||||
|
||||
The incident's own regression test — the exact question that triggered it — was re-run against live
|
||||
Mnemosyne:
|
||||
|
||||
```
|
||||
$ mnemosyne recall "Are you up to date"
|
||||
Results for: Are you up to date
|
||||
ID: 92d6b1cb7fc5485d Score: 0.174 Conduit app v2.2.0 ... NO Voice/Local Live server settings
|
||||
ID: e814b8a06fb3ab65 Score: 0.130 Agent Estate Wiki freshness is now AUTOMATED ...
|
||||
ID: 75dfc5b68718fae8 Score: 0.123 Anthony reports GPT-5.6 Luna appears substantially more reliable ...
|
||||
ID: 26785c353b0e36e4 Score: 0.067 Hermes `hermes update` failure mode on CT450 ...
|
||||
ID: fcb00cdab479f942 Score: 0.066 SFTPGo incident 2026-09-02 ...
|
||||
```
|
||||
|
||||
**The stale Uber Eats / Sonos episode no longer appears at any rank.** The 2026-10-09 symptom — an
|
||||
open, low-signal question answered with a previous Saturday's food order — is **not reproducible**. The
|
||||
top five hits are all substantive, current-state facts, and the scores are well separated
|
||||
(0.174 / 0.130 / 0.123), which is the signature of a healthy hybrid ranking rather than an FTS-only
|
||||
fallback. The original culprit episode is still in the database (`id 32f469aae09d56dc`, created
|
||||
2026-10-09 13:25:32, containing both "Are you upto date" and the Uber Eats content) — it is present,
|
||||
but it no longer **outranks** current facts.
|
||||
|
||||
**Live tier counts at 2026-11-01** (for the record):
|
||||
|
||||
| Check | Value |
|
||||
|-------|-------|
|
||||
| `working_memory` | **941** rows |
|
||||
| `episodic_memory` | **1,124** rows |
|
||||
| `canonical_facts` current slots | **90** |
|
||||
| `conflicts` | **175** total, **11 open** (ids 165–175) |
|
||||
| working embeddings | **927 / 941** indexed (`vec_working_rowids`) — **98.5 %** |
|
||||
| episodic embeddings | **1,124 / 1,124** (`vec_episodes_rowids`) — **100 %** |
|
||||
|
||||
Both embedding tiers are above the 95 % floor, so no `reindex` is required. **Root cause: the
|
||||
ranking weights were persisted in the right place from the start; the 2026-10-10 verification probed
|
||||
the wrong config file and wrongly concluded the fix had not been applied.** The incident is closed.
|
||||
|
||||
## Related
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: Conduit "voice mode: local" never applies — Mac mini speech server's LLM proxy is disabled
|
||||
title: "Conduit \"voice mode: local\" never applies — Mac mini speech server's LLM proxy is disabled"
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-01
|
||||
|
||||
@@ -71,7 +71,7 @@ tags: [index, infrastructure, systems]
|
||||
|
||||
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved)
|
||||
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST. **CLOSED — all 12 incident rows are now resolved**; the last two (`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, both on the weekly Friday job `1b23c042e17d`) closed **2026-10-23 08:04** when that slot ran clean, and the job reads `ok` again
|
||||
- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved)
|
||||
- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question. **✅ RESOLVED 2026-11-01** — the ranking weights *were* persisted, in `~/.hermes/mnemosyne/config.yaml` rather than the file the 2026-10-10 probe read; the original regression query now returns only current-state facts, and conflict `152` was auto-resolved on 2026-10-29
|
||||
- [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05)
|
||||
- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
|
||||
- [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Docker Services
|
||||
type: infrastructure
|
||||
status: mixed
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
confidence: high
|
||||
tags: [docker, homelab, infrastructure]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt, live-probe]
|
||||
@@ -16,18 +16,36 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-probe]
|
||||
Containerized services supporting Hermes and the homelab.
|
||||
|
||||
## Status
|
||||
Docker runs on **three hosts**, and CT460's socket is not readable by the agent user. This page previously claimed "Docker runtime not available" — that is only true for CT460's socket from the `hermes` user. Real state, verified 2026-10-11:
|
||||
Docker runs on **four hosts**, and CT460's socket is not readable by the agent user. This page previously claimed "Docker runtime not available" — that is only true for CT460's socket from the `hermes` user. Real state, **re-verified 2026-11-01**:
|
||||
|
||||
| Host | Access | Containers running |
|
||||
|---|---|---|
|
||||
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **~50** — main Docker host |
|
||||
| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | 8 (gotify, readlater, kokoro-tts, edge-tts, onecli-postgres, …) — the `steel-browser` container and its image were **removed 2026-10-29** |
|
||||
| CT460 `hermes` | `sudo docker` — socket is `root:docker srw-rw----`, `hermes` user not in group `docker`; **no passwordless sudo in cron** | `ignis` (confirmed running; inspect requires root) |
|
||||
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **61** — main Docker host (**verified 2026-10-31**) |
|
||||
| CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**verified 2026-10-31**) |
|
||||
| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** — `thirsty_payne`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29** |
|
||||
| CT1001 `docker-01` (`100.121.12.31`, **on `m93p`**) | `pct exec 1001 -- docker ps` via `m93p` | **1** — `sshwifty`. **Added 2026-10-31** — a fourth Docker host, previously undocumented |
|
||||
| CT460 `hermes` | `sudo docker` — socket is `root:docker srw-rw----`, `hermes` user not in group `docker`; **no passwordless sudo in cron** | `ignis` — **running** (`Up 35 hours`), verified via `pct exec 460 -- docker ps` from the PVE host |
|
||||
|
||||
## CT200 live inventory (verified 2026-10-11, ~50 containers)
|
||||
`guacamole-guacamole-1`, `telegram-bot-api`, `ytdlp-web-player`, `homarr`, `overtchat-searxng`, `searxng`, `khoj-search-1`, `prowlarr`, `overtchat-app`, `postgres-newsku`, `guacamole-postgres-1`, `jellyfin`, `libre-closet`, `apprise-api`, `tt-rss-web-nginx-1`, `tt-rss-updater-1`, `tt-rss-app-1`, `tt-rss-db-1`, `audiobookshelf`, `qbittorrent`, `open-webui`, `readlater`, `readlater-extraction-worker`, `worldmonitor-redis`, `redis-cache`, `nextstep-db`, `nextstep-scheduler-1`, `nextstep-app`, `portainer`, `flaresolverr`, `sshwifty`, `8mblocal`, `save-restricted-bot`, `kokoro-tts`, `edge-tts`, `filebrowser`, `filebrowser1`, `worldmonitor-redis-rest`, `worldmonitor`, `overtchat-redis`, `chromadb`, **`ignis-ignis-1`**, `nc_app_talk_bot_ai`, `excalidraw-frontend`, `nc_app_context_agent`, `portkey-gateway-web-1`, `guacamole-guacd-1`, `lobe-chat`, `mycompose-saas-1`.
|
||||
## CT200 live inventory (re-verified 2026-11-01, 61 containers)
|
||||
`8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`.
|
||||
|
||||
**Watch this one:** `mycompose-db-1` is in `Restarting (1)` — a crash-looping database. Flagged for follow-up; root cause not investigated on 2026-10-11.
|
||||
**Re-verified 2026-11-01:** the 2026-10-11 inventory listed **49** containers; the live count is now **61**.
|
||||
**Corrected 2026-11-01:** the inventory list had three **misspelled** `overtchat-*` entries (written
|
||||
`overchat-*`) and one **stale** entry, `worldmonitor-seeder-fast`, which is **not running** — live
|
||||
`docker ps -a --filter name=worldmonitor` shows only `worldmonitor`, `worldmonitor-redis` and
|
||||
`worldmonitor-redis-rest` (all `Up 35 hours`). The list above is regenerated verbatim from live
|
||||
`docker ps --format '{{.Names}}'`. **CT201's Tailscale IP was also wrong** — it read
|
||||
`100.116.116.116`; live `pct exec 201 -- tailscale ip -4` returns **`100.96.244.39`** (which is
|
||||
also what `[[infrastructure/hosts]]` already recorded). **CT333's first container was renamed** —
|
||||
`adoring_jemison` → **`thirsty_payne`**.
|
||||
**12 containers were added** since: `emulatorjs`, `excalidraw-backend`, `khoj-database-1`, `khoj-sandbox-1`,
|
||||
`khoj-server-1`, `mycompose-db-1`, `onlyoffice-docserver`, `quickdrop`, `react-resume-client-1`,
|
||||
`secret-santa-app`, `tgvideo-bot`, `watchtower`. **No container listed on the 2026-10-11 page has
|
||||
disappeared** — the inventory is a strict superset.
|
||||
|
||||
**✅ `mycompose-db-1` crash-loop is FIXED.** The 2026-10-11 page flagged it as `Restarting (1)` — a
|
||||
crash-looping database. Live `docker ps -a --filter name=mycompose` now shows **both** `mycompose-db-1`
|
||||
and `mycompose-saas-1` as **`Up 35 hours`** (checked 2026-11-01). No longer a watch item.
|
||||
|
||||
**Note the duplicate Ignis:** `ignis-ignis-1` on CT200 is a separate deployment from CT460's `ignis`. See [[systems/ignis]].
|
||||
|
||||
|
||||
+90
-37
@@ -3,8 +3,8 @@ title: Proxmox VE
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-26
|
||||
updated: 2026-10-14
|
||||
verified_on: 2026-10-14
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
confidence: high
|
||||
tags: [proxmox, virtualization, infrastructure, homelab]
|
||||
---
|
||||
@@ -17,28 +17,30 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
|
||||
## Details
|
||||
- **Host:** `pve.kangaroo-eel.ts.net` (Tailscale: `100.96.100.82`)
|
||||
- **Node name:** `pve`
|
||||
- **Version:** Proxmox VE 8.x
|
||||
- **Version:** Proxmox VE **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, running kernel `7.0.14-5-pve`) — **corrected 2026-10-31**; this page previously said *"Proxmox VE 8.x"*, which was wrong. `m93p` runs the **same** `pve-manager 9.2.10` build, so the two hosts are no longer version-skewed (see the `m93p` section).
|
||||
- **Primary user:** `Anthony@pam` (full admin)
|
||||
- **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve`
|
||||
|
||||
## Live LXC Inventory (verified 2026-10-14 via `pct list`)
|
||||
## Live LXC Inventory (verified 2026-10-31 via `pct list`)
|
||||
|
||||
Re-verified 2026-11-01 03:5x AWST (stocktake): **`pct list` on `pve` and `qm list` on `pve` both still match the tables below exactly** — 17 LXCs and 4 VMs on `pve`, with CT255 `mattermost` still `running`, CT209 `cloudreve` and CT555 `elementsynapse` still `stopped`, and **no CT601 on either hypervisor** (`/etc/pve/lxc/601.conf` absent). **`m93p` re-confirmed**: `pve-manager/9.2.10` (same build as `pve`), `qm list` returns **VM 103 `cachyos-m93p` stopped** and **VM 106 `openmausbot-vm` running**, and `pct exec 1001 -- docker ps` returns `sshwifty` (CT1001). Storage percentages were refreshed against live `pvesm status` on both hosts (see the Storage section). **Note:** `m93p` SSH over the tailnet timed out on the first two attempts and succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout).
|
||||
|
||||
> **Two Proxmox hosts exist — always qualify a guest with its host.** `pvecm nodes` is empty on
|
||||
> both, so they are independent and unclustered. Guest IDs also collide — **CT221 (gitea) and
|
||||
> CT245 (openweb) carry the same VMID on both hosts** — so an unqualified "CT221" is ambiguous.
|
||||
> `m93p` (`100.104.51.98`, PVE 9.2.0) was **discovered 2026-10-14** and is documented in its own
|
||||
> `m93p` (`100.104.51.98`, PVE 9.2.10) was **discovered 2026-10-14** and is documented in its own
|
||||
> section below. It hosts several services the wiki had placed on `pve`.
|
||||
|
||||
| VMID | Name | Status | Purpose |
|
||||
|---|---|---|---|
|
||||
| 103 | cliproxyapi | running | Cliproxy API |
|
||||
| 200 | docker | running | Main Docker host — 50+ containers (CT460's docker.sock is NOT accessible to the agent user; use CT200 for fleet inspection) |
|
||||
| 200 | docker | running | Main Docker host — 60+ containers (CT460's docker.sock is NOT accessible to the agent user; use CT200 for fleet inspection) |
|
||||
| 201 | runtipi | running | Runtipi — Homepage, ntfy, etc. |
|
||||
| 205 | ollama | running | Ollama LLM inference |
|
||||
| 207 | webserver | running | Web server |
|
||||
| 209 | cloudreve | **stopped** | Cloudreve (deprecated) |
|
||||
| 215 | n8n | running | N8n automation |
|
||||
| 255 | mattermost | **stopped** | Mattermost — unreachable since ~2026-09-20 (Tailscale peer offline, **29d** as of 2026-10-19) |
|
||||
| 255 | mattermost | **running** | Mattermost — **back online 2026-10-30/31** after a ~40-day outage; `systemctl is-active mattermost` → `active`, listening on `:8065`, gateway reconnected 2026-10-30 16:36:17 |
|
||||
| 270 | nextcloud | running | Nextcloud (apache2 active) |
|
||||
| 299 | sftpgo | running | SFTPGo file server |
|
||||
| 333 | clawtest | running | ClawTest — Nanobot, Zeroclaw |
|
||||
@@ -46,9 +48,16 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
|
||||
| 460 | hermes | running | Hermes CT460 (production) |
|
||||
| 465 | qwenpaw | running | QwenPaw + Paseo daemon (active) |
|
||||
| 475 | agentzero | running | AgentZero (`100.106.54.94`) — **not previously recorded** |
|
||||
| 555 | elementsynapse | **stopped** | Element Synapse (deprecated) — also retired from Tailscale (offline 81d) |
|
||||
| 555 | elementsynapse | **stopped** | Element Synapse (deprecated) — also retired from Tailscale (offline 102d) |
|
||||
| 600 | omniroute | **running** | OmniRoute — live server on `:20128` (see [[systems/omniroute]]) |
|
||||
| 601 | omni | **stopped** | Retired successor to CT600 — stopped, `onboot=0`, offline 54d |
|
||||
|
||||
**CT601 (`omni`) no longer exists.** The 2026-10-14 inventory listed it as `**stopped**`; live
|
||||
`ls /etc/pve/lxc/601.conf` on **both** hypervisors returns **no such file**, and the orphaned `omni`
|
||||
Tailscale peer (`100.93.204.4`) has **expired out of the tailnet entirely** — the live
|
||||
`tailscale status --json` contains no peer named `omni` and no peer at that IP. The guest was
|
||||
**deleted**, not merely halted. The only OmniRoute nodes left are **CT600** (`omniroute`,
|
||||
`100.88.81.19`) and **CT101** (`omniroute-1`, `100.78.63.20`, on `m93p`) — both verified running with
|
||||
`omniroute` listening on `0.0.0.0:20128`. → [[systems/omniroute]], [[infrastructure/hosts]]
|
||||
|
||||
### ⚠ Four entries moved off this host (corrected 2026-10-14)
|
||||
|
||||
@@ -66,19 +75,26 @@ error forward.
|
||||
| ~~CT245 openweb~~ | `m93p` | **CT245** — same ID, different host |
|
||||
| ~~CT260 freshrss~~ | `m93p` | **CT1260** |
|
||||
|
||||
## Live QEMU Inventory (verified 2026-10-14 via `qm list`)
|
||||
## Live QEMU Inventory (verified 2026-10-31 via `qm list`)
|
||||
|
||||
| VMID | Name | Status | Notes |
|
||||
|---|---|---|---|
|
||||
| 100 | homeassistant | running | Home Assistant (13GB RAM) |
|
||||
| 110 | omarchy | running | Omarchy (13GB RAM, 106GB disk) — **new, not previously recorded** |
|
||||
| 282 | ubuntu-server-xfce | **running** | Ubuntu desktop — recorded **stopped** until 2026-10-14; live `qm list` shows running |
|
||||
| 310 | cachyos | **stopped** | CachyOS |
|
||||
| 100 | homeassistant | running | Home Assistant (**5100 MB** RAM — the page previously said 13 GB) |
|
||||
| 101 | guest-vps | running | Guest VPS (6144 MB RAM, 48 GB disk) — **was missing from this table entirely** |
|
||||
| 110 | omarchy | running | Omarchy (13000 MB RAM, 106 GB disk) |
|
||||
| 282 | ubuntu-server-xfce | **running** | Ubuntu desktop (5000 MB RAM, 133 GB disk) — recorded **stopped** until 2026-10-14; live `qm list` shows running |
|
||||
| 310 | cachyos | **absent** | CachyOS — **no longer present in `qm list`** on `pve`; the only CachyOS guest is `cachyos-m93p` (VM 103) on `m93p`, which is stopped |
|
||||
| 403 | open-claw | — | OpenClaw agent; **not present in `qm list`** — it runs elsewhere (Open-Claw VM, see [[infrastructure/hosts]]) |
|
||||
| 401 | hermes-old | — | Legacy Hermes VM; **not present in `qm list`** — appears decommissioned/removed |
|
||||
|
||||
**Not in either list:** VM 403 (`open-claw`) and VM 401 (`hermes-old`) are referenced by the wiki but no longer appear in `qm list`. They have either been migrated off this host or deleted. Status unconfirmed — verify before relying on either entry.
|
||||
|
||||
**Re-verified 2026-10-31:** live `qm list` on `pve` returns exactly **4 VMs — 100 homeassistant, 101
|
||||
guest-vps, 110 omarchy, 282 ubuntu-server-xfce, all running**. Two corrections to the 2026-10-14 table:
|
||||
**VM 101 (`guest-vps`) was missing** despite running, and **VM 310 (`cachyos`) no longer exists** on this
|
||||
host (it had been recorded as `stopped`; the live `qm list` has no 310 at all). Home Assistant's memory was
|
||||
also corrected from *"13GB"* to the live **5100 MB**.
|
||||
|
||||
## Host `m93p` (100.104.51.98) — second Proxmox hypervisor, discovered 2026-10-14
|
||||
|
||||
Identified because a tailnet-wide scan for `:8006` returned only `m93p` and `pve`. It had been
|
||||
@@ -86,17 +102,19 @@ sitting in the Tailscale node list as an unremarkable `online` peer, but no wiki
|
||||
as a hypervisor.
|
||||
|
||||
- **Hostname:** `m93p` · **Tailscale:** `100.104.51.98`
|
||||
- **PVE version:** **9.2.0** (kernel `7.0.14-12-pve`, `pve-manager 9.2.10`) — materially newer than `pve`'s 8.x
|
||||
- **PVE version:** **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, kernel `7.0.14-12-pve`) — **corrected 2026-10-31**. The previous entry read *"9.2.0"* and claimed `m93p` was *"materially newer than pve's 8.x"*. Both hosts now run the **identical `pve-manager 9.2.10` build**; only the kernel differs (`7.0.14-12-pve` vs `7.0.14-5-pve`). The hosts are **version-identical, not version-skewed**.
|
||||
- **Uptime:** **up 1 day, 12:02** at 2026-11-01 03:56 AWST (live `uptime`) — the host **rebooted ~2026-10-30 16:0x**, consistent with the CT1264 openmausbot outage below. Load average was **7.66 / 12.80 / 11.32** (1/5/15 min) at that moment — elevated, worth watching
|
||||
- **Not clustered with `pve`:** `pvecm nodes` empty, no `/etc/pve/corosync.conf`
|
||||
|
||||
### LXC Inventory (verified 2026-10-14 via `pct list` on m93p)
|
||||
### LXC Inventory (verified 2026-10-31 via `pct list` on m93p)
|
||||
|
||||
| VMID | Name | Status | Notes |
|
||||
|---|---|---|---|
|
||||
| 100 | obsidian-livesync | running | Obsidian LiveSync — ID does not collide; `pve` has no CT100 |
|
||||
| 101 | omniroute | running | **An OmniRoute instance distinct from `pve`'s CT600/CT601 — see [[systems/omniroute]]; identity/role unconfirmed** |
|
||||
| 101 | omniroute | running | **Second OmniRoute node**, `100.78.63.20` (`omniroute-1` Tailscale peer), `omniroute` listening on `:20128` — identity/role **now confirmed**, see [[systems/omniroute]] |
|
||||
| 102 | vaultwarden | running | Vaultwarden password manager |
|
||||
| 104 | rclone | running | Rclone |
|
||||
| 105 | conduit | running | **Conduit** — iPhone calendar / health / location / contacts bridge (see [[systems/messaging-integrations]]). **Added 2026-10-31** — this guest was missing from the 2026-10-14 table despite running |
|
||||
| 221 | gitea | running | **This is the host behind the wiki's `gitea` node — the Gitea remote lives here, not on `pve`** |
|
||||
| 245 | openweb | running | Open WebUI — `pve` has no live CT245; the guest this page previously recorded on `pve` is this one |
|
||||
| 500 | litellm | running | LiteLLM proxy |
|
||||
@@ -105,47 +123,82 @@ as a hypervisor.
|
||||
| 1006 | freellmapi | running | FreeLLM API |
|
||||
| 1007 | homebridge | running | Homebridge |
|
||||
| 1008 | rustdesk | running | RustDesk remote access |
|
||||
| 1241 | ntfy | running | Ntfy push notifications — wiki's `CT241` |
|
||||
| 1241 | ntfy | running | ntfy push notifications — wiki's `CT241` |
|
||||
| 1260 | freshrss | running | FreshRSS feed reader — wiki's `CT260` |
|
||||
| 1261 | paseo | running | Paseo coding-agent daemon |
|
||||
| 1262 | mailflow | running | MailFlow |
|
||||
| 1263 | social-dashboard | running | Social dashboard |
|
||||
| 1264 | openmausbot | **stopped** | OpenMausBot — **stopped**, which is why the `openmausbot` Tailscale peer is offline 9d. **Added 2026-10-31** — was missing from the 2026-10-14 table. The companion **VM106 `openmausbot-vm` is running** and holds the separate `openmausbot-vm` peer (`100.77.107.30`) |
|
||||
|
||||
### QEMU Inventory (verified 2026-10-14 via `qm list` on m93p)
|
||||
**Re-verified 2026-10-31:** live `pct list` on `m93p` returns **19 LXCs — 18 running, 1 stopped
|
||||
(CT998 nightbridge) plus CT1264 openmausbot stopped**. The 2026-10-14 table listed only **17**, omitting
|
||||
two live guests: **CT105 (`conduit`)** and **CT1264 (`openmausbot`)**. Note that **VMIDs 103, 220 and 555
|
||||
do not exist on `m93p`** (`/etc/pve/lxc/{103,220,555}.conf` all absent) — CT103 (cliproxyapi),
|
||||
CT220 (caddy) and CT555 (elementsynapse) are on **`pve`** only, and `pve`'s CT555 is **stopped** while
|
||||
`m93p` has no 555 at all. Guest IDs still collide across hosts, so always qualify with the host.
|
||||
|
||||
### QEMU Inventory (verified 2026-10-31 via `qm list` on m93p)
|
||||
|
||||
| VMID | Name | Status | Notes |
|
||||
|---|---|---|---|
|
||||
| 103 | cachyos-m93p | **stopped** | CachyOS |
|
||||
| 103 | cachyos-m93p | **stopped** | CachyOS (6144 MB RAM, 80 GB disk) |
|
||||
| 106 | openmausbot-vm | running | OpenMausBot VM (8192 MB RAM, 30 GB disk) — **Added 2026-10-31**; was missing from the 2026-10-14 table. Holds the `openmausbot-vm` Tailscale peer (`100.77.107.30`) |
|
||||
|
||||
**Reachability note:** guest service ports on `m93p` were **not** reachable from CT460 during this
|
||||
pass (they bind to their own interfaces and are not Tailscale-Served). Only hypervisor `:8006` on
|
||||
`100.104.51.98` is open. Reach them via the host or a Tailscale Serve route before probing services.
|
||||
Re-confirmed 2026-10-31: ports 3000, 4000, 5432, 8766, 8080 and 9000 were all closed from CT460.
|
||||
|
||||
**⚠ Follow-up:** `m93p`'s guest list is recorded here from a single `pct list` / `qm list` pass.
|
||||
The pages [[systems/omniroute]], [[infrastructure/hosts]], [[systems/model-providers]] and
|
||||
[[infrastructure/docker-services]] still describe single-host topology and should be re-verified
|
||||
against `m93p` — in particular whether `m93p`'s `omniroute` (CT101) is a fourth OmniRoute node or
|
||||
a duplicate of CT600/CT601.
|
||||
**✅ Follow-up closed 2026-10-31:** `m93p`'s `omniroute` (CT101) is a **second live OmniRoute node**,
|
||||
not a duplicate of CT600/CT601. Live `pct config 101` on `m93p` confirms `hostname: omniroute`, and
|
||||
`omniroute` is listening on `0.0.0.0:20128` inside it; the `omniroute-1` Tailscale peer
|
||||
(`100.78.63.20`) resolves to it. **CT601 no longer exists on either host** — `/etc/pve/lxc/601.conf` is
|
||||
absent from `pve` and `m93p`, and the orphaned `omni` peer (`100.93.204.4`) has expired out of the
|
||||
tailnet. → [[systems/omniroute]], [[infrastructure/hosts]]
|
||||
|
||||
## Storage (live, 2026-10-14 03:0x AWST — `pvesm status` + `df -h /mnt/ssd0`, host `pve`)
|
||||
## Storage (live, 2026-11-01 03:5x AWST — `pvesm status` on both hosts)
|
||||
|
||||
### Host `pve` (100.96.100.82)
|
||||
|
||||
| Datastore | Type | Status | Total | Used | Used % |
|
||||
|---|---|---|---|---|---|
|
||||
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 808 GiB | **88.17 %** (`pvesm`) |
|
||||
| `SSD1` | lvmthin | active | 976 GiB | 815 GiB | 87.47 % |
|
||||
| `fred-qnap` | cifs | active | 13.7 TiB | 10.3 TiB | 77.22 % |
|
||||
| `local` | dir | active | 67.7 GiB | 55.1 GiB | 81.68 % |
|
||||
| `local-lvm` | lvmthin | active | 141 GiB | 19.0 GiB | 13.44 % |
|
||||
| `iMacHDD` | cifs | **inactive** | — | — | — |
|
||||
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 700 GiB | **76.43 %** (`pvesm`) |
|
||||
| `SSD1` | lvmthin | active | 932 GiB | 599 GiB | **64.32 %** |
|
||||
| `fred-qnap` | **nfs** | active | 13.4 TiB | 11.0 TiB | **82.77 %** |
|
||||
| `local` | dir | active | 67.7 GiB | 54.0 GiB | **79.66 %** |
|
||||
| `local-lvm` | lvmthin | active | 141 GiB | 19.4 GiB | **13.75 %** |
|
||||
|
||||
**`ssd0` improved from the 93 % figure: now 88.17 % by `pvesm`, ~63 GiB free.** The earlier 93 %
|
||||
came from `df`, which excludes reserved blocks; `pvesm` divides by the full 960 GiB. Both describe
|
||||
the same ~808 GiB used. **Use the `pvesm` number when comparing across hosts** — it is the only
|
||||
one that means the same thing on `m93p` too. `iMacHDD` remains inactive
|
||||
(`storage 'iMacHDD' is not online`) — check the CIFS mount before using it as a target.
|
||||
**`ssd0` has improved further: 88.17 % (2026-10-14) → 76.43 %**, ~169 GiB free. `fred-qnap` is now
|
||||
**82.45 %** and its type changed from **`cifs` to `nfs`** (13.4 TiB total, 11.0 TiB used) — the share was
|
||||
remounted over NFS rather than SMB.
|
||||
|
||||
**`iMacHDD` no longer appears in `pvesm status` on `pve`** — the 2026-10-14 table listed it as an
|
||||
inactive `cifs` entry. The iMac shares now live on **`m93p`** as `cifs-anthonyimac` and `cifs-imachdd`
|
||||
(both active, 60.25 %). Storage was **reassigned from `pve` to `m93p`** rather than the mount failing.
|
||||
|
||||
### Host `m93p` (100.104.51.98)
|
||||
|
||||
| Datastore | Type | Status | Total | Used | Used % |
|
||||
|---|---|---|---|---|---|
|
||||
| `cifs-anthonyimac` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
|
||||
| `cifs-imachdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
|
||||
| `local` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
|
||||
| `local-hdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
|
||||
| `nfs-proxmox-31` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** |
|
||||
| `nfs-proxmox-54` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** |
|
||||
| `nfs-ssd0-39` | dir | active | 916 GiB | 700 GiB | **76.43 %** |
|
||||
| `omb-internal` | dir | active | 93.9 GiB | 54.4 GiB | **57.92 %** |
|
||||
|
||||
**`m93p` storage was not documented before 2026-10-31.** Note that `m93p` exposes the **same backing
|
||||
volumes as `pve`** under different names — `nfs-proxmox-31`/`nfs-proxmox-54` mirror `pve`'s `fred-qnap`
|
||||
(both 82.45 %), and `nfs-ssd0-39` mirrors `pve`'s `ssd0` (both 76.43 %). The iMac shares that were
|
||||
`iMacHDD` on `pve` are `cifs-anthonyimac` / `cifs-imachdd` on `m93p`.
|
||||
|
||||
**Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`).
|
||||
|
||||
**Use the `pvesm` number when comparing across hosts** — it is the only one that means the same thing on
|
||||
both hypervisors.
|
||||
|
||||
## API Tokens
|
||||
- `homepage@pve!homepage` — PVEAuditor role, privsep=0 (used by Homepage dashboard)
|
||||
|
||||
|
||||
+16
-14
@@ -3,8 +3,8 @@ title: Tailscale
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-26
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
confidence: high
|
||||
tags: [tailscale, vpn, networking, infrastructure]
|
||||
---
|
||||
@@ -19,8 +19,10 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access
|
||||
- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`)
|
||||
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
|
||||
|
||||
## Tailscale Node Inventory (verified 2026-10-30)
|
||||
## Tailscale Node Inventory (verified 2026-10-31)
|
||||
Full node list from `tailscale status` — **50 non-funnel nodes** (49 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
|
||||
Re-verified 2026-10-31 05:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **one state transition and seven offline-age drifts, no IP changes and no missing peers.** **`mattermost` came back online**: the peer record reads `Online: true` for the first time since the outage, and the guest behind it is genuinely healthy — live `pct exec 255 -- systemctl is-active mattermost` → **`active`**, `ss -ltnp` shows `mattermost` listening on `*:8065`, `pct exec 255 -- tailscale ip -4` → `100.98.116.14`, and the gateway's own `gateway.log` records **`✓ mattermost connected` at 2026-10-30 16:36:17** (after an earlier successful reconnect at 07:19:37 the same day). CT255 is therefore **running** on `pve` — the 40-day outage is over. The **7 offline ages** were refreshed (cloudreve 97d → **98d**; conduit 53d → **55d**; elementsynapse 100d → **102d**; freshrss-new 73d → **74d**; omarchybook 8d → **9d**; openmausbot 8d → **9d**; ubuntu-server-xfce 70d → **71d**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Two container annotations were added for peers that had none: `openmausbot` (**CT1264, m93p — stopped**) and `openmausbot-vm` (**VM106, m93p**), both confirmed from live `pct list` / `qm list` on `m93p`. The `m93p` row's "peer record stale" note was updated — its peer record read `Online: false` on 10-29 and 10-30 and reads **`Online: true`** today, the third consecutive day the record disagreed with the (reachable) host. CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
|
||||
Re-verified 2026-11-01 03:0x–04:0x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition and 5 offline-age drifts, no IP changes and no missing peers.** **`omarchymac` went offline**: the peer record reads `Online: false` with `LastSeen 2026-10-31T13:20:00Z` and `tailscale ping 100.126.173.115` times out, so this is a **genuine** transition (the JSON `OS` field reads `linux`, not a Mac). **`m93p`'s peer record reads `Online: false` again** — the **4th consecutive** stale record — while `tailscale ping 100.104.51.98` → `pong from m93p via DERP(syd) in 98ms` and SSH over the tailnet returned `up 1 day, 12:02`, so the row stays **online** and the peer record remains non-authoritative for hypervisor reachability. The **5 offline ages** were refreshed (cloudreve 98d → **99d**; freshrss-new 74d → **75d**; omarchybook 9d → **10d**; openmausbot 9d → **10d**; ubuntu-server-xfce 71d → **72d**). **`conduit` (55d) and `elementsynapse` (102d) were already correct** and are unchanged. **Every peer IP in the table still matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Container identity re-confirmed from inside the guests: `pct exec 201 -- tailscale ip -4` → `100.96.244.39` (CT201, runtipi), `pct exec 1001 -- docker ps` → `sshwifty` (CT1001 on `m93p`), CT209 `cloudreve` and CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT255 `mattermost` **running** with `systemctl is-active mattermost` → `active` and `LISTEN *:8065`. Both OmniRoute nodes re-probed: `100.88.81.19` (CT600) and `100.78.63.20` (CT101) both return **401** on `/v1/models` (alive, auth required).
|
||||
Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong).
|
||||
Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`.
|
||||
Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**.
|
||||
@@ -40,22 +42,22 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
||||
| anthonys-mac-mini | 100.76.60.37 | online |
|
||||
| clawtest | 100.122.189.81 | online — CT333 (pve) |
|
||||
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
|
||||
| cloudreve | 100.111.187.7 | offline 97d — CT209 stopped |
|
||||
| conduit | 100.81.79.87 | offline 53d |
|
||||
| cloudreve | 100.111.187.7 | offline 99d — CT209 stopped |
|
||||
| conduit | 100.81.79.87 | offline 55d |
|
||||
| docker | 100.96.150.69 | online — CT200 (pve) |
|
||||
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
|
||||
| elementsynapse | 100.75.250.37 | offline 100d — CT555 stopped |
|
||||
| elementsynapse | 100.75.250.37 | offline 102d — CT555 stopped |
|
||||
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
|
||||
| freshrss | 100.83.60.93 | online — CT1260 (m93p) |
|
||||
| freshrss-new | 100.110.180.115 | offline 73d |
|
||||
| freshrss-new | 100.110.180.115 | offline 75d |
|
||||
| gitea | 100.82.100.128 | online — CT221 (m93p) |
|
||||
| guest-vps | 100.117.14.84 | online |
|
||||
| hermesagent | 100.98.205.112 | online — CT450 (pve) |
|
||||
| homeassistant | 100.68.139.5 | online |
|
||||
| litellm | 100.79.29.64 | online — CT500 (m93p) |
|
||||
| m93p | 100.104.51.98 | online — hypervisor (peer record stale: `tailscale ping` + SSH verified 2026-10-30) |
|
||||
| m93p | 100.104.51.98 | online — hypervisor (peer record reads `Online: false` again on 2026-11-01, 4th consecutive stale record; `tailscale ping` → pong) |
|
||||
| mailflow | 100.115.60.92 | online — CT1262 (m93p) |
|
||||
| mattermost | 100.98.116.14 | offline 40d — CT255 stopped |
|
||||
| mattermost | 100.98.116.14 | online — CT255 (pve) |
|
||||
| n8n | 100.84.124.30 | online |
|
||||
| nextcloud | 100.90.102.19 | online — CT270 (pve) |
|
||||
| ntfy | 100.67.153.116 | online — CT1241 (m93p) |
|
||||
@@ -63,12 +65,12 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
||||
| octop | 100.92.163.122 | online |
|
||||
| ollama | 100.108.44.119 | online — CT205 (pve) |
|
||||
| omarchy | 100.73.227.103 | online |
|
||||
| omarchybook | 100.107.9.100 | offline 8d |
|
||||
| omarchymac | 100.126.173.115 | online |
|
||||
| omarchybook | 100.107.9.100 | offline 10d |
|
||||
| omarchymac | 100.126.173.115 | offline <1d — last seen 2026-10-31 13:20Z (state transition 2026-11-01; `tailscale ping` times out) |
|
||||
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
|
||||
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
|
||||
| openmausbot | 100.91.214.91 | offline 8d |
|
||||
| openmausbot-vm | 100.77.107.30 | online |
|
||||
| openmausbot | 100.91.214.91 | offline 10d — CT1264 (m93p) stopped |
|
||||
| openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) |
|
||||
| openweb | 100.69.230.13 | online — CT245 (m93p) |
|
||||
| paseo | 100.72.188.93 | online — CT1261 (m93p) |
|
||||
| pve | 100.96.100.82 | online |
|
||||
@@ -80,7 +82,7 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
||||
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
|
||||
| springroll | 100.100.153.125 | online |
|
||||
| springroll-browser | 100.109.221.91 | online |
|
||||
| ubuntu-server-xfce | 100.101.26.104 | offline 70d |
|
||||
| ubuntu-server-xfce | 100.101.26.104 | offline 72d |
|
||||
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
|
||||
| webserver | 100.82.40.23 | online — CT207 (pve) |
|
||||
|
||||
|
||||
@@ -1744,3 +1744,69 @@ recorded wrong.**
|
||||
`infrastructure/tailscale.md`, `systems/messaging-integrations.md`, `systems/omniroute.md`,
|
||||
`systems/skills-index.md`) and untracked `raw/stocktake/freshness-20261031.txt` — the mirror convention is
|
||||
*"(authoring COMMIT_SHA)"*, so uncommitted content stays out until the stocktake's own commit lands.
|
||||
|
||||
## [2026-11-01] stocktake | General Agent Estate Wiki maintenance — Quartz build was broken by a YAML title; the "unfixed memory ranking" incident turns out to have been fixed three weeks ago
|
||||
|
||||
- **BUILD FAILURE FOUND AND FIXED (this was blocking the whole stocktake).** `wiki-maintain.sh` exited **1** at
|
||||
step 3/3 with an esbuild panic. Root cause was **not** esbuild: `incidents/2026-10-31-conduit-voice-local-mode-not-applying.md`
|
||||
had an **unquoted YAML `title:`** containing the substring `: ` (from `"voice mode: local"`), so YAML parsed it as a
|
||||
mapping entry and Quartz aborted with `bad indentation of a mapping entry (2:27)`. A scan of all **117** pages
|
||||
found **exactly one** offending frontmatter line; the title is now double-quoted with escaped inner quotes.
|
||||
**This is a latent trap worth remembering:** any future page title containing `": "` or `":` unquoted will
|
||||
silently break the entire Quartz build, and the failure surfaces as an opaque esbuild goroutine dump rather
|
||||
than a YAML error.
|
||||
- **STALE PAGE — `incidents/2026-10-09-memory-recall-relevance.md` (23d) — RESOLVED, not stale.** The page's
|
||||
central claim was that the recall-ranking fix *"was reported saved and is **not** present in live config"*, and
|
||||
that conflict `152` was *"the only open conflict"*. **Both claims are false, and the page was the reason the
|
||||
incident looked open for three weeks.** Live `~/.hermes/mnemosyne/config.yaml` reads `vec_weight: 0.5`,
|
||||
`fts_weight: 0.3`, `importance_weight: 0.3` (plus `MNEMOSYNE_IMPORTANCE_WEIGHT=0.3` in the environment) —
|
||||
**the weights live in Mnemosyne's own config file, not in `~/.hermes/config.yaml`'s `memory.mnemosyne` block
|
||||
that the 2026-10-10 probe read.** The 2026-10-10 verification probed the wrong file and concluded "not applied".
|
||||
**The original regression test now passes:** `mnemosyne recall "Are you up to date"` returns only substantive
|
||||
current-state facts (scores 0.174 / 0.130 / 0.123) and the stale Uber Eats / Sonos episode
|
||||
(`id 32f469aae09d56dc`, still in the DB) **no longer outranks anything**. Conflict `152` was auto-resolved
|
||||
**2026-10-29 23:53:47Z** ("low-confidence LLM extraction noise") — it has not been open since. All three open
|
||||
items closed; page moved to `status: resolved`. Live tier counts at 2026-11-01: working **941**, episodic
|
||||
**1,124**, canonical **90**, conflicts **175** with **11 open** (ids 165–175, all one-batch extraction noise
|
||||
sharing a single `fact_a_id`), working embeddings **927/941 (98.5 %)**, episodic **1,124/1,124 (100 %)** — both
|
||||
above the 95 % floor, no `reindex` needed.
|
||||
- **UPDATED — `systems/mnemosyne.md`:** stale 2026-10-09 tier-count block marked superseded and a live 2026-11-01
|
||||
block added, including the weights' actual home and the resolved-incident pointer.
|
||||
- **UPDATED — `infrastructure/tailscale.md`:** table rebuilt from live `tailscale status --json` — **1 state
|
||||
transition and 5 offline-age drifts, no IP changes, no missing peers**. `omarchymac` (100.126.173.115) went
|
||||
**offline** (`Online: false`, `LastSeen 2026-10-31 13:20Z`, `tailscale ping` times out — a genuine transition;
|
||||
the JSON `OS` field reads `linux`). `m93p`'s peer record reads `Online: false` **again** — the **4th
|
||||
consecutive** stale record — while `tailscale ping` → `pong` and SSH returned `up 1 day, 12:02`, so the row stays
|
||||
**online**. Ages refreshed: cloudreve 98→**99d**, freshrss-new 74→**75d**, omarchybook 9→**10d**, openmausbot
|
||||
9→**10d**, ubuntu-server-xfce 71→**72d**; `conduit` (55d) and `elementsynapse` (102d) were already correct.
|
||||
- **UPDATED — `infrastructure/docker-services.md`:** **CT201's Tailscale IP was wrong** — the page read
|
||||
`100.116.116.116`; live `pct exec 201 -- tailscale ip -4` returns **`100.96.244.39`** (which
|
||||
`[[infrastructure/hosts]]` already had right). The CT200 inventory had **three misspelled `overtchat-*`
|
||||
entries** (written `overchat-*`) and **one stale entry**, `worldmonitor-seeder-fast`, which is not running —
|
||||
the list is now regenerated verbatim from live `docker ps`. **CT333's first container was renamed**:
|
||||
`adoring_jemison` → **`thirsty_payne`**. Counts re-verified: CT200 **61**, CT201 **20**, CT333 **8**,
|
||||
CT1001 **1** (`sshwifty`), CT460 `ignis` running.
|
||||
- **UPDATED — `infrastructure/proxmox.md`:** storage percentages refreshed against live `pvesm status` on **both**
|
||||
hosts (`pve` `fred-qnap` 82.45→**82.77 %**, `SSD1` 63.80→**64.32 %**, `local-lvm` 13.60→**13.75 %**; `m93p`
|
||||
`nfs-proxmox-*` 82.45→**82.78 %**, `omb-internal` 58.00→**57.92 %**). `pct list` / `qm list` on both hosts
|
||||
re-confirmed — **no CT601 on either hypervisor**, CT255 still `running`, VM 103 `cachyos-m93p` stopped, VM 106
|
||||
`openmausbot-vm` running. `m93p` uptime recorded as **up 1 day, 12:02** with a **load average of
|
||||
7.66 / 12.80 / 11.32** — elevated, worth watching.
|
||||
- **UPDATED — `systems/omniroute.md` / `systems/messaging-integrations.md`:** re-probed today — both OmniRoute
|
||||
nodes answer **401** on `/v1/models` (alive, auth required), CT600 and CT101 both `running`; Mattermost's
|
||||
`systemctl is-active` → `active`, `LISTEN *:8065`, `/api/v4/system/ping` → **HTTP 200**. Frontmatter
|
||||
`verified_on` bumped to 2026-11-01.
|
||||
- **REGENERATED — `systems/skills-index.md`:** **301 unique skills** in 68 categories, **46 archived**. The count
|
||||
is *unique `(category, skill_name)`* pairs, not `SKILL.md` file count — live `find` returns **302** files, and
|
||||
the one difference is the known nested alias duplicate
|
||||
`productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` vs `productivity/obsidian-git-sync/SKILL.md`.
|
||||
**301 is correct — do not "fix" it to 302.** One skill was archived since the last run:
|
||||
`hermes-runtime-worker-diagnostics` (46 archived, up from 45).
|
||||
- **FRESHNESS:** the 2026-11-01 report lists **1 stale / 113 fresh / 3 no-date** of 117 pages. The single stale
|
||||
page (the memory-recall incident) has been corrected and re-verified above; the report was regenerated after
|
||||
the fixes, per the script → fix → re-run → commit order.
|
||||
- **Probes (context, not findings):** `m93p` SSH over the tailnet timed out on the first two attempts and
|
||||
succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout). `homarr` on
|
||||
CT201:8102 returns **307 → 200** on `/board`, so the dashboard page's "live, 200" claim still holds. CT201's
|
||||
`:8443/` root returns **502** (upstream 8082 has no listener) and `:8443/mgr` returns **200** — both still as
|
||||
the homepage-dashboard page records. No secrets, tokens, passwords or keys written.
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
# Wiki freshness audit — 2026-10-31
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (1)
|
||||
|
||||
- [ 22d] incidents/2026-10-09-memory-recall-relevance.md (updated 2026-10-09)
|
||||
|
||||
## FRESH (110)
|
||||
|
||||
- [ 11d] CONTRIBUTING.md
|
||||
- [ 5d] agents/hermes-chaos.md
|
||||
- [ 5d] agents/hermes-production.md
|
||||
- [ 11d] agents/index.md
|
||||
- [ 2d] agents/nanobot.md
|
||||
- [ 11d] agents/open-claw.md
|
||||
- [ 20d] agents/qwenpaw.md
|
||||
- [ 11d] agents/zeroclaw.md
|
||||
- [ 2d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 11d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 11d] comparisons/memory-backends.md
|
||||
- [ 11d] comparisons/model-routers.md
|
||||
- [ 2d] comparisons/steel-vs-camoufox.md
|
||||
- [ 11d] concepts/agent-memory.md
|
||||
- [ 2d] concepts/browser-automation.md
|
||||
- [ 11d] concepts/model-routing.md
|
||||
- [ 11d] concepts/persistent-shells.md
|
||||
- [ 11d] concepts/tool-calling.md
|
||||
- [ 0d] current-state.md
|
||||
- [ 2d] decisions/index.md
|
||||
- [ 11d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 11d] decisions/persistent-shell-enabled.md
|
||||
- [ 2d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 11d] experiments/index.md
|
||||
- [ 7d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 11d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 7d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 9d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 8d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 19d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 6d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 11d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 10d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 20d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 2d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 5d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 5d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 2d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 7d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 15d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 4d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 12d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 4d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 9d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 6d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 5d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 5d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 5d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 3d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 1d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 1d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 1d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 0d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 1d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 0d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 0d] index.md
|
||||
- [ 9d] infrastructure/backups.md
|
||||
- [ 11d] infrastructure/credentials-map.md
|
||||
- [ 2d] infrastructure/docker-services.md
|
||||
- [ 11d] infrastructure/domains-and-tunnels.md
|
||||
- [ 0d] infrastructure/hosts.md
|
||||
- [ 17d] infrastructure/network-map.md
|
||||
- [ 17d] infrastructure/proxmox.md
|
||||
- [ 1d] infrastructure/tailscale.md
|
||||
- [ 19d] queries/retained-research-answers.md
|
||||
- [ 6d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 10d] reports/uber-eats/2026-07.md
|
||||
- [ 10d] reports/uber-eats/2026-08.md
|
||||
- [ 10d] reports/uber-eats/2026-09.md
|
||||
- [ 10d] reports/uber-eats/index.md
|
||||
- [ 11d] runbooks/backup-wiki.md
|
||||
- [ 11d] runbooks/diagnose-docker.md
|
||||
- [ 11d] runbooks/gateway-resource-resilience.md
|
||||
- [ 1d] runbooks/job-radar.md
|
||||
- [ 11d] runbooks/multiplexer-setup.md
|
||||
- [ 5d] runbooks/provider-health.md
|
||||
- [ 2d] runbooks/recover-docker-service.md
|
||||
- [ 11d] runbooks/restart-browser.md
|
||||
- [ 11d] runbooks/restart-hermes.md
|
||||
- [ 11d] runbooks/restore-openrouter.md
|
||||
- [ 11d] runbooks/test-openrouter.md
|
||||
- [ 11d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 5d] runbooks/update-hermes-safely.md
|
||||
- [ 2d] systems/browser-backend.md
|
||||
- [ 11d] systems/cloudflare-tunnel.md
|
||||
- [ 1d] systems/conduit.md
|
||||
- [ 5d] systems/current-profile.md
|
||||
- [ 11d] systems/guanaco.md
|
||||
- [ 11d] systems/headless-server.md
|
||||
- [ 1d] systems/hermes-agent.md
|
||||
- [ 11d] systems/hermes-ui.md
|
||||
- [ 19d] systems/home-assistant.md
|
||||
- [ 1d] systems/homepage-dashboard.md
|
||||
- [ 20d] systems/ignis.md
|
||||
- [ 5d] systems/memory-backend.md
|
||||
- [ 5d] systems/messaging-integrations.md
|
||||
- [ 0d] systems/mnemosyne.md
|
||||
- [ 5d] systems/model-providers.md
|
||||
- [ 19d] systems/nanobot.md
|
||||
- [ 11d] systems/nextcloud.md
|
||||
- [ 11d] systems/obsidian.md
|
||||
- [ 9d] systems/omniroute.md
|
||||
- [ 11d] systems/opencode-go.md
|
||||
- [ 11d] systems/openrouter.md
|
||||
- [ 20d] systems/paseo.md
|
||||
- [ 12d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 11d] systems/terminal-backend.md
|
||||
- [ 11d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 1 stale, 110 fresh, 3 no-date of 114 pages
|
||||
@@ -0,0 +1,130 @@
|
||||
# Wiki freshness audit — 2026-11-01
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (0)
|
||||
|
||||
|
||||
## FRESH (114)
|
||||
|
||||
- [ 12d] CONTRIBUTING.md
|
||||
- [ 6d] agents/hermes-chaos.md
|
||||
- [ 6d] agents/hermes-production.md
|
||||
- [ 12d] agents/index.md
|
||||
- [ 3d] agents/nanobot.md
|
||||
- [ 12d] agents/open-claw.md
|
||||
- [ 21d] agents/qwenpaw.md
|
||||
- [ 12d] agents/zeroclaw.md
|
||||
- [ 3d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 12d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 12d] comparisons/memory-backends.md
|
||||
- [ 12d] comparisons/model-routers.md
|
||||
- [ 3d] comparisons/steel-vs-camoufox.md
|
||||
- [ 12d] concepts/agent-memory.md
|
||||
- [ 3d] concepts/browser-automation.md
|
||||
- [ 12d] concepts/model-routing.md
|
||||
- [ 12d] concepts/persistent-shells.md
|
||||
- [ 12d] concepts/tool-calling.md
|
||||
- [ 1d] current-state.md
|
||||
- [ 3d] decisions/index.md
|
||||
- [ 12d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 12d] decisions/persistent-shell-enabled.md
|
||||
- [ 3d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 12d] experiments/index.md
|
||||
- [ 8d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 12d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 8d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 10d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 9d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 20d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 7d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 12d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 11d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 21d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 3d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 6d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 6d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 3d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 8d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 0d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 16d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 5d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 13d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 5d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 10d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 7d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 6d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 6d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 6d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 4d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 2d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 2d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 2d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 1d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 2d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 1d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 0d] incidents/2026-10-31-asana-access-failures.md
|
||||
- [ 0d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||
- [ 0d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||
- [ 0d] index.md
|
||||
- [ 10d] infrastructure/backups.md
|
||||
- [ 12d] infrastructure/credentials-map.md
|
||||
- [ 0d] infrastructure/docker-services.md
|
||||
- [ 12d] infrastructure/domains-and-tunnels.md
|
||||
- [ 1d] infrastructure/hosts.md
|
||||
- [ 18d] infrastructure/network-map.md
|
||||
- [ 0d] infrastructure/proxmox.md
|
||||
- [ 0d] infrastructure/tailscale.md
|
||||
- [ 20d] queries/retained-research-answers.md
|
||||
- [ 7d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 11d] reports/uber-eats/2026-07.md
|
||||
- [ 11d] reports/uber-eats/2026-08.md
|
||||
- [ 11d] reports/uber-eats/2026-09.md
|
||||
- [ 11d] reports/uber-eats/index.md
|
||||
- [ 12d] runbooks/backup-wiki.md
|
||||
- [ 12d] runbooks/diagnose-docker.md
|
||||
- [ 12d] runbooks/gateway-resource-resilience.md
|
||||
- [ 2d] runbooks/job-radar.md
|
||||
- [ 12d] runbooks/multiplexer-setup.md
|
||||
- [ 6d] runbooks/provider-health.md
|
||||
- [ 3d] runbooks/recover-docker-service.md
|
||||
- [ 12d] runbooks/restart-browser.md
|
||||
- [ 12d] runbooks/restart-hermes.md
|
||||
- [ 12d] runbooks/restore-openrouter.md
|
||||
- [ 12d] runbooks/test-openrouter.md
|
||||
- [ 12d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 6d] runbooks/update-hermes-safely.md
|
||||
- [ 3d] systems/browser-backend.md
|
||||
- [ 12d] systems/cloudflare-tunnel.md
|
||||
- [ 0d] systems/conduit.md
|
||||
- [ 6d] systems/current-profile.md
|
||||
- [ 12d] systems/guanaco.md
|
||||
- [ 12d] systems/headless-server.md
|
||||
- [ 2d] systems/hermes-agent.md
|
||||
- [ 12d] systems/hermes-ui.md
|
||||
- [ 20d] systems/home-assistant.md
|
||||
- [ 2d] systems/homepage-dashboard.md
|
||||
- [ 21d] systems/ignis.md
|
||||
- [ 6d] systems/memory-backend.md
|
||||
- [ 0d] systems/messaging-integrations.md
|
||||
- [ 0d] systems/mnemosyne.md
|
||||
- [ 6d] systems/model-providers.md
|
||||
- [ 20d] systems/nanobot.md
|
||||
- [ 12d] systems/nextcloud.md
|
||||
- [ 12d] systems/obsidian.md
|
||||
- [ 0d] systems/omniroute.md
|
||||
- [ 12d] systems/opencode-go.md
|
||||
- [ 12d] systems/openrouter.md
|
||||
- [ 21d] systems/paseo.md
|
||||
- [ 13d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 12d] systems/terminal-backend.md
|
||||
- [ 12d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 0 stale, 114 fresh, 3 no-date of 117 pages
|
||||
@@ -3,8 +3,8 @@ title: Messaging Integrations
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
confidence: high
|
||||
tags: [messaging, telegram, discord, email, integrations]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
|
||||
@@ -125,3 +125,30 @@ one of those happens — restarting the adapter while the host is stopped achiev
|
||||
process's** attempts, not the outage's length — do not quote it as downtime.
|
||||
- **Impact unchanged:** messaging path only; no cron job delivers via Mattermost.
|
||||
- **Not attempted:** `pct start 255` — starting a service on the PVE host is Anthony's call.
|
||||
|
||||
## 2026-10-31 06:2x AWST — ✅ Mattermost RECOVERED (outage over)
|
||||
|
||||
**The ~40-day Mattermost outage is over.** All three live checks pass, and the gateway log confirms
|
||||
the adapter stayed connected since.
|
||||
|
||||
| Check | Live result (2026-10-31) |
|
||||
|---|---|
|
||||
| `pct list` on `pve` | **CT255 (`mattermost`) → `running`** (was `stopped`) |
|
||||
| `pct exec 255 -- systemctl is-active mattermost` | **`active`** |
|
||||
| `pct exec 255 -- ss -ltn` | `LISTEN *:8065` |
|
||||
| `tailscale ping 100.98.116.14` | **`pong from mattermost ... in 0s`** |
|
||||
| `curl http://100.98.116.14:8065/api/v4/system/ping` | **HTTP 200** |
|
||||
| Tailscale peer record | **`Online: true`** — first time since the outage began |
|
||||
|
||||
- **Gateway reconnection (CT460's own `gateway.log`):** `Mattermost: authenticated as @rhino-bot` at
|
||||
**2026-10-30 07:19:37**, with `✓ mattermost reconnected successfully` and a successful WebSocket
|
||||
handshake at the same second. A second connect cycle at **2026-10-30 16:36:07** produced
|
||||
`✓ mattermost connected` at **16:36:17**.
|
||||
- **No errors since.** Grepping every `mattermost` line in `gateway.log` after `2026-10-30 16:36:17`
|
||||
returns **zero** `error` / `timeout` / `disconnect` / `fail` lines — the 5-minute reconnect loop that
|
||||
produced 264–268 timeouts a day is **no longer running**.
|
||||
- **The prior "cause: CT255 is stopped" finding is superseded.** CT255 is running; the gateway can
|
||||
reach `http://100.98.116.14:8065/` and authenticate. → [[infrastructure/proxmox]],
|
||||
[[infrastructure/tailscale]]
|
||||
- **Open item now closed:** the platform is no longer an outage. The home channel startup notification
|
||||
was delivered successfully on reconnect (`Sent home-channel startup notification to mattermost:…`).
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Mnemosyne
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-31
|
||||
verified_on: 2026-10-31
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
review_after: 2026-11-07
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, backend]
|
||||
@@ -52,7 +52,9 @@ Primary agent memory store for Hermes.
|
||||
|
||||
**Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below.
|
||||
|
||||
**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. Ranking relevance is the live issue, not coverage — see [[incidents/2026-10-09-memory-recall-relevance]].
|
||||
**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. ~~Ranking relevance is the live issue, not coverage~~ — see [[incidents/2026-10-09-memory-recall-relevance]].
|
||||
|
||||
**Tier counts (live, 2026-11-01 03:0x — the 2026-10-09 ranking incident is now RESOLVED):** working `941`, episodic `1,124`, `canonical_facts` `90`, `conflicts` `175` (**11 open** — ids **165–175**, all `contradiction`, all created in one batch at 2026-10-31 19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature), working embeddings `927/941 (98.5%)`, episodic embeddings `1,124/1,124 (100%)` — both tiers above the 95% floor, no `reindex` needed. **The recall weights are live in `~/.hermes/mnemosyne/config.yaml`** (`vec_weight: 0.5`, `fts_weight: 0.3`, `importance_weight: 0.3`, plus `MNEMOSYNE_IMPORTANCE_WEIGHT=0.3` in the environment) — **not** in `~/.hermes/config.yaml`'s `memory.mnemosyne` block, which is why the 2026-10-10 probe wrongly concluded the fix was never applied. The original regression query now returns only substantive current-state facts; the stale Uber Eats episode no longer outranks them → [[incidents/2026-10-09-memory-recall-relevance]] (**resolved**).
|
||||
|
||||
**Tier counts (live, 2026-10-12 12:02 — health job `ea79a2107e36`, green):** working **760/786 embedded (96.7%)**, episodic **1050/1056 (99.4%)** — both above the 95% floor, no `reindex` needed (`b63ed5817212` continues to backfill). Orphan embeddings **0**. FTS5 exactly aligned: `fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876. **0 unconsolidated** working rows; **1 unresolved conflict** (threshold 10); last consolidation 0.3 h earlier; 2,252 memories total (786 working / 1,056 episodic / 1 triple); disk 15 GB free (81% used). Interpretation: coverage is *not* the live issue — ranking relevance is (see [[incidents/2026-10-09-memory-recall-relevance]]).
|
||||
|
||||
|
||||
+18
-17
@@ -3,8 +3,8 @@ title: OmniRoute
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
confidence: high
|
||||
tags: [routing, provider, model, inference]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -54,30 +54,31 @@ The CT600 provider key `custom:omni` (`http://omniroute:20128/v1`) remains auxil
|
||||
→ [[systems/current-profile]]
|
||||
|
||||
## Current Configuration
|
||||
- **Hermes provider name:** `custom:omni` (maps to provider `omniroute:20128`)
|
||||
- **Base URL:** `http://omniroute:20128/v1`
|
||||
- **Hermes default provider:** `custom:omni-101` → base URL `http://100.78.63.20:20128` (CT101, `m93p`), model default **`smart-route`** — **re-verified live 2026-10-31** from `~/.hermes/config.yaml` (`model.default: smart-route`, `model.provider: custom:omni-101`, `model.base_url: http://100.78.63.20:20128`)
|
||||
- **Secondary provider entry:** `custom:Omniroute:20128` → base URL `http://omniroute:20128/v1` (CT600, `pve`), referenced by the `moa` reference-model set (`vertex/gemini-3.6-flash`, `kilo-gateway/kilo-auto/free`, `nvidia/deepseek-ai/deepseek-v4-pro`, `freeapi/nemotron-3-ultra-550b`) — auxiliary/MoA use, **not** the main chat default
|
||||
- **Host reachability:** bare name `omniroute` resolves to `100.88.81.19`; direct HTTP on `:20128` answers `401` (auth-required, service live)
|
||||
- **Runs on:** Proxmox CT600 (`omniroute`, Tailscale `100.88.81.19`, onboot=1) — verified running 2026-10-11
|
||||
- **Runs on:** two live nodes — CT600 (`omniroute`, Tailscale `100.88.81.19`, on `pve`) and CT101 (`omniroute`, Tailscale `100.78.63.20`, on `m93p`) — **both verified running and listening on `0.0.0.0:20128` on 2026-10-31**
|
||||
- **DB path:** `/opt/omniroute/storage.sqlite` (was `/var/lib/omniroute/storage.sqlite`)
|
||||
- **Backup:** pre-migration config saved at `~/.hermes/config.yaml.bak.omniroute-removal`
|
||||
- **Correction (2026-10-26):** the earlier *"Hermes default provider path does not use OmniRoute for main chat"* note is **stale** — the default is OmniRoute again (`custom:omni-101`, model `smart-route`). The **CT600** key `custom:omni` is still auxiliary-only.
|
||||
- **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp`
|
||||
- **Correction (2026-10-26):** the earlier *"Hermes default provider path does not use OmniRoute for main chat"* note is **stale** — the default is OmniRoute again (`custom:omni-101`, model `smart-route`). The **CT600** key `custom:Omniroute:20128` is still auxiliary-only (MoA reference models).
|
||||
- **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp` — all on `custom:omni-101` with model `no-think/freeapi/auto` (live-verified 2026-10-31)
|
||||
- **Key prefix visible in config:** `sk-5c67fbdc0...` — full value not stored in wiki
|
||||
|
||||
## CT600 / CT601 / omniroute-1 — actual state (2026-10-11)
|
||||
## CT600 / CT601 / omniroute-1 — actual state (re-verified 2026-10-31)
|
||||
|
||||
The wiki previously recorded CT600 as retired and CT601 as the live successor. **That is no longer true.** Live PVE and Tailscale checks return:
|
||||
The wiki previously recorded CT600 as retired and CT601 as the live successor, then CT601 as stopped.
|
||||
**All three framings are now superseded.** Live `pct list` on **both** hypervisors and live HTTP probes return:
|
||||
|
||||
| Node | Status (live 2026-10-11) | Tailscale IP | Notes |
|
||||
| Node | Status (live 2026-10-31) | Tailscale IP | Notes |
|
||||
|---|---|---|---|
|
||||
| CT600 `omniroute` | **running**, onboot=1 | `100.88.81.19` | OmniRoute listening on `0.0.0.0:20128`; `/v1/models` returns `401` (alive, auth required) |
|
||||
| CT601 `omni` | **stopped**, onboot=0 | `100.93.204.4` | Tailscale peer last seen 54d ago |
|
||||
| `omniroute-1` | **active** (not in PVE `pct list`) | `100.78.63.20` | Third OmniRoute node; `/v1/models` returns `401`. Registered in Hermes config as custom provider `omni-101`, which carries `skills_hub` (model `free-stack`, 5384 models, default `smart-route`) |
|
||||
| CT600 `omniroute` | **running** | `100.88.81.19` | On `pve`. OmniRoute listening on `0.0.0.0:20128`; `/v1/models` and `/api/combos` both return `401` (alive, auth required) |
|
||||
| CT101 `omniroute-1` | **running** | `100.78.63.20` | On `m93p`. Live `pct config 101` → `hostname: omniroute`; `omniroute` listening on `0.0.0.0:20128`; `/v1/models` and `/api/combos` both return `401`. Registered in Hermes config as `custom:omni-101` — **the live default provider** |
|
||||
| CT601 `omni` | **does not exist** | — | `/etc/pve/lxc/601.conf` absent on **both** `pve` and `m93p`. The `omni` peer (`100.93.204.4`) has **expired out of the tailnet entirely** — live `tailscale status --json` has no peer named `omni` and no peer at that IP |
|
||||
|
||||
- Hermes config resolving to the omniroute hostname (`custom:omni` → `omniroute:20128`) therefore points at **CT600**, not CT601.
|
||||
- CT600's combo API is reachable with its local key and returns combos (e.g. `go-pro`), confirming it is the serving instance.
|
||||
- **Do not treat CT601 as the production OmniRoute node.** It is stopped with `onboot=0`.
|
||||
- Treat `omniroute-1` (`100.78.63.20`) as a second live instance used by the `omni-101` provider. It is not visible in `pct list`, so it is likely a VM or off-PVE host; its identity was not confirmed.
|
||||
- Hermes config resolving to the omniroute hostname (`custom:Omniroute:20128` → `omniroute:20128`) therefore points at **CT600**; the live default `custom:omni-101` points at **CT101**.
|
||||
- **CT601 was deleted, not merely stopped** — the earlier *"stopped, onboot=0"* entry was wrong.
|
||||
- **The `omniroute-1` identity is now confirmed** — it is **CT101 on `m93p`**, not "a VM or off-PVE host". The 2026-10-11 note that it was "not visible in `pct list`" was an artifact of probing only `pve`.
|
||||
- **Two live OmniRoute nodes remain** (CT600, CT101), both serving `:20128` with separate credentials — a pin's usability must be probed on the node that actually serves it.
|
||||
|
||||
|
||||
## Catalog and Routing Facts
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
confidence: high
|
||||
tags: [skills, hermes]
|
||||
sources: []
|
||||
@@ -12,9 +12,9 @@ sources: []
|
||||
|
||||
# Installed Skills
|
||||
|
||||
**Total installed: 302 skills** in `~/.hermes/skills/`.
|
||||
**Total installed: 301 skills** in `~/.hermes/skills/`.
|
||||
|
||||
- **45 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
|
||||
- **46 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
|
||||
- The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory.
|
||||
|
||||
This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
@@ -308,10 +308,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
|
||||
- `hermes-mcp-troubleshooting` — # Hermes MCP Troubleshooting
|
||||
|
||||
## Hermes Runtime Worker Diagnostics (1)
|
||||
|
||||
- `hermes-runtime-worker-diagnostics` — # Hermes Runtime / Cron Worker Diagnostics
|
||||
|
||||
## Hermes Themes (1)
|
||||
|
||||
- `hermes-themes` — # Hermes Themes Skill
|
||||
|
||||
Reference in New Issue
Block a user