stocktake(2026-11-01): fix the Quartz build (YAML title), close the "unfixed memory ranking" incident, and correct live drift in tailscale/docker/proxmox

BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic.
Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an
unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a
mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)".
A scan of all 117 pages found exactly one such line; it is now properly quoted.
Quartz now builds cleanly: 690 files emitted and deployed.

RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page).
The page claimed the recall-ranking fix "was not present in live config" and that
conflict 152 was "the only open conflict". Both claims were false:
- The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5,
  fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 —
  not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the
  2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks.
- The original regression test now passes: "mnemosyne recall 'Are you up to date'"
  returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale
  Uber Eats episode no longer outranks anything.
- Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise.
All three open items closed; page moved to status: resolved.

LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01):
- infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping
  times out); m93p peer record reads false for the 4th consecutive time while the
  host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages
  refreshed; no IP changes, no missing peers.
- infrastructure/docker-services.md: CT201's Tailscale IP was wrong
  (100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale
  worldmonitor-seeder-fast entry removed from the CT200 list, which is now
  regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to
  thirsty_payne.
- infrastructure/proxmox.md: storage percentages refreshed on both hosts from live
  pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an
  elevated load average recorded.
- systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md:
  re-probed and verified_on bumped.
- systems/skills-index.md: regenerated — 301 unique skills in 68 categories,
  46 archived. (301 is the unique (category, skill_name) count; find returns 302
  files due to the known nested obsidian-git-sync alias duplicate.)

Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
This commit is contained in:
Tony0410
2026-11-01 04:48:30 +08:00
parent c69681dc16
commit a0b3734c95
14 changed files with 596 additions and 111 deletions
+27 -9
View File
@@ -3,8 +3,8 @@ title: Docker Services
type: infrastructure
status: mixed
created: 2026-07-22
updated: 2026-10-29
verified_on: 2026-10-29
updated: 2026-11-01
verified_on: 2026-11-01
confidence: high
tags: [docker, homelab, infrastructure]
sources: [raw/configs/hermes-config-sanitized.txt, live-probe]
@@ -16,18 +16,36 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-probe]
Containerized services supporting Hermes and the homelab.
## Status
Docker runs on **three hosts**, and CT460's socket is not readable by the agent user. This page previously claimed "Docker runtime not available" — that is only true for CT460's socket from the `hermes` user. Real state, verified 2026-10-11:
Docker runs on **four hosts**, and CT460's socket is not readable by the agent user. This page previously claimed "Docker runtime not available" — that is only true for CT460's socket from the `hermes` user. Real state, **re-verified 2026-11-01**:
| Host | Access | Containers running |
|---|---|---|
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **~50** — main Docker host |
| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | 8 (gotify, readlater, kokoro-tts, edge-tts, onecli-postgres, …) — the `steel-browser` container and its image were **removed 2026-10-29** |
| CT460 `hermes` | `sudo docker` — socket is `root:docker srw-rw----`, `hermes` user not in group `docker`; **no passwordless sudo in cron** | `ignis` (confirmed running; inspect requires root) |
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **61** — main Docker host (**verified 2026-10-31**) |
| CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**verified 2026-10-31**) |
| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** — `thirsty_payne`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29** |
| CT1001 `docker-01` (`100.121.12.31`, **on `m93p`**) | `pct exec 1001 -- docker ps` via `m93p` | **1** — `sshwifty`. **Added 2026-10-31** — a fourth Docker host, previously undocumented |
| CT460 `hermes` | `sudo docker` — socket is `root:docker srw-rw----`, `hermes` user not in group `docker`; **no passwordless sudo in cron** | `ignis` — **running** (`Up 35 hours`), verified via `pct exec 460 -- docker ps` from the PVE host |
## CT200 live inventory (verified 2026-10-11, ~50 containers)
`guacamole-guacamole-1`, `telegram-bot-api`, `ytdlp-web-player`, `homarr`, `overtchat-searxng`, `searxng`, `khoj-search-1`, `prowlarr`, `overtchat-app`, `postgres-newsku`, `guacamole-postgres-1`, `jellyfin`, `libre-closet`, `apprise-api`, `tt-rss-web-nginx-1`, `tt-rss-updater-1`, `tt-rss-app-1`, `tt-rss-db-1`, `audiobookshelf`, `qbittorrent`, `open-webui`, `readlater`, `readlater-extraction-worker`, `worldmonitor-redis`, `redis-cache`, `nextstep-db`, `nextstep-scheduler-1`, `nextstep-app`, `portainer`, `flaresolverr`, `sshwifty`, `8mblocal`, `save-restricted-bot`, `kokoro-tts`, `edge-tts`, `filebrowser`, `filebrowser1`, `worldmonitor-redis-rest`, `worldmonitor`, `overtchat-redis`, `chromadb`, **`ignis-ignis-1`**, `nc_app_talk_bot_ai`, `excalidraw-frontend`, `nc_app_context_agent`, `portkey-gateway-web-1`, `guacamole-guacd-1`, `lobe-chat`, `mycompose-saas-1`.
## CT200 live inventory (re-verified 2026-11-01, 61 containers)
`8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`.
**Watch this one:** `mycompose-db-1` is in `Restarting (1)` — a crash-looping database. Flagged for follow-up; root cause not investigated on 2026-10-11.
**Re-verified 2026-11-01:** the 2026-10-11 inventory listed **49** containers; the live count is now **61**.
**Corrected 2026-11-01:** the inventory list had three **misspelled** `overtchat-*` entries (written
`overchat-*`) and one **stale** entry, `worldmonitor-seeder-fast`, which is **not running** — live
`docker ps -a --filter name=worldmonitor` shows only `worldmonitor`, `worldmonitor-redis` and
`worldmonitor-redis-rest` (all `Up 35 hours`). The list above is regenerated verbatim from live
`docker ps --format '{{.Names}}'`. **CT201's Tailscale IP was also wrong** — it read
`100.116.116.116`; live `pct exec 201 -- tailscale ip -4` returns **`100.96.244.39`** (which is
also what `[[infrastructure/hosts]]` already recorded). **CT333's first container was renamed** —
`adoring_jemison` → **`thirsty_payne`**.
**12 containers were added** since: `emulatorjs`, `excalidraw-backend`, `khoj-database-1`, `khoj-sandbox-1`,
`khoj-server-1`, `mycompose-db-1`, `onlyoffice-docserver`, `quickdrop`, `react-resume-client-1`,
`secret-santa-app`, `tgvideo-bot`, `watchtower`. **No container listed on the 2026-10-11 page has
disappeared** — the inventory is a strict superset.
**✅ `mycompose-db-1` crash-loop is FIXED.** The 2026-10-11 page flagged it as `Restarting (1)` — a
crash-looping database. Live `docker ps -a --filter name=mycompose` now shows **both** `mycompose-db-1`
and `mycompose-saas-1` as **`Up 35 hours`** (checked 2026-11-01). No longer a watch item.
**Note the duplicate Ignis:** `ignis-ignis-1` on CT200 is a separate deployment from CT460's `ignis`. See [[systems/ignis]].
+90 -37
View File
@@ -3,8 +3,8 @@ title: Proxmox VE
type: infrastructure
status: active
created: 2026-07-26
updated: 2026-10-14
verified_on: 2026-10-14
updated: 2026-11-01
verified_on: 2026-11-01
confidence: high
tags: [proxmox, virtualization, infrastructure, homelab]
---
@@ -17,28 +17,30 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
## Details
- **Host:** `pve.kangaroo-eel.ts.net` (Tailscale: `100.96.100.82`)
- **Node name:** `pve`
- **Version:** Proxmox VE 8.x
- **Version:** Proxmox VE **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, running kernel `7.0.14-5-pve`) — **corrected 2026-10-31**; this page previously said *"Proxmox VE 8.x"*, which was wrong. `m93p` runs the **same** `pve-manager 9.2.10` build, so the two hosts are no longer version-skewed (see the `m93p` section).
- **Primary user:** `Anthony@pam` (full admin)
- **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve`
## Live LXC Inventory (verified 2026-10-14 via `pct list`)
## Live LXC Inventory (verified 2026-10-31 via `pct list`)
Re-verified 2026-11-01 03:5x AWST (stocktake): **`pct list` on `pve` and `qm list` on `pve` both still match the tables below exactly** — 17 LXCs and 4 VMs on `pve`, with CT255 `mattermost` still `running`, CT209 `cloudreve` and CT555 `elementsynapse` still `stopped`, and **no CT601 on either hypervisor** (`/etc/pve/lxc/601.conf` absent). **`m93p` re-confirmed**: `pve-manager/9.2.10` (same build as `pve`), `qm list` returns **VM 103 `cachyos-m93p` stopped** and **VM 106 `openmausbot-vm` running**, and `pct exec 1001 -- docker ps` returns `sshwifty` (CT1001). Storage percentages were refreshed against live `pvesm status` on both hosts (see the Storage section). **Note:** `m93p` SSH over the tailnet timed out on the first two attempts and succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout).
> **Two Proxmox hosts exist — always qualify a guest with its host.** `pvecm nodes` is empty on
> both, so they are independent and unclustered. Guest IDs also collide — **CT221 (gitea) and
> CT245 (openweb) carry the same VMID on both hosts** — so an unqualified "CT221" is ambiguous.
> `m93p` (`100.104.51.98`, PVE 9.2.0) was **discovered 2026-10-14** and is documented in its own
> `m93p` (`100.104.51.98`, PVE 9.2.10) was **discovered 2026-10-14** and is documented in its own
> section below. It hosts several services the wiki had placed on `pve`.
| VMID | Name | Status | Purpose |
|---|---|---|---|
| 103 | cliproxyapi | running | Cliproxy API |
| 200 | docker | running | Main Docker host — 50+ containers (CT460's docker.sock is NOT accessible to the agent user; use CT200 for fleet inspection) |
| 200 | docker | running | Main Docker host — 60+ containers (CT460's docker.sock is NOT accessible to the agent user; use CT200 for fleet inspection) |
| 201 | runtipi | running | Runtipi — Homepage, ntfy, etc. |
| 205 | ollama | running | Ollama LLM inference |
| 207 | webserver | running | Web server |
| 209 | cloudreve | **stopped** | Cloudreve (deprecated) |
| 215 | n8n | running | N8n automation |
| 255 | mattermost | **stopped** | Mattermost — unreachable since ~2026-09-20 (Tailscale peer offline, **29d** as of 2026-10-19) |
| 255 | mattermost | **running** | Mattermost — **back online 2026-10-30/31** after a ~40-day outage; `systemctl is-active mattermost` → `active`, listening on `:8065`, gateway reconnected 2026-10-30 16:36:17 |
| 270 | nextcloud | running | Nextcloud (apache2 active) |
| 299 | sftpgo | running | SFTPGo file server |
| 333 | clawtest | running | ClawTest — Nanobot, Zeroclaw |
@@ -46,9 +48,16 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
| 460 | hermes | running | Hermes CT460 (production) |
| 465 | qwenpaw | running | QwenPaw + Paseo daemon (active) |
| 475 | agentzero | running | AgentZero (`100.106.54.94`) — **not previously recorded** |
| 555 | elementsynapse | **stopped** | Element Synapse (deprecated) — also retired from Tailscale (offline 81d) |
| 555 | elementsynapse | **stopped** | Element Synapse (deprecated) — also retired from Tailscale (offline 102d) |
| 600 | omniroute | **running** | OmniRoute — live server on `:20128` (see [[systems/omniroute]]) |
| 601 | omni | **stopped** | Retired successor to CT600 — stopped, `onboot=0`, offline 54d |
**CT601 (`omni`) no longer exists.** The 2026-10-14 inventory listed it as `**stopped**`; live
`ls /etc/pve/lxc/601.conf` on **both** hypervisors returns **no such file**, and the orphaned `omni`
Tailscale peer (`100.93.204.4`) has **expired out of the tailnet entirely** — the live
`tailscale status --json` contains no peer named `omni` and no peer at that IP. The guest was
**deleted**, not merely halted. The only OmniRoute nodes left are **CT600** (`omniroute`,
`100.88.81.19`) and **CT101** (`omniroute-1`, `100.78.63.20`, on `m93p`) — both verified running with
`omniroute` listening on `0.0.0.0:20128`. → [[systems/omniroute]], [[infrastructure/hosts]]
### ⚠ Four entries moved off this host (corrected 2026-10-14)
@@ -66,19 +75,26 @@ error forward.
| ~~CT245 openweb~~ | `m93p` | **CT245** — same ID, different host |
| ~~CT260 freshrss~~ | `m93p` | **CT1260** |
## Live QEMU Inventory (verified 2026-10-14 via `qm list`)
## Live QEMU Inventory (verified 2026-10-31 via `qm list`)
| VMID | Name | Status | Notes |
|---|---|---|---|
| 100 | homeassistant | running | Home Assistant (13GB RAM) |
| 110 | omarchy | running | Omarchy (13GB RAM, 106GB disk) — **new, not previously recorded** |
| 282 | ubuntu-server-xfce | **running** | Ubuntu desktop — recorded **stopped** until 2026-10-14; live `qm list` shows running |
| 310 | cachyos | **stopped** | CachyOS |
| 100 | homeassistant | running | Home Assistant (**5100 MB** RAM — the page previously said 13 GB) |
| 101 | guest-vps | running | Guest VPS (6144 MB RAM, 48 GB disk) — **was missing from this table entirely** |
| 110 | omarchy | running | Omarchy (13000 MB RAM, 106 GB disk) |
| 282 | ubuntu-server-xfce | **running** | Ubuntu desktop (5000 MB RAM, 133 GB disk) — recorded **stopped** until 2026-10-14; live `qm list` shows running |
| 310 | cachyos | **absent** | CachyOS — **no longer present in `qm list`** on `pve`; the only CachyOS guest is `cachyos-m93p` (VM 103) on `m93p`, which is stopped |
| 403 | open-claw | — | OpenClaw agent; **not present in `qm list`** — it runs elsewhere (Open-Claw VM, see [[infrastructure/hosts]]) |
| 401 | hermes-old | — | Legacy Hermes VM; **not present in `qm list`** — appears decommissioned/removed |
**Not in either list:** VM 403 (`open-claw`) and VM 401 (`hermes-old`) are referenced by the wiki but no longer appear in `qm list`. They have either been migrated off this host or deleted. Status unconfirmed — verify before relying on either entry.
**Re-verified 2026-10-31:** live `qm list` on `pve` returns exactly **4 VMs — 100 homeassistant, 101
guest-vps, 110 omarchy, 282 ubuntu-server-xfce, all running**. Two corrections to the 2026-10-14 table:
**VM 101 (`guest-vps`) was missing** despite running, and **VM 310 (`cachyos`) no longer exists** on this
host (it had been recorded as `stopped`; the live `qm list` has no 310 at all). Home Assistant's memory was
also corrected from *"13GB"* to the live **5100 MB**.
## Host `m93p` (100.104.51.98) — second Proxmox hypervisor, discovered 2026-10-14
Identified because a tailnet-wide scan for `:8006` returned only `m93p` and `pve`. It had been
@@ -86,17 +102,19 @@ sitting in the Tailscale node list as an unremarkable `online` peer, but no wiki
as a hypervisor.
- **Hostname:** `m93p` · **Tailscale:** `100.104.51.98`
- **PVE version:** **9.2.0** (kernel `7.0.14-12-pve`, `pve-manager 9.2.10`) — materially newer than `pve`'s 8.x
- **PVE version:** **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, kernel `7.0.14-12-pve`) — **corrected 2026-10-31**. The previous entry read *"9.2.0"* and claimed `m93p` was *"materially newer than pve's 8.x"*. Both hosts now run the **identical `pve-manager 9.2.10` build**; only the kernel differs (`7.0.14-12-pve` vs `7.0.14-5-pve`). The hosts are **version-identical, not version-skewed**.
- **Uptime:** **up 1 day, 12:02** at 2026-11-01 03:56 AWST (live `uptime`) — the host **rebooted ~2026-10-30 16:0x**, consistent with the CT1264 openmausbot outage below. Load average was **7.66 / 12.80 / 11.32** (1/5/15 min) at that moment — elevated, worth watching
- **Not clustered with `pve`:** `pvecm nodes` empty, no `/etc/pve/corosync.conf`
### LXC Inventory (verified 2026-10-14 via `pct list` on m93p)
### LXC Inventory (verified 2026-10-31 via `pct list` on m93p)
| VMID | Name | Status | Notes |
|---|---|---|---|
| 100 | obsidian-livesync | running | Obsidian LiveSync — ID does not collide; `pve` has no CT100 |
| 101 | omniroute | running | **An OmniRoute instance distinct from `pve`'s CT600/CT601 — see [[systems/omniroute]]; identity/role unconfirmed** |
| 101 | omniroute | running | **Second OmniRoute node**, `100.78.63.20` (`omniroute-1` Tailscale peer), `omniroute` listening on `:20128` — identity/role **now confirmed**, see [[systems/omniroute]] |
| 102 | vaultwarden | running | Vaultwarden password manager |
| 104 | rclone | running | Rclone |
| 105 | conduit | running | **Conduit** — iPhone calendar / health / location / contacts bridge (see [[systems/messaging-integrations]]). **Added 2026-10-31** — this guest was missing from the 2026-10-14 table despite running |
| 221 | gitea | running | **This is the host behind the wiki's `gitea` node — the Gitea remote lives here, not on `pve`** |
| 245 | openweb | running | Open WebUI — `pve` has no live CT245; the guest this page previously recorded on `pve` is this one |
| 500 | litellm | running | LiteLLM proxy |
@@ -105,47 +123,82 @@ as a hypervisor.
| 1006 | freellmapi | running | FreeLLM API |
| 1007 | homebridge | running | Homebridge |
| 1008 | rustdesk | running | RustDesk remote access |
| 1241 | ntfy | running | Ntfy push notifications — wiki's `CT241` |
| 1241 | ntfy | running | ntfy push notifications — wiki's `CT241` |
| 1260 | freshrss | running | FreshRSS feed reader — wiki's `CT260` |
| 1261 | paseo | running | Paseo coding-agent daemon |
| 1262 | mailflow | running | MailFlow |
| 1263 | social-dashboard | running | Social dashboard |
| 1264 | openmausbot | **stopped** | OpenMausBot — **stopped**, which is why the `openmausbot` Tailscale peer is offline 9d. **Added 2026-10-31** — was missing from the 2026-10-14 table. The companion **VM106 `openmausbot-vm` is running** and holds the separate `openmausbot-vm` peer (`100.77.107.30`) |
### QEMU Inventory (verified 2026-10-14 via `qm list` on m93p)
**Re-verified 2026-10-31:** live `pct list` on `m93p` returns **19 LXCs — 18 running, 1 stopped
(CT998 nightbridge) plus CT1264 openmausbot stopped**. The 2026-10-14 table listed only **17**, omitting
two live guests: **CT105 (`conduit`)** and **CT1264 (`openmausbot`)**. Note that **VMIDs 103, 220 and 555
do not exist on `m93p`** (`/etc/pve/lxc/{103,220,555}.conf` all absent) — CT103 (cliproxyapi),
CT220 (caddy) and CT555 (elementsynapse) are on **`pve`** only, and `pve`'s CT555 is **stopped** while
`m93p` has no 555 at all. Guest IDs still collide across hosts, so always qualify with the host.
### QEMU Inventory (verified 2026-10-31 via `qm list` on m93p)
| VMID | Name | Status | Notes |
|---|---|---|---|
| 103 | cachyos-m93p | **stopped** | CachyOS |
| 103 | cachyos-m93p | **stopped** | CachyOS (6144 MB RAM, 80 GB disk) |
| 106 | openmausbot-vm | running | OpenMausBot VM (8192 MB RAM, 30 GB disk) — **Added 2026-10-31**; was missing from the 2026-10-14 table. Holds the `openmausbot-vm` Tailscale peer (`100.77.107.30`) |
**Reachability note:** guest service ports on `m93p` were **not** reachable from CT460 during this
pass (they bind to their own interfaces and are not Tailscale-Served). Only hypervisor `:8006` on
`100.104.51.98` is open. Reach them via the host or a Tailscale Serve route before probing services.
Re-confirmed 2026-10-31: ports 3000, 4000, 5432, 8766, 8080 and 9000 were all closed from CT460.
**⚠ Follow-up:** `m93p`'s guest list is recorded here from a single `pct list` / `qm list` pass.
The pages [[systems/omniroute]], [[infrastructure/hosts]], [[systems/model-providers]] and
[[infrastructure/docker-services]] still describe single-host topology and should be re-verified
against `m93p` — in particular whether `m93p`'s `omniroute` (CT101) is a fourth OmniRoute node or
a duplicate of CT600/CT601.
**✅ Follow-up closed 2026-10-31:** `m93p`'s `omniroute` (CT101) is a **second live OmniRoute node**,
not a duplicate of CT600/CT601. Live `pct config 101` on `m93p` confirms `hostname: omniroute`, and
`omniroute` is listening on `0.0.0.0:20128` inside it; the `omniroute-1` Tailscale peer
(`100.78.63.20`) resolves to it. **CT601 no longer exists on either host** — `/etc/pve/lxc/601.conf` is
absent from `pve` and `m93p`, and the orphaned `omni` peer (`100.93.204.4`) has expired out of the
tailnet. → [[systems/omniroute]], [[infrastructure/hosts]]
## Storage (live, 2026-10-14 03:0x AWST — `pvesm status` + `df -h /mnt/ssd0`, host `pve`)
## Storage (live, 2026-11-01 03:5x AWST — `pvesm status` on both hosts)
### Host `pve` (100.96.100.82)
| Datastore | Type | Status | Total | Used | Used % |
|---|---|---|---|---|---|
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 808 GiB | **88.17 %** (`pvesm`) |
| `SSD1` | lvmthin | active | 976 GiB | 815 GiB | 87.47 % |
| `fred-qnap` | cifs | active | 13.7 TiB | 10.3 TiB | 77.22 % |
| `local` | dir | active | 67.7 GiB | 55.1 GiB | 81.68 % |
| `local-lvm` | lvmthin | active | 141 GiB | 19.0 GiB | 13.44 % |
| `iMacHDD` | cifs | **inactive** | — | — | — |
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 700 GiB | **76.43 %** (`pvesm`) |
| `SSD1` | lvmthin | active | 932 GiB | 599 GiB | **64.32 %** |
| `fred-qnap` | **nfs** | active | 13.4 TiB | 11.0 TiB | **82.77 %** |
| `local` | dir | active | 67.7 GiB | 54.0 GiB | **79.66 %** |
| `local-lvm` | lvmthin | active | 141 GiB | 19.4 GiB | **13.75 %** |
**`ssd0` improved from the 93 % figure: now 88.17 % by `pvesm`, ~63 GiB free.** The earlier 93 %
came from `df`, which excludes reserved blocks; `pvesm` divides by the full 960 GiB. Both describe
the same ~808 GiB used. **Use the `pvesm` number when comparing across hosts** — it is the only
one that means the same thing on `m93p` too. `iMacHDD` remains inactive
(`storage 'iMacHDD' is not online`) — check the CIFS mount before using it as a target.
**`ssd0` has improved further: 88.17 % (2026-10-14) → 76.43 %**, ~169 GiB free. `fred-qnap` is now
**82.45 %** and its type changed from **`cifs` to `nfs`** (13.4 TiB total, 11.0 TiB used) — the share was
remounted over NFS rather than SMB.
**`iMacHDD` no longer appears in `pvesm status` on `pve`** — the 2026-10-14 table listed it as an
inactive `cifs` entry. The iMac shares now live on **`m93p`** as `cifs-anthonyimac` and `cifs-imachdd`
(both active, 60.25 %). Storage was **reassigned from `pve` to `m93p`** rather than the mount failing.
### Host `m93p` (100.104.51.98)
| Datastore | Type | Status | Total | Used | Used % |
|---|---|---|---|---|---|
| `cifs-anthonyimac` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `cifs-imachdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `local` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `local-hdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `nfs-proxmox-31` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** |
| `nfs-proxmox-54` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** |
| `nfs-ssd0-39` | dir | active | 916 GiB | 700 GiB | **76.43 %** |
| `omb-internal` | dir | active | 93.9 GiB | 54.4 GiB | **57.92 %** |
**`m93p` storage was not documented before 2026-10-31.** Note that `m93p` exposes the **same backing
volumes as `pve`** under different names — `nfs-proxmox-31`/`nfs-proxmox-54` mirror `pve`'s `fred-qnap`
(both 82.45 %), and `nfs-ssd0-39` mirrors `pve`'s `ssd0` (both 76.43 %). The iMac shares that were
`iMacHDD` on `pve` are `cifs-anthonyimac` / `cifs-imachdd` on `m93p`.
**Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`).
**Use the `pvesm` number when comparing across hosts** — it is the only one that means the same thing on
both hypervisors.
## API Tokens
- `homepage@pve!homepage` — PVEAuditor role, privsep=0 (used by Homepage dashboard)
+16 -14
View File
@@ -3,8 +3,8 @@ title: Tailscale
type: infrastructure
status: active
created: 2026-07-26
updated: 2026-10-30
verified_on: 2026-10-30
updated: 2026-11-01
verified_on: 2026-11-01
confidence: high
tags: [tailscale, vpn, networking, infrastructure]
---
@@ -19,8 +19,10 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access
- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`)
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
## Tailscale Node Inventory (verified 2026-10-30)
## Tailscale Node Inventory (verified 2026-10-31)
Full node list from `tailscale status` — **50 non-funnel nodes** (49 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
Re-verified 2026-10-31 05:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **one state transition and seven offline-age drifts, no IP changes and no missing peers.** **`mattermost` came back online**: the peer record reads `Online: true` for the first time since the outage, and the guest behind it is genuinely healthy — live `pct exec 255 -- systemctl is-active mattermost` → **`active`**, `ss -ltnp` shows `mattermost` listening on `*:8065`, `pct exec 255 -- tailscale ip -4` → `100.98.116.14`, and the gateway's own `gateway.log` records **`✓ mattermost connected` at 2026-10-30 16:36:17** (after an earlier successful reconnect at 07:19:37 the same day). CT255 is therefore **running** on `pve` — the 40-day outage is over. The **7 offline ages** were refreshed (cloudreve 97d → **98d**; conduit 53d → **55d**; elementsynapse 100d → **102d**; freshrss-new 73d → **74d**; omarchybook 8d → **9d**; openmausbot 8d → **9d**; ubuntu-server-xfce 70d → **71d**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Two container annotations were added for peers that had none: `openmausbot` (**CT1264, m93p — stopped**) and `openmausbot-vm` (**VM106, m93p**), both confirmed from live `pct list` / `qm list` on `m93p`. The `m93p` row's "peer record stale" note was updated — its peer record read `Online: false` on 10-29 and 10-30 and reads **`Online: true`** today, the third consecutive day the record disagreed with the (reachable) host. CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
Re-verified 2026-11-01 03:0x–04:0x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition and 5 offline-age drifts, no IP changes and no missing peers.** **`omarchymac` went offline**: the peer record reads `Online: false` with `LastSeen 2026-10-31T13:20:00Z` and `tailscale ping 100.126.173.115` times out, so this is a **genuine** transition (the JSON `OS` field reads `linux`, not a Mac). **`m93p`'s peer record reads `Online: false` again** — the **4th consecutive** stale record — while `tailscale ping 100.104.51.98` → `pong from m93p via DERP(syd) in 98ms` and SSH over the tailnet returned `up 1 day, 12:02`, so the row stays **online** and the peer record remains non-authoritative for hypervisor reachability. The **5 offline ages** were refreshed (cloudreve 98d → **99d**; freshrss-new 74d → **75d**; omarchybook 9d → **10d**; openmausbot 9d → **10d**; ubuntu-server-xfce 71d → **72d**). **`conduit` (55d) and `elementsynapse` (102d) were already correct** and are unchanged. **Every peer IP in the table still matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Container identity re-confirmed from inside the guests: `pct exec 201 -- tailscale ip -4` → `100.96.244.39` (CT201, runtipi), `pct exec 1001 -- docker ps` → `sshwifty` (CT1001 on `m93p`), CT209 `cloudreve` and CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT255 `mattermost` **running** with `systemctl is-active mattermost` → `active` and `LISTEN *:8065`. Both OmniRoute nodes re-probed: `100.88.81.19` (CT600) and `100.78.63.20` (CT101) both return **401** on `/v1/models` (alive, auth required).
Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong).
Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`.
Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**.
@@ -40,22 +42,22 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| anthonys-mac-mini | 100.76.60.37 | online |
| clawtest | 100.122.189.81 | online — CT333 (pve) |
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
| cloudreve | 100.111.187.7 | offline 97d — CT209 stopped |
| conduit | 100.81.79.87 | offline 53d |
| cloudreve | 100.111.187.7 | offline 99d — CT209 stopped |
| conduit | 100.81.79.87 | offline 55d |
| docker | 100.96.150.69 | online — CT200 (pve) |
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
| elementsynapse | 100.75.250.37 | offline 100d — CT555 stopped |
| elementsynapse | 100.75.250.37 | offline 102d — CT555 stopped |
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
| freshrss | 100.83.60.93 | online — CT1260 (m93p) |
| freshrss-new | 100.110.180.115 | offline 73d |
| freshrss-new | 100.110.180.115 | offline 75d |
| gitea | 100.82.100.128 | online — CT221 (m93p) |
| guest-vps | 100.117.14.84 | online |
| hermesagent | 100.98.205.112 | online — CT450 (pve) |
| homeassistant | 100.68.139.5 | online |
| litellm | 100.79.29.64 | online — CT500 (m93p) |
| m93p | 100.104.51.98 | online — hypervisor (peer record stale: `tailscale ping` + SSH verified 2026-10-30) |
| m93p | 100.104.51.98 | online — hypervisor (peer record reads `Online: false` again on 2026-11-01, 4th consecutive stale record; `tailscale ping` → pong) |
| mailflow | 100.115.60.92 | online — CT1262 (m93p) |
| mattermost | 100.98.116.14 | offline 40d — CT255 stopped |
| mattermost | 100.98.116.14 | online — CT255 (pve) |
| n8n | 100.84.124.30 | online |
| nextcloud | 100.90.102.19 | online — CT270 (pve) |
| ntfy | 100.67.153.116 | online — CT1241 (m93p) |
@@ -63,12 +65,12 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| octop | 100.92.163.122 | online |
| ollama | 100.108.44.119 | online — CT205 (pve) |
| omarchy | 100.73.227.103 | online |
| omarchybook | 100.107.9.100 | offline 8d |
| omarchymac | 100.126.173.115 | online |
| omarchybook | 100.107.9.100 | offline 10d |
| omarchymac | 100.126.173.115 | offline <1d — last seen 2026-10-31 13:20Z (state transition 2026-11-01; `tailscale ping` times out) |
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
| openmausbot | 100.91.214.91 | offline 8d |
| openmausbot-vm | 100.77.107.30 | online |
| openmausbot | 100.91.214.91 | offline 10d — CT1264 (m93p) stopped |
| openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) |
| openweb | 100.69.230.13 | online — CT245 (m93p) |
| paseo | 100.72.188.93 | online — CT1261 (m93p) |
| pve | 100.96.100.82 | online |
@@ -80,7 +82,7 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
| springroll | 100.100.153.125 | online |
| springroll-browser | 100.109.221.91 | online |
| ubuntu-server-xfce | 100.101.26.104 | offline 70d |
| ubuntu-server-xfce | 100.101.26.104 | offline 72d |
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
| webserver | 100.82.40.23 | online — CT207 (pve) |