Files
agent-estate-wiki/infrastructure/proxmox.md
T
Tony0410 a0b3734c95 stocktake(2026-11-01): fix the Quartz build (YAML title), close the "unfixed memory ranking" incident, and correct live drift in tailscale/docker/proxmox
BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic.
Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an
unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a
mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)".
A scan of all 117 pages found exactly one such line; it is now properly quoted.
Quartz now builds cleanly: 690 files emitted and deployed.

RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page).
The page claimed the recall-ranking fix "was not present in live config" and that
conflict 152 was "the only open conflict". Both claims were false:
- The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5,
  fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 —
  not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the
  2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks.
- The original regression test now passes: "mnemosyne recall 'Are you up to date'"
  returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale
  Uber Eats episode no longer outranks anything.
- Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise.
All three open items closed; page moved to status: resolved.

LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01):
- infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping
  times out); m93p peer record reads false for the 4th consecutive time while the
  host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages
  refreshed; no IP changes, no missing peers.
- infrastructure/docker-services.md: CT201's Tailscale IP was wrong
  (100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale
  worldmonitor-seeder-fast entry removed from the CT200 list, which is now
  regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to
  thirsty_payne.
- infrastructure/proxmox.md: storage percentages refreshed on both hosts from live
  pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an
  elevated load average recorded.
- systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md:
  re-probed and verified_on bumped.
- systems/skills-index.md: regenerated — 301 unique skills in 68 categories,
  46 archived. (301 is the unique (category, skill_name) count; find returns 302
  files due to the known nested obsidian-git-sync alias duplicate.)

Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
2026-11-01 04:48:30 +08:00

214 lines
14 KiB
Markdown

---
title: Proxmox VE
type: infrastructure
status: active
created: 2026-07-26
updated: 2026-11-01
verified_on: 2026-11-01
confidence: high
tags: [proxmox, virtualization, infrastructure, homelab]
---
# Proxmox VE
## Purpose
Hypervisor hosting all VMs and LXC containers for the homelab.
## Details
- **Host:** `pve.kangaroo-eel.ts.net` (Tailscale: `100.96.100.82`)
- **Node name:** `pve`
- **Version:** Proxmox VE **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, running kernel `7.0.14-5-pve`) — **corrected 2026-10-31**; this page previously said *"Proxmox VE 8.x"*, which was wrong. `m93p` runs the **same** `pve-manager 9.2.10` build, so the two hosts are no longer version-skewed (see the `m93p` section).
- **Primary user:** `Anthony@pam` (full admin)
- **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve`
## Live LXC Inventory (verified 2026-10-31 via `pct list`)
Re-verified 2026-11-01 03:5x AWST (stocktake): **`pct list` on `pve` and `qm list` on `pve` both still match the tables below exactly** — 17 LXCs and 4 VMs on `pve`, with CT255 `mattermost` still `running`, CT209 `cloudreve` and CT555 `elementsynapse` still `stopped`, and **no CT601 on either hypervisor** (`/etc/pve/lxc/601.conf` absent). **`m93p` re-confirmed**: `pve-manager/9.2.10` (same build as `pve`), `qm list` returns **VM 103 `cachyos-m93p` stopped** and **VM 106 `openmausbot-vm` running**, and `pct exec 1001 -- docker ps` returns `sshwifty` (CT1001). Storage percentages were refreshed against live `pvesm status` on both hosts (see the Storage section). **Note:** `m93p` SSH over the tailnet timed out on the first two attempts and succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout).
> **Two Proxmox hosts exist — always qualify a guest with its host.** `pvecm nodes` is empty on
> both, so they are independent and unclustered. Guest IDs also collide — **CT221 (gitea) and
> CT245 (openweb) carry the same VMID on both hosts** — so an unqualified "CT221" is ambiguous.
> `m93p` (`100.104.51.98`, PVE 9.2.10) was **discovered 2026-10-14** and is documented in its own
> section below. It hosts several services the wiki had placed on `pve`.
| VMID | Name | Status | Purpose |
|---|---|---|---|
| 103 | cliproxyapi | running | Cliproxy API |
| 200 | docker | running | Main Docker host — 60+ containers (CT460's docker.sock is NOT accessible to the agent user; use CT200 for fleet inspection) |
| 201 | runtipi | running | Runtipi — Homepage, ntfy, etc. |
| 205 | ollama | running | Ollama LLM inference |
| 207 | webserver | running | Web server |
| 209 | cloudreve | **stopped** | Cloudreve (deprecated) |
| 215 | n8n | running | N8n automation |
| 255 | mattermost | **running** | Mattermost — **back online 2026-10-30/31** after a ~40-day outage; `systemctl is-active mattermost` → `active`, listening on `:8065`, gateway reconnected 2026-10-30 16:36:17 |
| 270 | nextcloud | running | Nextcloud (apache2 active) |
| 299 | sftpgo | running | SFTPGo file server |
| 333 | clawtest | running | ClawTest — Nanobot, Zeroclaw |
| 450 | hermesagent | running | Hermes Agent CT450 (Chaos Lab) |
| 460 | hermes | running | Hermes CT460 (production) |
| 465 | qwenpaw | running | QwenPaw + Paseo daemon (active) |
| 475 | agentzero | running | AgentZero (`100.106.54.94`) — **not previously recorded** |
| 555 | elementsynapse | **stopped** | Element Synapse (deprecated) — also retired from Tailscale (offline 102d) |
| 600 | omniroute | **running** | OmniRoute — live server on `:20128` (see [[systems/omniroute]]) |
**CT601 (`omni`) no longer exists.** The 2026-10-14 inventory listed it as `**stopped**`; live
`ls /etc/pve/lxc/601.conf` on **both** hypervisors returns **no such file**, and the orphaned `omni`
Tailscale peer (`100.93.204.4`) has **expired out of the tailnet entirely** — the live
`tailscale status --json` contains no peer named `omni` and no peer at that IP. The guest was
**deleted**, not merely halted. The only OmniRoute nodes left are **CT600** (`omniroute`,
`100.88.81.19`) and **CT101** (`omniroute-1`, `100.78.63.20`, on `m93p`) — both verified running with
`omniroute` listening on `0.0.0.0:20128`. → [[systems/omniroute]], [[infrastructure/hosts]]
### ⚠ Four entries moved off this host (corrected 2026-10-14)
The 2026-10-11 inventory listed **CT221 (gitea), CT241 (ntfy), CT245 (openweb) and CT260
(freshrss)** as guests of `pve`. They are **absent from live `pct list`** on `pve` — yet all four
services are still up, because they live on `m93p`. `/etc/pve/lxc/` on `pve` holds only
`260.conf.bak` (no active config) and no vzdump task log references CT 221/241/245/260, so these
were never deleted here: **they were never on this host.** The 2026-10-11 audit carried the older
error forward.
| Wiki entry (removed) | Real host | Real VMID |
|---|---|---|
| ~~CT221 gitea~~ | `m93p` | **CT221** — same ID, different host (coincidence) |
| ~~CT241 ntfy~~ | `m93p` | **CT1241** |
| ~~CT245 openweb~~ | `m93p` | **CT245** — same ID, different host |
| ~~CT260 freshrss~~ | `m93p` | **CT1260** |
## Live QEMU Inventory (verified 2026-10-31 via `qm list`)
| VMID | Name | Status | Notes |
|---|---|---|---|
| 100 | homeassistant | running | Home Assistant (**5100 MB** RAM — the page previously said 13 GB) |
| 101 | guest-vps | running | Guest VPS (6144 MB RAM, 48 GB disk) — **was missing from this table entirely** |
| 110 | omarchy | running | Omarchy (13000 MB RAM, 106 GB disk) |
| 282 | ubuntu-server-xfce | **running** | Ubuntu desktop (5000 MB RAM, 133 GB disk) — recorded **stopped** until 2026-10-14; live `qm list` shows running |
| 310 | cachyos | **absent** | CachyOS — **no longer present in `qm list`** on `pve`; the only CachyOS guest is `cachyos-m93p` (VM 103) on `m93p`, which is stopped |
| 403 | open-claw | — | OpenClaw agent; **not present in `qm list`** — it runs elsewhere (Open-Claw VM, see [[infrastructure/hosts]]) |
| 401 | hermes-old | — | Legacy Hermes VM; **not present in `qm list`** — appears decommissioned/removed |
**Not in either list:** VM 403 (`open-claw`) and VM 401 (`hermes-old`) are referenced by the wiki but no longer appear in `qm list`. They have either been migrated off this host or deleted. Status unconfirmed — verify before relying on either entry.
**Re-verified 2026-10-31:** live `qm list` on `pve` returns exactly **4 VMs — 100 homeassistant, 101
guest-vps, 110 omarchy, 282 ubuntu-server-xfce, all running**. Two corrections to the 2026-10-14 table:
**VM 101 (`guest-vps`) was missing** despite running, and **VM 310 (`cachyos`) no longer exists** on this
host (it had been recorded as `stopped`; the live `qm list` has no 310 at all). Home Assistant's memory was
also corrected from *"13GB"* to the live **5100 MB**.
## Host `m93p` (100.104.51.98) — second Proxmox hypervisor, discovered 2026-10-14
Identified because a tailnet-wide scan for `:8006` returned only `m93p` and `pve`. It had been
sitting in the Tailscale node list as an unremarkable `online` peer, but no wiki page recorded it
as a hypervisor.
- **Hostname:** `m93p` · **Tailscale:** `100.104.51.98`
- **PVE version:** **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, kernel `7.0.14-12-pve`) — **corrected 2026-10-31**. The previous entry read *"9.2.0"* and claimed `m93p` was *"materially newer than pve's 8.x"*. Both hosts now run the **identical `pve-manager 9.2.10` build**; only the kernel differs (`7.0.14-12-pve` vs `7.0.14-5-pve`). The hosts are **version-identical, not version-skewed**.
- **Uptime:** **up 1 day, 12:02** at 2026-11-01 03:56 AWST (live `uptime`) — the host **rebooted ~2026-10-30 16:0x**, consistent with the CT1264 openmausbot outage below. Load average was **7.66 / 12.80 / 11.32** (1/5/15 min) at that moment — elevated, worth watching
- **Not clustered with `pve`:** `pvecm nodes` empty, no `/etc/pve/corosync.conf`
### LXC Inventory (verified 2026-10-31 via `pct list` on m93p)
| VMID | Name | Status | Notes |
|---|---|---|---|
| 100 | obsidian-livesync | running | Obsidian LiveSync — ID does not collide; `pve` has no CT100 |
| 101 | omniroute | running | **Second OmniRoute node**, `100.78.63.20` (`omniroute-1` Tailscale peer), `omniroute` listening on `:20128` — identity/role **now confirmed**, see [[systems/omniroute]] |
| 102 | vaultwarden | running | Vaultwarden password manager |
| 104 | rclone | running | Rclone |
| 105 | conduit | running | **Conduit** — iPhone calendar / health / location / contacts bridge (see [[systems/messaging-integrations]]). **Added 2026-10-31** — this guest was missing from the 2026-10-14 table despite running |
| 221 | gitea | running | **This is the host behind the wiki's `gitea` node — the Gitea remote lives here, not on `pve`** |
| 245 | openweb | running | Open WebUI — `pve` has no live CT245; the guest this page previously recorded on `pve` is this one |
| 500 | litellm | running | LiteLLM proxy |
| 998 | nightbridge | **stopped** | Nightbridge |
| 1001 | docker-01 | running | Secondary Docker host |
| 1006 | freellmapi | running | FreeLLM API |
| 1007 | homebridge | running | Homebridge |
| 1008 | rustdesk | running | RustDesk remote access |
| 1241 | ntfy | running | ntfy push notifications — wiki's `CT241` |
| 1260 | freshrss | running | FreshRSS feed reader — wiki's `CT260` |
| 1261 | paseo | running | Paseo coding-agent daemon |
| 1262 | mailflow | running | MailFlow |
| 1263 | social-dashboard | running | Social dashboard |
| 1264 | openmausbot | **stopped** | OpenMausBot — **stopped**, which is why the `openmausbot` Tailscale peer is offline 9d. **Added 2026-10-31** — was missing from the 2026-10-14 table. The companion **VM106 `openmausbot-vm` is running** and holds the separate `openmausbot-vm` peer (`100.77.107.30`) |
**Re-verified 2026-10-31:** live `pct list` on `m93p` returns **19 LXCs — 18 running, 1 stopped
(CT998 nightbridge) plus CT1264 openmausbot stopped**. The 2026-10-14 table listed only **17**, omitting
two live guests: **CT105 (`conduit`)** and **CT1264 (`openmausbot`)**. Note that **VMIDs 103, 220 and 555
do not exist on `m93p`** (`/etc/pve/lxc/{103,220,555}.conf` all absent) — CT103 (cliproxyapi),
CT220 (caddy) and CT555 (elementsynapse) are on **`pve`** only, and `pve`'s CT555 is **stopped** while
`m93p` has no 555 at all. Guest IDs still collide across hosts, so always qualify with the host.
### QEMU Inventory (verified 2026-10-31 via `qm list` on m93p)
| VMID | Name | Status | Notes |
|---|---|---|---|
| 103 | cachyos-m93p | **stopped** | CachyOS (6144 MB RAM, 80 GB disk) |
| 106 | openmausbot-vm | running | OpenMausBot VM (8192 MB RAM, 30 GB disk) — **Added 2026-10-31**; was missing from the 2026-10-14 table. Holds the `openmausbot-vm` Tailscale peer (`100.77.107.30`) |
**Reachability note:** guest service ports on `m93p` were **not** reachable from CT460 during this
pass (they bind to their own interfaces and are not Tailscale-Served). Only hypervisor `:8006` on
`100.104.51.98` is open. Reach them via the host or a Tailscale Serve route before probing services.
Re-confirmed 2026-10-31: ports 3000, 4000, 5432, 8766, 8080 and 9000 were all closed from CT460.
**✅ Follow-up closed 2026-10-31:** `m93p`'s `omniroute` (CT101) is a **second live OmniRoute node**,
not a duplicate of CT600/CT601. Live `pct config 101` on `m93p` confirms `hostname: omniroute`, and
`omniroute` is listening on `0.0.0.0:20128` inside it; the `omniroute-1` Tailscale peer
(`100.78.63.20`) resolves to it. **CT601 no longer exists on either host** — `/etc/pve/lxc/601.conf` is
absent from `pve` and `m93p`, and the orphaned `omni` peer (`100.93.204.4`) has expired out of the
tailnet. → [[systems/omniroute]], [[infrastructure/hosts]]
## Storage (live, 2026-11-01 03:5x AWST — `pvesm status` on both hosts)
### Host `pve` (100.96.100.82)
| Datastore | Type | Status | Total | Used | Used % |
|---|---|---|---|---|---|
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 700 GiB | **76.43 %** (`pvesm`) |
| `SSD1` | lvmthin | active | 932 GiB | 599 GiB | **64.32 %** |
| `fred-qnap` | **nfs** | active | 13.4 TiB | 11.0 TiB | **82.77 %** |
| `local` | dir | active | 67.7 GiB | 54.0 GiB | **79.66 %** |
| `local-lvm` | lvmthin | active | 141 GiB | 19.4 GiB | **13.75 %** |
**`ssd0` has improved further: 88.17 % (2026-10-14) → 76.43 %**, ~169 GiB free. `fred-qnap` is now
**82.45 %** and its type changed from **`cifs` to `nfs`** (13.4 TiB total, 11.0 TiB used) — the share was
remounted over NFS rather than SMB.
**`iMacHDD` no longer appears in `pvesm status` on `pve`** — the 2026-10-14 table listed it as an
inactive `cifs` entry. The iMac shares now live on **`m93p`** as `cifs-anthonyimac` and `cifs-imachdd`
(both active, 60.25 %). Storage was **reassigned from `pve` to `m93p`** rather than the mount failing.
### Host `m93p` (100.104.51.98)
| Datastore | Type | Status | Total | Used | Used % |
|---|---|---|---|---|---|
| `cifs-anthonyimac` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `cifs-imachdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `local` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `local-hdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `nfs-proxmox-31` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** |
| `nfs-proxmox-54` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** |
| `nfs-ssd0-39` | dir | active | 916 GiB | 700 GiB | **76.43 %** |
| `omb-internal` | dir | active | 93.9 GiB | 54.4 GiB | **57.92 %** |
**`m93p` storage was not documented before 2026-10-31.** Note that `m93p` exposes the **same backing
volumes as `pve`** under different names — `nfs-proxmox-31`/`nfs-proxmox-54` mirror `pve`'s `fred-qnap`
(both 82.45 %), and `nfs-ssd0-39` mirrors `pve`'s `ssd0` (both 76.43 %). The iMac shares that were
`iMacHDD` on `pve` are `cifs-anthonyimac` / `cifs-imachdd` on `m93p`.
**Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`).
**Use the `pvesm` number when comparing across hosts** — it is the only one that means the same thing on
both hypervisors.
## API Tokens
- `homepage@pve!homepage` — PVEAuditor role, privsep=0 (used by Homepage dashboard)
## Key Paths
- `/etc/pve/` — PVE config directory
- `/var/lib/vz/` — Local storage
- `/etc/pve/priv/token/` — API token storage
## Related
- [[systems/homepage-dashboard]] — Dashboard shows PVE stats
- [[infrastructure/hosts]] — Full host inventory
- [[infrastructure/network-map]] — Network topology