Files
agent-estate-wiki/infrastructure/proxmox.md
T
Tony0410 0b4ff25fd2 resolve: Gitea config backup push dead since 2026-08-08 — 92-commit backlog drained, script hardened to fail loudly
- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo
- Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved)
- Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB
- Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification,
  decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree,
  honest AgentMail/Telegram delivery reporting
- Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed
- Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated
- Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with
  'lost its durable fire claim ownership' and left the tree dirty)
2026-11-07 05:28:39 +08:00

226 lines
17 KiB
Markdown

---
title: Proxmox VE
type: infrastructure
status: active
created: 2026-07-26
updated: 2026-11-07
verified_on: 2026-11-07
confidence: high
tags: [proxmox, virtualization, infrastructure, homelab]
---
# Proxmox VE
## Purpose
Hypervisor hosting all VMs and LXC containers for the homelab.
## Details
- **Host:** `pve.kangaroo-eel.ts.net` (Tailscale: `100.96.100.82`)
- **Node name:** `pve`
- **Version:** Proxmox VE **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, running kernel `7.0.14-5-pve`) — **corrected 2026-10-31**; this page previously said *"Proxmox VE 8.x"*, which was wrong. `m93p` runs the **same** `pve-manager 9.2.10` build, so the two hosts are no longer version-skewed (see the `m93p` section).
- **Primary user:** `Anthony@pam` (full admin)
- **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve`
## Live LXC Inventory (verified 2026-11-07 via `pct list`)
Re-verified 2026-11-07 03:2x AWST (stocktake): **live `pct list` on `pve` still returns exactly the 17 LXCs in the table below** (CT255 `mattermost` `running`, CT209 `cloudreve` and CT555 `elementsynapse` `stopped`, CT600 `omniroute` `running`, and **no CT601 on either hypervisor**), and **`qm list` on `pve` still returns exactly the 4 VMs** (100 `homeassistant`, 101 `guest-vps`, 110 `omarchy`, 282 `ubuntu-server-xfce` — all running; **no 310, no 403, no 401**). Both `pve-manager` packages are still **9.2.10** on **both** hosts. The **one correction** was on the `m93p` side: see the `m93p` section below — **CT107 (`mem0`) was missing**, so the m93p guest count was one short.
Re-verified 2026-11-01 03:5x AWST (stocktake): **`pct list` on `pve` and `qm list` on `pve` both still match the tables below exactly** — 17 LXCs and 4 VMs on `pve`, with CT255 `mattermost` still `running`, CT209 `cloudreve` and CT555 `elementsynapse` still `stopped`, and **no CT601 on either hypervisor** (`/etc/pve/lxc/601.conf` absent). **`m93p` re-confirmed**: `pve-manager/9.2.10` (same build as `pve`), `qm list` returns **VM 103 `cachyos-m93p` stopped** and **VM 106 `openmausbot-vm` running**, and `pct exec 1001 -- docker ps` returns `sshwifty` (CT1001). Storage percentages were refreshed against live `pvesm status` on both hosts (see the Storage section). **Note:** `m93p` SSH over the tailnet timed out on the first two attempts and succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout).
> **Two Proxmox hosts exist — always qualify a guest with its host.** `pvecm nodes` is empty on
> both, so they are independent and unclustered. Guest IDs also collide — **CT221 (gitea) and
> CT245 (openweb) carry the same VMID on both hosts** — so an unqualified "CT221" is ambiguous.
> `m93p` (`100.104.51.98`, PVE 9.2.10) was **discovered 2026-10-14** and is documented in its own
> section below. It hosts several services the wiki had placed on `pve`.
| VMID | Name | Status | Purpose |
|---|---|---|---|
| 103 | cliproxyapi | running | Cliproxy API |
| 200 | docker | running | Main Docker host — 60+ containers (CT460's docker.sock is NOT accessible to the agent user; use CT200 for fleet inspection) |
| 201 | runtipi | running | Runtipi — Homepage, ntfy, etc. |
| 205 | ollama | running | Ollama LLM inference |
| 207 | webserver | running | Web server |
| 209 | cloudreve | **stopped** | Cloudreve (deprecated) |
| 215 | n8n | running | N8n automation |
| 255 | mattermost | **running** | Mattermost — **back online 2026-10-30/31** after a ~40-day outage; `systemctl is-active mattermost` → `active`, listening on `:8065`, gateway reconnected 2026-10-30 16:36:17 |
| 270 | nextcloud | running | Nextcloud (apache2 active) |
| 299 | sftpgo | running | SFTPGo file server |
| 333 | clawtest | running | ClawTest — Nanobot, Zeroclaw |
| 450 | hermesagent | running | Hermes Agent CT450 (Chaos Lab) |
| 460 | hermes | running | Hermes CT460 (production) |
| 465 | qwenpaw | running | QwenPaw + Paseo daemon (active) |
| 475 | agentzero | running | AgentZero (`100.106.54.94`) — **not previously recorded** |
| 555 | elementsynapse | **stopped** | Element Synapse (deprecated) — also retired from Tailscale (offline 102d) |
| 600 | omniroute | **running** | OmniRoute — live server on `:20128` (see [[systems/omniroute]]) |
**CT601 (`omni`) no longer exists.** The 2026-10-14 inventory listed it as `**stopped**`; live
`ls /etc/pve/lxc/601.conf` on **both** hypervisors returns **no such file**, and the orphaned `omni`
Tailscale peer (`100.93.204.4`) has **expired out of the tailnet entirely** — the live
`tailscale status --json` contains no peer named `omni` and no peer at that IP. The guest was
**deleted**, not merely halted. The only OmniRoute nodes left are **CT600** (`omniroute`,
`100.88.81.19`) and **CT101** (`omniroute-1`, `100.78.63.20`, on `m93p`) — both verified running with
`omniroute` listening on `0.0.0.0:20128`. → [[systems/omniroute]], [[infrastructure/hosts]]
### ⚠ Four entries moved off this host (corrected 2026-10-14)
The 2026-10-11 inventory listed **CT221 (gitea), CT241 (ntfy), CT245 (openweb) and CT260
(freshrss)** as guests of `pve`. They are **absent from live `pct list`** on `pve` — yet all four
services are still up, because they live on `m93p`. `/etc/pve/lxc/` on `pve` holds only
`260.conf.bak` (no active config) and no vzdump task log references CT 221/241/245/260, so these
were never deleted here: **they were never on this host.** The 2026-10-11 audit carried the older
error forward.
| Wiki entry (removed) | Real host | Real VMID |
|---|---|---|
| ~~CT221 gitea~~ | `m93p` | **CT221** — same ID, different host (coincidence) |
| ~~CT241 ntfy~~ | `m93p` | **CT1241** |
| ~~CT245 openweb~~ | `m93p` | **CT245** — same ID, different host |
| ~~CT260 freshrss~~ | `m93p` | **CT1260** |
## Live QEMU Inventory (verified 2026-10-31 via `qm list`)
| VMID | Name | Status | Notes |
|---|---|---|---|
| 100 | homeassistant | running | Home Assistant (**5100 MB** RAM — the page previously said 13 GB) |
| 101 | guest-vps | running | Guest VPS (6144 MB RAM, 48 GB disk) — **was missing from this table entirely** |
| 110 | omarchy | running | Omarchy (13000 MB RAM, 106 GB disk) |
| 282 | ubuntu-server-xfce | **running** | Ubuntu desktop (5000 MB RAM, 133 GB disk) — recorded **stopped** until 2026-10-14; live `qm list` shows running |
| 310 | cachyos | **absent** | CachyOS — **no longer present in `qm list`** on `pve`; the only CachyOS guest is `cachyos-m93p` (VM 103) on `m93p`, which is stopped |
| 403 | open-claw | — | OpenClaw agent; **not present in `qm list`** — it runs elsewhere (Open-Claw VM, see [[infrastructure/hosts]]) |
| 401 | hermes-old | — | Legacy Hermes VM; **not present in `qm list`** — appears decommissioned/removed |
**Not in either list:** VM 403 (`open-claw`) and VM 401 (`hermes-old`) are referenced by the wiki but no longer appear in `qm list`. They have either been migrated off this host or deleted. Status unconfirmed — verify before relying on either entry.
**Re-verified 2026-10-31:** live `qm list` on `pve` returns exactly **4 VMs — 100 homeassistant, 101
guest-vps, 110 omarchy, 282 ubuntu-server-xfce, all running**. Two corrections to the 2026-10-14 table:
**VM 101 (`guest-vps`) was missing** despite running, and **VM 310 (`cachyos`) no longer exists** on this
host (it had been recorded as `stopped`; the live `qm list` has no 310 at all). Home Assistant's memory was
also corrected from *"13GB"* to the live **5100 MB**.
## Host `m93p` (100.104.51.98) — second Proxmox hypervisor, discovered 2026-10-14
Identified because a tailnet-wide scan for `:8006` returned only `m93p` and `pve`. It had been
sitting in the Tailscale node list as an unremarkable `online` peer, but no wiki page recorded it
as a hypervisor.
- **Hostname:** `m93p` · **Tailscale:** `100.104.51.98`
- **PVE version:** **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, kernel `7.0.14-12-pve`) — **corrected 2026-10-31**. The previous entry read *"9.2.0"* and claimed `m93p` was *"materially newer than pve's 8.x"*. Both hosts now run the **identical `pve-manager 9.2.10` build**; only the kernel differs (`7.0.14-12-pve` vs `7.0.14-5-pve`). The hosts are **version-identical, not version-skewed**.
- **Uptime:** **up 1 day, 7:18** at 2026-11-07 04:38 AWST (live `uptime -s` → **boot 2026-11-05 21:19:52**; `uptime` → `up 1 day, 7:18`, load average **0.76 / 1.08 / 1.24** — normal). ⚠ **Corrected 2026-11-07:** the previous entry read *"up 1 day, 12:02 at 2026-11-01 03:56 AWST — the host rebooted ~2026-10-30 16:0x"*. The **reboot itself is still real** (`pve` booted **2026-10-30 16:24:00** per `uptime -s`, and the CT1264 openmausbot outage below follows from it), but the **uptime figure and the implied boot date were wrong** — the host has since rebooted again on **2026-11-05 21:19:52**, so it was up only **1 day, 7:18** when probed, not 12 h. The `pve` host's own boot time is **2026-10-30 16:24:00** (up **7 days, 12:14** at probe time)
- **Not clustered with `pve`:** `pvecm nodes` empty, no `/etc/pve/corosync.conf`
### LXC Inventory (verified 2026-10-31 via `pct list` on m93p)
| VMID | Name | Status | Notes |
|---|---|---|---|
| 100 | obsidian-livesync | running | Obsidian LiveSync — ID does not collide; `pve` has no CT100 |
| 101 | omniroute | running | **Second OmniRoute node**, `100.78.63.20` (`omniroute-1` Tailscale peer), `omniroute` listening on `:20128` — identity/role **now confirmed**, see [[systems/omniroute]] |
| 102 | vaultwarden | running | Vaultwarden password manager |
| 104 | rclone | running | Rclone |
| 105 | conduit | running | **Conduit** — iPhone calendar / health / location / contacts bridge (see [[systems/messaging-integrations]]). **Added 2026-10-31** — this guest was missing from the 2026-10-14 table despite running |
| 107 | mem0 | running | **mem0 MCP server** (`mem0-mcp.service` → `:8890`), Tailscale `100.73.94.53` (`mem0` peer) — **added 2026-11-07**, this guest was missing from the table despite running |
| 221 | gitea | running | **This is the host behind the wiki's `gitea` node — the Gitea remote lives here, not on `pve`** |
| 245 | openweb | running | Open WebUI — `pve` has no live CT245; the guest this page previously recorded on `pve` is this one |
| 500 | litellm | running | LiteLLM proxy |
| 998 | nightbridge | **stopped** | Nightbridge |
| 1001 | docker-01 | running | Secondary Docker host |
| 1006 | freellmapi | running | FreeLLM API |
| 1007 | homebridge | running | Homebridge |
| 1008 | rustdesk | running | RustDesk remote access |
| 1241 | ntfy | running | ntfy push notifications — wiki's `CT241` |
| 1260 | freshrss | running | FreshRSS feed reader — wiki's `CT260` |
| 1261 | paseo | running | Paseo coding-agent daemon |
| 1262 | mailflow | running | MailFlow |
| 1263 | social-dashboard | running | Social dashboard |
| 1264 | openmausbot | **stopped** | OpenMausBot — **stopped**, which is why the `openmausbot` Tailscale peer is offline 9d. **Added 2026-10-31** — was missing from the 2026-10-14 table. The companion **VM106 `openmausbot-vm` is running** and holds the separate `openmausbot-vm` peer (`100.77.107.30`) |
**Re-verified 2026-11-07 03:2x AWST (stocktake):** live `pct list` on `m93p` returns **20 LXCs — 18 running,
2 stopped (CT998 `nightbridge` and CT1264 `openmausbot`)**. ⚠ **The count was one short**: the previous
figure of 19 omitted **CT107 (`mem0`)**, a live running guest confirmed from inside the container
(`pct exec 107 -- tailscale ip -4` → `100.73.94.53`, `systemctl is-active tailscaled` → `active`,
`mem0-mcp.service` → `active`). CT107 is now in the table above.
**Re-verified 2026-10-31:** the earlier reading of **19 LXCs — 18 running, 1 stopped
(CT998 nightbridge) plus CT1264 openmausbot stopped** was itself already one short for the same reason
(CT107 `mem0` was not counted). The 2026-10-14 table listed only **17**, omitting
two live guests: **CT105 (`conduit`)** and **CT1264 (`openmausbot`)**. Note that **VMIDs 103, 220 and 555
do not exist on `m93p`** (`/etc/pve/lxc/{103,220,555}.conf` all absent) — CT103 (cliproxyapi),
CT220 (caddy) and CT555 (elementsynapse) are on **`pve`** only, and `pve`'s CT555 is **stopped** while
`m93p` has no 555 at all. Guest IDs still collide across hosts, so always qualify with the host.
### QEMU Inventory (verified 2026-10-31 via `qm list` on m93p)
| VMID | Name | Status | Notes |
|---|---|---|---|
| 103 | cachyos-m93p | **stopped** | CachyOS (6144 MB RAM, 80 GB disk) |
| 106 | openmausbot-vm | running | OpenMausBot VM (8192 MB RAM, 30 GB disk) — **Added 2026-10-31**; was missing from the 2026-10-14 table. Holds the `openmausbot-vm` Tailscale peer (`100.77.107.30`) |
**Reachability note:** guest service ports on `m93p` were **not** reachable from CT460 during this
pass (they bind to their own interfaces and are not Tailscale-Served). Only hypervisor `:8006` on
`100.104.51.98` is open. Reach them via the host or a Tailscale Serve route before probing services.
Re-confirmed 2026-10-31: ports 3000, 4000, 5432, 8766, 8080 and 9000 were all closed from CT460.
**✅ Follow-up closed 2026-10-31:** `m93p`'s `omniroute` (CT101) is a **second live OmniRoute node**,
not a duplicate of CT600/CT601. Live `pct config 101` on `m93p` confirms `hostname: omniroute`, and
`omniroute` is listening on `0.0.0.0:20128` inside it; the `omniroute-1` Tailscale peer
(`100.78.63.20`) resolves to it. **CT601 no longer exists on either host** — `/etc/pve/lxc/601.conf` is
absent from `pve` and `m93p`, and the orphaned `omni` peer (`100.93.204.4`) has expired out of the
tailnet. → [[systems/omniroute]], [[infrastructure/hosts]]
## Storage (live, 2026-11-01 03:5x AWST — `pvesm status` on both hosts)
### Host `pve` (100.96.100.82)
| Datastore | Type | Status | Total | Used | Used % |
|---|---|---|---|---|---|
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 717 GiB | **78.29 %** (`pvesm`) |
| `SSD1` | lvmthin | active | 931 GiB | 610 GiB | **65.46 %** |
| `fred-qnap` | **nfs** | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
| `local` | dir | active | 67.7 GiB | 55.3 GiB | **81.62 %** |
| `local-lvm` | lvmthin | active | 141 GiB | 20.1 GiB | **14.25 %** |
**`ssd0` has improved further: 88.17 % (2026-10-14) → 76.43 % (2026-11-01) → 78.29 % (2026-11-07)**, ~152 GiB free. `fred-qnap` is now
**84.21 %** and its type changed from **`cifs` to `nfs`** (13.4 TiB total, 11.3 TiB used) — the share was
remounted over NFS rather than SMB.
**Storage re-verified 2026-11-07 03:2x AWST** against live `pvesm status` on both hosts — every
percentage on both tables below was refreshed.
**`iMacHDD` no longer appears in `pvesm status` on `pve`** — the 2026-10-14 table listed it as an
inactive `cifs` entry. The iMac shares now live on **`m93p`** as `cifs-anthonyimac` and `cifs-imachdd`
(both active, 76.21 %). Storage was **reassigned from `pve` to `m93p`** rather than the mount failing.
### Host `m93p` (100.104.51.98)
| Datastore | Type | Status | Total | Used | Used % |
|---|---|---|---|---|---|
| `cifs-anthonyimac` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `cifs-imachdd` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `local` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `local-hdd` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `nfs-proxmox-31` | dir | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
| `nfs-proxmox-54` | dir | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
| `nfs-ssd0-39` | dir | active | 916 GiB | 717 GiB | **78.29 %** |
| `omb-internal` | dir | active | 93.9 GiB | 60.7 GiB | **64.66 %** |
**`m93p` storage was not documented before 2026-10-31.** Note that `m93p` exposes the **same backing
volumes as `pve`** under different names — `nfs-proxmox-31`/`nfs-proxmox-54` mirror `pve`'s `fred-qnap`
(both 84.21 %), and `nfs-ssd0-39` mirrors `pve`'s `ssd0` (both 78.29 %). The iMac shares that were
`iMacHDD` on `pve` are `cifs-anthonyimac` / `cifs-imachdd` on `m93p`.
**Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`).
**Use the `pvesm` number when comparing across hosts** — it is the only one that means the same thing on
both hypervisors.
## API Tokens
- `homepage@pve!homepage` — PVEAuditor role, privsep=0 (used by Homepage dashboard)
## Key Paths
- `/etc/pve/` — PVE config directory
- `/var/lib/vz/` — Local storage
- `/etc/pve/priv/token/` — API token storage
## Related
- [[systems/homepage-dashboard]] — Dashboard shows PVE stats
- [[infrastructure/hosts]] — Full host inventory
- [[infrastructure/network-map]] — Network topology