Files
agent-estate-wiki/infrastructure/proxmox.md
T
Tony0410 0b4ff25fd2 resolve: Gitea config backup push dead since 2026-08-08 — 92-commit backlog drained, script hardened to fail loudly
- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo
- Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved)
- Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB
- Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification,
  decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree,
  honest AgentMail/Telegram delivery reporting
- Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed
- Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated
- Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with
  'lost its durable fire claim ownership' and left the tree dirty)
2026-11-07 05:28:39 +08:00

17 KiB

title, type, status, created, updated, verified_on, confidence, tags
title type status created updated verified_on confidence tags
Proxmox VE infrastructure active 2026-07-26 2026-11-07 2026-11-07 high
proxmox
virtualization
infrastructure
homelab

Proxmox VE

Purpose

Hypervisor hosting all VMs and LXC containers for the homelab.

Details

  • Host: pve.kangaroo-eel.ts.net (Tailscale: 100.96.100.82)
  • Node name: pve
  • Version: Proxmox VE 9.2.10 (pve-manager/9.2.10/43df2e01f27a1a19, running kernel 7.0.14-5-pve) — corrected 2026-10-31; this page previously said "Proxmox VE 8.x", which was wrong. m93p runs the same pve-manager 9.2.10 build, so the two hosts are no longer version-skewed (see the m93p section).
  • Primary user: Anthony@pam (full admin)
  • Additional users: root@pam, glance@pve, prom-exporter@pve, homepage@pve

Live LXC Inventory (verified 2026-11-07 via pct list)

Re-verified 2026-11-07 03:2x AWST (stocktake): live pct list on pve still returns exactly the 17 LXCs in the table below (CT255 mattermost running, CT209 cloudreve and CT555 elementsynapse stopped, CT600 omniroute running, and no CT601 on either hypervisor), and qm list on pve still returns exactly the 4 VMs (100 homeassistant, 101 guest-vps, 110 omarchy, 282 ubuntu-server-xfce — all running; no 310, no 403, no 401). Both pve-manager packages are still 9.2.10 on both hosts. The one correction was on the m93p side: see the m93p section below — CT107 (mem0) was missing, so the m93p guest count was one short.

Re-verified 2026-11-01 03:5x AWST (stocktake): pct list on pve and qm list on pve both still match the tables below exactly — 17 LXCs and 4 VMs on pve, with CT255 mattermost still running, CT209 cloudreve and CT555 elementsynapse still stopped, and no CT601 on either hypervisor (/etc/pve/lxc/601.conf absent). m93p re-confirmed: pve-manager/9.2.10 (same build as pve), qm list returns VM 103 cachyos-m93p stopped and VM 106 openmausbot-vm running, and pct exec 1001 -- docker ps returns sshwifty (CT1001). Storage percentages were refreshed against live pvesm status on both hosts (see the Storage section). Note: m93p SSH over the tailnet timed out on the first two attempts and succeeded on the third — a flaky path, not a host outage (tailscale ping → pong throughout).

Two Proxmox hosts exist — always qualify a guest with its host. pvecm nodes is empty on both, so they are independent and unclustered. Guest IDs also collide — CT221 (gitea) and CT245 (openweb) carry the same VMID on both hosts — so an unqualified "CT221" is ambiguous. m93p (100.104.51.98, PVE 9.2.10) was discovered 2026-10-14 and is documented in its own section below. It hosts several services the wiki had placed on pve.

VMID Name Status Purpose
103 cliproxyapi running Cliproxy API
200 docker running Main Docker host — 60+ containers (CT460's docker.sock is NOT accessible to the agent user; use CT200 for fleet inspection)
201 runtipi running Runtipi — Homepage, ntfy, etc.
205 ollama running Ollama LLM inference
207 webserver running Web server
209 cloudreve stopped Cloudreve (deprecated)
215 n8n running N8n automation
255 mattermost running Mattermost — back online 2026-10-30/31 after a ~40-day outage; systemctl is-active mattermost → active, listening on :8065, gateway reconnected 2026-10-30 16:36:17
270 nextcloud running Nextcloud (apache2 active)
299 sftpgo running SFTPGo file server
333 clawtest running ClawTest — Nanobot, Zeroclaw
450 hermesagent running Hermes Agent CT450 (Chaos Lab)
460 hermes running Hermes CT460 (production)
465 qwenpaw running QwenPaw + Paseo daemon (active)
475 agentzero running AgentZero (100.106.54.94) — not previously recorded
555 elementsynapse stopped Element Synapse (deprecated) — also retired from Tailscale (offline 102d)
600 omniroute running OmniRoute — live server on :20128 (see systems/omniroute)

CT601 (omni) no longer exists. The 2026-10-14 inventory listed it as **stopped**; live ls /etc/pve/lxc/601.conf on both hypervisors returns no such file, and the orphaned omni Tailscale peer (100.93.204.4) has expired out of the tailnet entirely — the live tailscale status --json contains no peer named omni and no peer at that IP. The guest was deleted, not merely halted. The only OmniRoute nodes left are CT600 (omniroute, 100.88.81.19) and CT101 (omniroute-1, 100.78.63.20, on m93p) — both verified running with omniroute listening on 0.0.0.0:20128. → systems/omniroute, infrastructure/hosts

⚠ Four entries moved off this host (corrected 2026-10-14)

The 2026-10-11 inventory listed CT221 (gitea), CT241 (ntfy), CT245 (openweb) and CT260 (freshrss) as guests of pve. They are absent from live pct list on pve — yet all four services are still up, because they live on m93p. /etc/pve/lxc/ on pve holds only 260.conf.bak (no active config) and no vzdump task log references CT 221/241/245/260, so these were never deleted here: they were never on this host. The 2026-10-11 audit carried the older error forward.

Wiki entry (removed) Real host Real VMID
CT221 gitea m93p CT221 — same ID, different host (coincidence)
CT241 ntfy m93p CT1241
CT245 openweb m93p CT245 — same ID, different host
CT260 freshrss m93p CT1260

Live QEMU Inventory (verified 2026-10-31 via qm list)

VMID Name Status Notes
100 homeassistant running Home Assistant (5100 MB RAM — the page previously said 13 GB)
101 guest-vps running Guest VPS (6144 MB RAM, 48 GB disk) — was missing from this table entirely
110 omarchy running Omarchy (13000 MB RAM, 106 GB disk)
282 ubuntu-server-xfce running Ubuntu desktop (5000 MB RAM, 133 GB disk) — recorded stopped until 2026-10-14; live qm list shows running
310 cachyos absent CachyOS — no longer present in qm list on pve; the only CachyOS guest is cachyos-m93p (VM 103) on m93p, which is stopped
403 open-claw — OpenClaw agent; not present in qm list — it runs elsewhere (Open-Claw VM, see infrastructure/hosts)
401 hermes-old — Legacy Hermes VM; not present in qm list — appears decommissioned/removed

Not in either list: VM 403 (open-claw) and VM 401 (hermes-old) are referenced by the wiki but no longer appear in qm list. They have either been migrated off this host or deleted. Status unconfirmed — verify before relying on either entry.

Re-verified 2026-10-31: live qm list on pve returns exactly 4 VMs — 100 homeassistant, 101 guest-vps, 110 omarchy, 282 ubuntu-server-xfce, all running. Two corrections to the 2026-10-14 table: VM 101 (guest-vps) was missing despite running, and VM 310 (cachyos) no longer exists on this host (it had been recorded as stopped; the live qm list has no 310 at all). Home Assistant's memory was also corrected from "13GB" to the live 5100 MB.

Host m93p (100.104.51.98) — second Proxmox hypervisor, discovered 2026-10-14

Identified because a tailnet-wide scan for :8006 returned only m93p and pve. It had been sitting in the Tailscale node list as an unremarkable online peer, but no wiki page recorded it as a hypervisor.

  • Hostname: m93p · Tailscale: 100.104.51.98
  • PVE version: 9.2.10 (pve-manager/9.2.10/43df2e01f27a1a19, kernel 7.0.14-12-pve) — corrected 2026-10-31. The previous entry read "9.2.0" and claimed m93p was "materially newer than pve's 8.x". Both hosts now run the identical pve-manager 9.2.10 build; only the kernel differs (7.0.14-12-pve vs 7.0.14-5-pve). The hosts are version-identical, not version-skewed.
  • Uptime: up 1 day, 7:18 at 2026-11-07 04:38 AWST (live uptime -s → boot 2026-11-05 21:19:52; uptime → up 1 day, 7:18, load average 0.76 / 1.08 / 1.24 — normal). ⚠ Corrected 2026-11-07: the previous entry read "up 1 day, 12:02 at 2026-11-01 03:56 AWST — the host rebooted ~2026-10-30 16:0x". The reboot itself is still real (pve booted 2026-10-30 16:24:00 per uptime -s, and the CT1264 openmausbot outage below follows from it), but the uptime figure and the implied boot date were wrong — the host has since rebooted again on 2026-11-05 21:19:52, so it was up only 1 day, 7:18 when probed, not 12 h. The pve host's own boot time is 2026-10-30 16:24:00 (up 7 days, 12:14 at probe time)
  • Not clustered with pve: pvecm nodes empty, no /etc/pve/corosync.conf

LXC Inventory (verified 2026-10-31 via pct list on m93p)

VMID Name Status Notes
100 obsidian-livesync running Obsidian LiveSync — ID does not collide; pve has no CT100
101 omniroute running Second OmniRoute node, 100.78.63.20 (omniroute-1 Tailscale peer), omniroute listening on :20128 — identity/role now confirmed, see systems/omniroute
102 vaultwarden running Vaultwarden password manager
104 rclone running Rclone
105 conduit running Conduit — iPhone calendar / health / location / contacts bridge (see systems/messaging-integrations). Added 2026-10-31 — this guest was missing from the 2026-10-14 table despite running
107 mem0 running mem0 MCP server (mem0-mcp.service → :8890), Tailscale 100.73.94.53 (mem0 peer) — added 2026-11-07, this guest was missing from the table despite running
221 gitea running This is the host behind the wiki's gitea node — the Gitea remote lives here, not on pve
245 openweb running Open WebUI — pve has no live CT245; the guest this page previously recorded on pve is this one
500 litellm running LiteLLM proxy
998 nightbridge stopped Nightbridge
1001 docker-01 running Secondary Docker host
1006 freellmapi running FreeLLM API
1007 homebridge running Homebridge
1008 rustdesk running RustDesk remote access
1241 ntfy running ntfy push notifications — wiki's CT241
1260 freshrss running FreshRSS feed reader — wiki's CT260
1261 paseo running Paseo coding-agent daemon
1262 mailflow running MailFlow
1263 social-dashboard running Social dashboard
1264 openmausbot stopped OpenMausBot — stopped, which is why the openmausbot Tailscale peer is offline 9d. Added 2026-10-31 — was missing from the 2026-10-14 table. The companion VM106 openmausbot-vm is running and holds the separate openmausbot-vm peer (100.77.107.30)

Re-verified 2026-11-07 03:2x AWST (stocktake): live pct list on m93p returns 20 LXCs — 18 running, 2 stopped (CT998 nightbridge and CT1264 openmausbot). ⚠ The count was one short: the previous figure of 19 omitted CT107 (mem0), a live running guest confirmed from inside the container (pct exec 107 -- tailscale ip -4 → 100.73.94.53, systemctl is-active tailscaled → active, mem0-mcp.service → active). CT107 is now in the table above.

Re-verified 2026-10-31: the earlier reading of 19 LXCs — 18 running, 1 stopped (CT998 nightbridge) plus CT1264 openmausbot stopped was itself already one short for the same reason (CT107 mem0 was not counted). The 2026-10-14 table listed only 17, omitting two live guests: CT105 (conduit) and CT1264 (openmausbot). Note that VMIDs 103, 220 and 555 do not exist on m93p (/etc/pve/lxc/{103,220,555}.conf all absent) — CT103 (cliproxyapi), CT220 (caddy) and CT555 (elementsynapse) are on pve only, and pve's CT555 is stopped while m93p has no 555 at all. Guest IDs still collide across hosts, so always qualify with the host.

QEMU Inventory (verified 2026-10-31 via qm list on m93p)

VMID Name Status Notes
103 cachyos-m93p stopped CachyOS (6144 MB RAM, 80 GB disk)
106 openmausbot-vm running OpenMausBot VM (8192 MB RAM, 30 GB disk) — Added 2026-10-31; was missing from the 2026-10-14 table. Holds the openmausbot-vm Tailscale peer (100.77.107.30)

Reachability note: guest service ports on m93p were not reachable from CT460 during this pass (they bind to their own interfaces and are not Tailscale-Served). Only hypervisor :8006 on 100.104.51.98 is open. Reach them via the host or a Tailscale Serve route before probing services. Re-confirmed 2026-10-31: ports 3000, 4000, 5432, 8766, 8080 and 9000 were all closed from CT460.

✅ Follow-up closed 2026-10-31: m93p's omniroute (CT101) is a second live OmniRoute node, not a duplicate of CT600/CT601. Live pct config 101 on m93p confirms hostname: omniroute, and omniroute is listening on 0.0.0.0:20128 inside it; the omniroute-1 Tailscale peer (100.78.63.20) resolves to it. CT601 no longer exists on either host — /etc/pve/lxc/601.conf is absent from pve and m93p, and the orphaned omni peer (100.93.204.4) has expired out of the tailnet. → systems/omniroute, infrastructure/hosts

Storage (live, 2026-11-01 03:5x AWST — pvesm status on both hosts)

Host pve (100.96.100.82)

Datastore Type Status Total Used Used %
ssd0 (/mnt/ssd0, /dev/sdb) dir active 916 GiB 717 GiB 78.29 % (pvesm)
SSD1 lvmthin active 931 GiB 610 GiB 65.46 %
fred-qnap nfs active 13.4 TiB 11.3 TiB 84.21 %
local dir active 67.7 GiB 55.3 GiB 81.62 %
local-lvm lvmthin active 141 GiB 20.1 GiB 14.25 %

ssd0 has improved further: 88.17 % (2026-10-14) → 76.43 % (2026-11-01) → 78.29 % (2026-11-07), ~152 GiB free. fred-qnap is now 84.21 % and its type changed from cifs to nfs (13.4 TiB total, 11.3 TiB used) — the share was remounted over NFS rather than SMB.

Storage re-verified 2026-11-07 03:2x AWST against live pvesm status on both hosts — every percentage on both tables below was refreshed.

iMacHDD no longer appears in pvesm status on pve — the 2026-10-14 table listed it as an inactive cifs entry. The iMac shares now live on m93p as cifs-anthonyimac and cifs-imachdd (both active, 76.21 %). Storage was reassigned from pve to m93p rather than the mount failing.

Host m93p (100.104.51.98)

Datastore Type Status Total Used Used %
cifs-anthonyimac dir active 93.9 GiB 71.6 GiB 76.21 %
cifs-imachdd dir active 93.9 GiB 71.6 GiB 76.21 %
local dir active 93.9 GiB 71.6 GiB 76.21 %
local-hdd dir active 93.9 GiB 71.6 GiB 76.21 %
nfs-proxmox-31 dir active 13.4 TiB 11.3 TiB 84.21 %
nfs-proxmox-54 dir active 13.4 TiB 11.3 TiB 84.21 %
nfs-ssd0-39 dir active 916 GiB 717 GiB 78.29 %
omb-internal dir active 93.9 GiB 60.7 GiB 64.66 %

m93p storage was not documented before 2026-10-31. Note that m93p exposes the same backing volumes as pve under different names — nfs-proxmox-31/nfs-proxmox-54 mirror pve's fred-qnap (both 84.21 %), and nfs-ssd0-39 mirrors pve's ssd0 (both 78.29 %). The iMac shares that were iMacHDD on pve are cifs-anthonyimac / cifs-imachdd on m93p.

Mount points on the host (ls -d /mnt/*): /mnt/ssd0, /mnt/pve, /mnt/vm, /mnt/rootfs, plus per-guest mounts /mnt/ct200, /mnt/vm110-esp-ro, /mnt/vm282, /mnt/vm283, /mnt/vm400, /mnt/vm402, /mnt/esp110, /mnt/test-nfs, /mnt/tmp and retired ones (/mnt/old-openclaw-ro, /mnt/openclaw-old, /mnt/vm400-backup, /mnt/vm400-export, /mnt/vm402-fix).

Use the pvesm number when comparing across hosts — it is the only one that means the same thing on both hypervisors.

API Tokens

  • homepage@pve!homepage — PVEAuditor role, privsep=0 (used by Homepage dashboard)

Key Paths

  • /etc/pve/ — PVE config directory
  • /var/lib/vz/ — Local storage
  • /etc/pve/priv/token/ — API token storage