resolve: Gitea config backup push dead since 2026-08-08 — 92-commit backlog drained, script hardened to fail loudly

- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo
- Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved)
- Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB
- Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification,
  decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree,
  honest AgentMail/Telegram delivery reporting
- Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed
- Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated
- Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with
  'lost its durable fire claim ownership' and left the tree dirty)
This commit is contained in:
Tony0410
2026-11-07 05:28:39 +08:00
parent b820993ecc
commit 0b4ff25fd2
14 changed files with 370 additions and 110 deletions
+35 -23
View File
@@ -3,8 +3,8 @@ title: Proxmox VE
type: infrastructure
status: active
created: 2026-07-26
updated: 2026-11-01
verified_on: 2026-11-01
updated: 2026-11-07
verified_on: 2026-11-07
confidence: high
tags: [proxmox, virtualization, infrastructure, homelab]
---
@@ -21,7 +21,8 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
- **Primary user:** `Anthony@pam` (full admin)
- **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve`
## Live LXC Inventory (verified 2026-10-31 via `pct list`)
## Live LXC Inventory (verified 2026-11-07 via `pct list`)
Re-verified 2026-11-07 03:2x AWST (stocktake): **live `pct list` on `pve` still returns exactly the 17 LXCs in the table below** (CT255 `mattermost` `running`, CT209 `cloudreve` and CT555 `elementsynapse` `stopped`, CT600 `omniroute` `running`, and **no CT601 on either hypervisor**), and **`qm list` on `pve` still returns exactly the 4 VMs** (100 `homeassistant`, 101 `guest-vps`, 110 `omarchy`, 282 `ubuntu-server-xfce` — all running; **no 310, no 403, no 401**). Both `pve-manager` packages are still **9.2.10** on **both** hosts. The **one correction** was on the `m93p` side: see the `m93p` section below — **CT107 (`mem0`) was missing**, so the m93p guest count was one short.
Re-verified 2026-11-01 03:5x AWST (stocktake): **`pct list` on `pve` and `qm list` on `pve` both still match the tables below exactly** — 17 LXCs and 4 VMs on `pve`, with CT255 `mattermost` still `running`, CT209 `cloudreve` and CT555 `elementsynapse` still `stopped`, and **no CT601 on either hypervisor** (`/etc/pve/lxc/601.conf` absent). **`m93p` re-confirmed**: `pve-manager/9.2.10` (same build as `pve`), `qm list` returns **VM 103 `cachyos-m93p` stopped** and **VM 106 `openmausbot-vm` running**, and `pct exec 1001 -- docker ps` returns `sshwifty` (CT1001). Storage percentages were refreshed against live `pvesm status` on both hosts (see the Storage section). **Note:** `m93p` SSH over the tailnet timed out on the first two attempts and succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout).
@@ -103,7 +104,7 @@ as a hypervisor.
- **Hostname:** `m93p` · **Tailscale:** `100.104.51.98`
- **PVE version:** **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, kernel `7.0.14-12-pve`) — **corrected 2026-10-31**. The previous entry read *"9.2.0"* and claimed `m93p` was *"materially newer than pve's 8.x"*. Both hosts now run the **identical `pve-manager 9.2.10` build**; only the kernel differs (`7.0.14-12-pve` vs `7.0.14-5-pve`). The hosts are **version-identical, not version-skewed**.
- **Uptime:** **up 1 day, 12:02** at 2026-11-01 03:56 AWST (live `uptime`) — the host **rebooted ~2026-10-30 16:0x**, consistent with the CT1264 openmausbot outage below. Load average was **7.66 / 12.80 / 11.32** (1/5/15 min) at that moment — elevated, worth watching
- **Uptime:** **up 1 day, 7:18** at 2026-11-07 04:38 AWST (live `uptime -s` → **boot 2026-11-05 21:19:52**; `uptime` → `up 1 day, 7:18`, load average **0.76 / 1.08 / 1.24** — normal). ⚠ **Corrected 2026-11-07:** the previous entry read *"up 1 day, 12:02 at 2026-11-01 03:56 AWST — the host rebooted ~2026-10-30 16:0x"*. The **reboot itself is still real** (`pve` booted **2026-10-30 16:24:00** per `uptime -s`, and the CT1264 openmausbot outage below follows from it), but the **uptime figure and the implied boot date were wrong** — the host has since rebooted again on **2026-11-05 21:19:52**, so it was up only **1 day, 7:18** when probed, not 12 h. The `pve` host's own boot time is **2026-10-30 16:24:00** (up **7 days, 12:14** at probe time)
- **Not clustered with `pve`:** `pvecm nodes` empty, no `/etc/pve/corosync.conf`
### LXC Inventory (verified 2026-10-31 via `pct list` on m93p)
@@ -115,6 +116,7 @@ as a hypervisor.
| 102 | vaultwarden | running | Vaultwarden password manager |
| 104 | rclone | running | Rclone |
| 105 | conduit | running | **Conduit** — iPhone calendar / health / location / contacts bridge (see [[systems/messaging-integrations]]). **Added 2026-10-31** — this guest was missing from the 2026-10-14 table despite running |
| 107 | mem0 | running | **mem0 MCP server** (`mem0-mcp.service` → `:8890`), Tailscale `100.73.94.53` (`mem0` peer) — **added 2026-11-07**, this guest was missing from the table despite running |
| 221 | gitea | running | **This is the host behind the wiki's `gitea` node — the Gitea remote lives here, not on `pve`** |
| 245 | openweb | running | Open WebUI — `pve` has no live CT245; the guest this page previously recorded on `pve` is this one |
| 500 | litellm | running | LiteLLM proxy |
@@ -130,8 +132,15 @@ as a hypervisor.
| 1263 | social-dashboard | running | Social dashboard |
| 1264 | openmausbot | **stopped** | OpenMausBot — **stopped**, which is why the `openmausbot` Tailscale peer is offline 9d. **Added 2026-10-31** — was missing from the 2026-10-14 table. The companion **VM106 `openmausbot-vm` is running** and holds the separate `openmausbot-vm` peer (`100.77.107.30`) |
**Re-verified 2026-10-31:** live `pct list` on `m93p` returns **19 LXCs — 18 running, 1 stopped
(CT998 nightbridge) plus CT1264 openmausbot stopped**. The 2026-10-14 table listed only **17**, omitting
**Re-verified 2026-11-07 03:2x AWST (stocktake):** live `pct list` on `m93p` returns **20 LXCs — 18 running,
2 stopped (CT998 `nightbridge` and CT1264 `openmausbot`)**. ⚠ **The count was one short**: the previous
figure of 19 omitted **CT107 (`mem0`)**, a live running guest confirmed from inside the container
(`pct exec 107 -- tailscale ip -4` → `100.73.94.53`, `systemctl is-active tailscaled` → `active`,
`mem0-mcp.service` → `active`). CT107 is now in the table above.
**Re-verified 2026-10-31:** the earlier reading of **19 LXCs — 18 running, 1 stopped
(CT998 nightbridge) plus CT1264 openmausbot stopped** was itself already one short for the same reason
(CT107 `mem0` was not counted). The 2026-10-14 table listed only **17**, omitting
two live guests: **CT105 (`conduit`)** and **CT1264 (`openmausbot`)**. Note that **VMIDs 103, 220 and 555
do not exist on `m93p`** (`/etc/pve/lxc/{103,220,555}.conf` all absent) — CT103 (cliproxyapi),
CT220 (caddy) and CT555 (elementsynapse) are on **`pve`** only, and `pve`'s CT555 is **stopped** while
@@ -162,36 +171,39 @@ tailnet. → [[systems/omniroute]], [[infrastructure/hosts]]
| Datastore | Type | Status | Total | Used | Used % |
|---|---|---|---|---|---|
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 700 GiB | **76.43 %** (`pvesm`) |
| `SSD1` | lvmthin | active | 932 GiB | 599 GiB | **64.32 %** |
| `fred-qnap` | **nfs** | active | 13.4 TiB | 11.0 TiB | **82.77 %** |
| `local` | dir | active | 67.7 GiB | 54.0 GiB | **79.66 %** |
| `local-lvm` | lvmthin | active | 141 GiB | 19.4 GiB | **13.75 %** |
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 717 GiB | **78.29 %** (`pvesm`) |
| `SSD1` | lvmthin | active | 931 GiB | 610 GiB | **65.46 %** |
| `fred-qnap` | **nfs** | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
| `local` | dir | active | 67.7 GiB | 55.3 GiB | **81.62 %** |
| `local-lvm` | lvmthin | active | 141 GiB | 20.1 GiB | **14.25 %** |
**`ssd0` has improved further: 88.17 % (2026-10-14) → 76.43 %**, ~169 GiB free. `fred-qnap` is now
**82.45 %** and its type changed from **`cifs` to `nfs`** (13.4 TiB total, 11.0 TiB used) — the share was
**`ssd0` has improved further: 88.17 % (2026-10-14) → 76.43 % (2026-11-01) → 78.29 % (2026-11-07)**, ~152 GiB free. `fred-qnap` is now
**84.21 %** and its type changed from **`cifs` to `nfs`** (13.4 TiB total, 11.3 TiB used) — the share was
remounted over NFS rather than SMB.
**Storage re-verified 2026-11-07 03:2x AWST** against live `pvesm status` on both hosts — every
percentage on both tables below was refreshed.
**`iMacHDD` no longer appears in `pvesm status` on `pve`** — the 2026-10-14 table listed it as an
inactive `cifs` entry. The iMac shares now live on **`m93p`** as `cifs-anthonyimac` and `cifs-imachdd`
(both active, 60.25 %). Storage was **reassigned from `pve` to `m93p`** rather than the mount failing.
(both active, 76.21 %). Storage was **reassigned from `pve` to `m93p`** rather than the mount failing.
### Host `m93p` (100.104.51.98)
| Datastore | Type | Status | Total | Used | Used % |
|---|---|---|---|---|---|
| `cifs-anthonyimac` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `cifs-imachdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `local` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `local-hdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `nfs-proxmox-31` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** |
| `nfs-proxmox-54` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** |
| `nfs-ssd0-39` | dir | active | 916 GiB | 700 GiB | **76.43 %** |
| `omb-internal` | dir | active | 93.9 GiB | 54.4 GiB | **57.92 %** |
| `cifs-anthonyimac` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `cifs-imachdd` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `local` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `local-hdd` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `nfs-proxmox-31` | dir | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
| `nfs-proxmox-54` | dir | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
| `nfs-ssd0-39` | dir | active | 916 GiB | 717 GiB | **78.29 %** |
| `omb-internal` | dir | active | 93.9 GiB | 60.7 GiB | **64.66 %** |
**`m93p` storage was not documented before 2026-10-31.** Note that `m93p` exposes the **same backing
volumes as `pve`** under different names — `nfs-proxmox-31`/`nfs-proxmox-54` mirror `pve`'s `fred-qnap`
(both 82.45 %), and `nfs-ssd0-39` mirrors `pve`'s `ssd0` (both 76.43 %). The iMac shares that were
(both 84.21 %), and `nfs-ssd0-39` mirrors `pve`'s `ssd0` (both 78.29 %). The iMac shares that were
`iMacHDD` on `pve` are `cifs-anthonyimac` / `cifs-imachdd` on `m93p`.
**Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`).