--- title: vzdump Freeze Took CT299 / SFTPGo Offline for ~24h type: incident status: resolved created: 2026-11-04 updated: 2026-11-05 verified_on: 2026-11-05 confidence: high tags: [incident, proxmox, vzdump, backup, nfs, ct299, sftpgo, tailscale, manual-backup, ssd0] sources: [live-pve-probe, live-tailscale-probe, live-ssh, live-pve-tasks, live-probe-2026-11-05] --- # vzdump Froze CT299 / SFTPGo Twice - 11-03 (NFS target) and 11-04 (manual ssd0 run) > **Updated 2026-11-05 (nightly audit) - CT299 was frozen twice, not once.** The page below documents the > 2026-11-03 freeze. A **second, separate freeze** ran from **2026-11-04 08:30:04** to **2026-11-05 ~00:30** > (~16 h), caused by a **manual** `vzdump 299 ... --storage ssd0` run. Live probe at **00:54:21 AWST**: > `LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active` - **service up**. > See *Second freeze* below. **Window:** ~2026-11-03 03:00 AWST → 2026-11-04 04:4x AWST (≈25 h) **Status:** Resolved — wedged `vzdump` killed, guest unfrozen, SFTPGo verified healthy **Impact:** `sftpgo` (LXC CT299 on `pve`) was **frozen solid** — the container's filesystem was suspended by the backup job, so `sftpgo`, `tailscaled` and everything else inside it stopped. The tailnet peer went offline and the service was unreachable for the better part of a day. **Nothing alerted.** ## Symptoms - Tailscale peer `sftpgo` (`100.88.223.70`) offline; `tailscale ping` → timeout. - `ssh` into `100.88.223.70` hung until timeout; `pct exec 299 -- ...` also hung (exit 124). - The guest was not crashed — it was **frozen**: the backup job had taken an LVM snapshot and suspended the guest filesystem, then stalled before it could release it. ## Root cause A scheduled Proxmox backup job (`vzdump: backup-tue-g1-299`, `schedule tue 03:00`, `mode snapshot`, `storage fred-qnap`) started on **Tue 2026-11-03 at 03:00 AWST** and **never completed**. The task wedged at `create storage snapshot` on the **`fred-qnap` NFS storage** (`192.168.178.31:/Proxmox` mounted at `/mnt/pve/fred-qnap`): - The NFS write path was unresponsive — a `dd` write probe to `/mnt/pve/fred-qnap` timed out while the task was stuck. - The `vzdump` child process was in **`D` state** (uninterruptible I/O) on a FUSE `request_wait_answer` — it could not be killed by a normal signal while blocked in the kernel. - Its temp directory `/mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/` was left behind, and the guest's filesystem remained suspended for the whole ~24 h. **The failure mode is the important part: a backup task that cannot finish also cannot release the guest.** Proxmox's snapshot mode suspends the guest filesystem while the snapshot is created and the data is read out. A stalled task therefore does not merely "fail" — it **takes the guest down with it**, silently, until someone notices. ## Remediation (2026-11-04 04:4x AWST) 1. Identified the stuck task on `pve`: `vzdump` parent PID **274590**, child PID **274760** (state `D`). 2. `kill -TERM` on both PIDs — the parent died, the child remained blocked in `D` state on the FUSE wait. 3. `kill -KILL` on the remaining child — the kernel eventually released it once the NFS request timed out. 4. **The guest unfroze immediately.** `pct exec 299 -- /bin/echo ALIVE` → `ALIVE` (previously exit 124 / timeout). 5. Verified end to end: - `pct exec 299 -- systemctl is-active sftpgo` → **`active`** - `pct exec 299 -- ss -ltn` → `LISTEN *:8080` (WebAdmin) and `:2022` (SFTP) - `tailscale status` → peer `sftpgo` **Online: true** again - `tailscale ping 100.88.223.70` → **`pong from sftpgo via 192.168.178.92:41641 in 0s`** (direct, not DERP) - `curl http://100.88.223.70:8080/` → **HTTP 200** ## Post-remediation storage check `fred-qnap` was healthy again by the time the task was cleared: - `pvesm status` → `fred-qnap nfs active 14,380,169,216 KiB total / 12,084,987,904 used / 84.04 %` - `dd` write probe of 5 MiB with `oflag=direct` → **28.6 MB/s**, exit 0 — no stall. - The other backup task running concurrently (`vzdump: backup-sun-g1-100`, started 2026-11-03 10:36 AWST) **finished successfully** at 2026-11-04 04:56:59 (`TASK OK`, 00:16:25) against the same storage. So the NFS stall was **transient, not a dead mount** — it hit the one task that happened to hold a guest suspended. ## Second freeze - 2026-11-04 08:30:04 to 2026-11-05 ~00:30 (~16 h) The 11-03 freeze documented above was **not the only one**. A live probe of pve (`100.96.100.82`) on **2026-11-05 00:54:21 AWST**, plus the node's task history, shows a **second, separate wedge** the next morning: - `pvesh get /nodes/pve/tasks` lists **two CT299 `vzdump` tasks on 2026-11-04** - and **neither is the 03:00 fleet job**: - **`2026-11-04 07:36:44` - `vzdump` / status `job errors`** - a clean failure after ~11 minutes. Its log survives on the dump target: `/mnt/ssd0/dump/vzdump-lxc-299-2026_11_04-07_36_44.log` (597 B, mtime Nov 4 07:47). - **`2026-11-04 08:30:04` - `vzdump` / status `unexpected status`** - the wedge. `unexpected status` is what a killed task leaves behind; the 11-03 `03:00:04` row reads the same. - Three `push_file` tasks from client **`100.118.5.51` (CT460)** ran at **07:17:27 / 07:27:18 / 07:33:07**, immediately before the first attempt - so the sequence reads as operator/agent-initiated, not scheduled. - The abandoned temp dir **`/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299`** is still on disk (the dump dir holds 8.8 G total; the only other artefact is `openweb-245-preupgrade-20261101-211545.tar`). - **Both attempts targeted the SSD that is bind-mounted into the guest** (`--storage ssd0`, `/mnt/ssd0/dump`) - the same hazard class as the 11-03 NFS target: the guest is snapshotted onto storage it is actively using. **Recovery state, verified live 2026-11-05 00:54:21 AWST:** `LOCK: 0` in `/etc/pve/lxc/299.conf`; `/sys/fs/cgroup/lxc/299/cgroup.freeze = 0` **and** `ns/cgroup.freeze = 0`; `pct exec 299 -- /bin/echo ALIVE` returns `ALIVE`; `systemctl is-active sftpgo` -> `active`; **0** `vzdump`/`lxc-freeze` processes; SFTPGo reachable end-to-end from CT460 at `http://100.88.223.70:8080`. **CT299 is not frozen now.** **Open:** why a 299 dump was attempted twice on 11-04 outside any schedule, and whether the `ssd0` target should be used for this guest at all. See [[infrastructure/proxmox]] and the open items below. The abandoned `vzdumptmp3750030_299` directory should be reclaimed once nobody needs it. ## Prevention / open items - **Nothing monitored the backup tasks.** A `vzdump` that hangs for 24 h on a guest-holding storage is an outage generator, and no page, cron job or dashboard covers it. A watchdog that alerts on a `vzdump` task older than ~2 h — or that checks for `vzdump` tasks holding a suspended guest — is the missing control. - **NFS `hard` mount semantics amplify this.** `fred-qnap` is mounted `hard,fatal_neterrors=none`, so a stalled NFS server parks I/O in uninterruptible sleep (`D` state) indefinitely rather than failing fast. A `soft`/`intr` mount or a `timeout=` tuned for the backup path would let the task die instead of freezing the guest. - **The temp dir must be reclaimed manually.** `/mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/` was left on disk by the killed task; the existing cleanup cron skips a directory whose task is still registered. - **Do not assume "guest offline" means "guest crashed."** This guest was healthy and simply frozen. The first two probes (ping, ssh) both time out in exactly the same way as a dead guest — the distinguishing signal is a `vzdump` task in `D` state with an open snapshot on that guest. ## Related - [[infrastructure/proxmox]] — host, guest and storage inventory - [[infrastructure/backups]] — Hermes-side config backups (a separate mechanism from PVE `vzdump`) - [[incidents/2026-09-02-sftpgo-outage]] — the previous CT299 outage (swap exhaustion + cgroup pinning) — a **different** root cause with the same symptom - [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] — prior NFS-layer failure in this estate - [[systems/scheduled-tasks]] — cron fleet (no backup-task watchdog exists today)