Files
agent-estate-wiki/incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
T

8.1 KiB

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
vzdump Freeze Took CT299 / SFTPGo Offline for ~24h incident resolved 2026-11-04 2026-11-05 2026-11-05 high
incident
proxmox
vzdump
backup
nfs
ct299
sftpgo
tailscale
manual-backup
ssd0
live-pve-probe
live-tailscale-probe
live-ssh
live-pve-tasks
live-probe-2026-11-05

vzdump Froze CT299 / SFTPGo Twice - 11-03 (NFS target) and 11-04 (manual ssd0 run)

Updated 2026-11-05 (nightly audit) - CT299 was frozen twice, not once. The page below documents the 2026-11-03 freeze. A second, separate freeze ran from 2026-11-04 08:30:04 to 2026-11-05 ~00:30 (~16 h), caused by a manual vzdump 299 ... --storage ssd0 run. Live probe at 00:54:21 AWST: LOCK: 0, cgroup.freeze=0, ns/cgroup.freeze=0, guest ALIVE, sftpgo: active - service up. See Second freeze below.

Window: ~2026-11-03 03:00 AWST → 2026-11-04 04:4x AWST (≈25 h) Status: Resolved — wedged vzdump killed, guest unfrozen, SFTPGo verified healthy Impact: sftpgo (LXC CT299 on pve) was frozen solid — the container's filesystem was suspended by the backup job, so sftpgo, tailscaled and everything else inside it stopped. The tailnet peer went offline and the service was unreachable for the better part of a day. Nothing alerted.

Symptoms

  • Tailscale peer sftpgo (100.88.223.70) offline; tailscale ping → timeout.
  • ssh into 100.88.223.70 hung until timeout; pct exec 299 -- ... also hung (exit 124).
  • The guest was not crashed — it was frozen: the backup job had taken an LVM snapshot and suspended the guest filesystem, then stalled before it could release it.

Root cause

A scheduled Proxmox backup job (vzdump: backup-tue-g1-299, schedule tue 03:00, mode snapshot, storage fred-qnap) started on Tue 2026-11-03 at 03:00 AWST and never completed. The task wedged at create storage snapshot on the fred-qnap NFS storage (192.168.178.31:/Proxmox mounted at /mnt/pve/fred-qnap):

  • The NFS write path was unresponsive — a dd write probe to /mnt/pve/fred-qnap timed out while the task was stuck.
  • The vzdump child process was in D state (uninterruptible I/O) on a FUSE request_wait_answer — it could not be killed by a normal signal while blocked in the kernel.
  • Its temp directory /mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/ was left behind, and the guest's filesystem remained suspended for the whole ~24 h.

The failure mode is the important part: a backup task that cannot finish also cannot release the guest. Proxmox's snapshot mode suspends the guest filesystem while the snapshot is created and the data is read out. A stalled task therefore does not merely "fail" — it takes the guest down with it, silently, until someone notices.

Remediation (2026-11-04 04:4x AWST)

  1. Identified the stuck task on pve: vzdump parent PID 274590, child PID 274760 (state D).
  2. kill -TERM on both PIDs — the parent died, the child remained blocked in D state on the FUSE wait.
  3. kill -KILL on the remaining child — the kernel eventually released it once the NFS request timed out.
  4. The guest unfroze immediately. pct exec 299 -- /bin/echo ALIVE → ALIVE (previously exit 124 / timeout).
  5. Verified end to end:
    • pct exec 299 -- systemctl is-active sftpgo → active
    • pct exec 299 -- ss -ltn → LISTEN *:8080 (WebAdmin) and :2022 (SFTP)
    • tailscale status → peer sftpgo Online: true again
    • tailscale ping 100.88.223.70 → pong from sftpgo via 192.168.178.92:41641 in 0s (direct, not DERP)
    • curl http://100.88.223.70:8080/ → HTTP 200

Post-remediation storage check

fred-qnap was healthy again by the time the task was cleared:

  • pvesm status → fred-qnap nfs active 14,380,169,216 KiB total / 12,084,987,904 used / 84.04 %
  • dd write probe of 5 MiB with oflag=direct → 28.6 MB/s, exit 0 — no stall.
  • The other backup task running concurrently (vzdump: backup-sun-g1-100, started 2026-11-03 10:36 AWST) finished successfully at 2026-11-04 04:56:59 (TASK OK, 00:16:25) against the same storage.

So the NFS stall was transient, not a dead mount — it hit the one task that happened to hold a guest suspended.

Second freeze - 2026-11-04 08:30:04 to 2026-11-05 ~00:30 (~16 h)

The 11-03 freeze documented above was not the only one. A live probe of pve (100.96.100.82) on 2026-11-05 00:54:21 AWST, plus the node's task history, shows a second, separate wedge the next morning:

  • pvesh get /nodes/pve/tasks lists two CT299 vzdump tasks on 2026-11-04 - and neither is the 03:00 fleet job:
    • 2026-11-04 07:36:44 - vzdump / status job errors - a clean failure after ~11 minutes. Its log survives on the dump target: /mnt/ssd0/dump/vzdump-lxc-299-2026_11_04-07_36_44.log (597 B, mtime Nov 4 07:47).
    • 2026-11-04 08:30:04 - vzdump / status unexpected status - the wedge. unexpected status is what a killed task leaves behind; the 11-03 03:00:04 row reads the same.
  • Three push_file tasks from client 100.118.5.51 (CT460) ran at 07:17:27 / 07:27:18 / 07:33:07, immediately before the first attempt - so the sequence reads as operator/agent-initiated, not scheduled.
  • The abandoned temp dir /mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299 is still on disk (the dump dir holds 8.8 G total; the only other artefact is openweb-245-preupgrade-20261101-211545.tar).
  • Both attempts targeted the SSD that is bind-mounted into the guest (--storage ssd0, /mnt/ssd0/dump) - the same hazard class as the 11-03 NFS target: the guest is snapshotted onto storage it is actively using.

Recovery state, verified live 2026-11-05 00:54:21 AWST: LOCK: 0 in /etc/pve/lxc/299.conf; /sys/fs/cgroup/lxc/299/cgroup.freeze = 0 and ns/cgroup.freeze = 0; pct exec 299 -- /bin/echo ALIVE returns ALIVE; systemctl is-active sftpgo -> active; 0 vzdump/lxc-freeze processes; SFTPGo reachable end-to-end from CT460 at http://100.88.223.70:8080. CT299 is not frozen now.

Open: why a 299 dump was attempted twice on 11-04 outside any schedule, and whether the ssd0 target should be used for this guest at all. See infrastructure/proxmox and the open items below. The abandoned vzdumptmp3750030_299 directory should be reclaimed once nobody needs it.

Prevention / open items

  • Nothing monitored the backup tasks. A vzdump that hangs for 24 h on a guest-holding storage is an outage generator, and no page, cron job or dashboard covers it. A watchdog that alerts on a vzdump task older than ~2 h — or that checks for vzdump tasks holding a suspended guest — is the missing control.
  • NFS hard mount semantics amplify this. fred-qnap is mounted hard,fatal_neterrors=none, so a stalled NFS server parks I/O in uninterruptible sleep (D state) indefinitely rather than failing fast. A soft/intr mount or a timeout= tuned for the backup path would let the task die instead of freezing the guest.
  • The temp dir must be reclaimed manually. /mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/ was left on disk by the killed task; the existing cleanup cron skips a directory whose task is still registered.
  • Do not assume "guest offline" means "guest crashed." This guest was healthy and simply frozen. The first two probes (ping, ssh) both time out in exactly the same way as a dead guest — the distinguishing signal is a vzdump task in D state with an open snapshot on that guest.