8.1 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| vzdump Freeze Took CT299 / SFTPGo Offline for ~24h | incident | resolved | 2026-11-04 | 2026-11-05 | 2026-11-05 | high |
|
|
vzdump Froze CT299 / SFTPGo Twice - 11-03 (NFS target) and 11-04 (manual ssd0 run)
Updated 2026-11-05 (nightly audit) - CT299 was frozen twice, not once. The page below documents the 2026-11-03 freeze. A second, separate freeze ran from 2026-11-04 08:30:04 to 2026-11-05 ~00:30 (~16 h), caused by a manual
vzdump 299 ... --storage ssd0run. Live probe at 00:54:21 AWST:LOCK: 0,cgroup.freeze=0,ns/cgroup.freeze=0, guestALIVE,sftpgo: active- service up. See Second freeze below.
Window: ~2026-11-03 03:00 AWST → 2026-11-04 04:4x AWST (≈25 h)
Status: Resolved — wedged vzdump killed, guest unfrozen, SFTPGo verified healthy
Impact: sftpgo (LXC CT299 on pve) was frozen solid — the container's filesystem was suspended by the
backup job, so sftpgo, tailscaled and everything else inside it stopped. The tailnet peer went offline and the
service was unreachable for the better part of a day. Nothing alerted.
Symptoms
- Tailscale peer
sftpgo(100.88.223.70) offline;tailscale ping→ timeout. sshinto100.88.223.70hung until timeout;pct exec 299 -- ...also hung (exit 124).- The guest was not crashed — it was frozen: the backup job had taken an LVM snapshot and suspended the guest filesystem, then stalled before it could release it.
Root cause
A scheduled Proxmox backup job (vzdump: backup-tue-g1-299, schedule tue 03:00, mode snapshot,
storage fred-qnap) started on Tue 2026-11-03 at 03:00 AWST and never completed. The task wedged at
create storage snapshot on the fred-qnap NFS storage (192.168.178.31:/Proxmox mounted at /mnt/pve/fred-qnap):
- The NFS write path was unresponsive — a
ddwrite probe to/mnt/pve/fred-qnaptimed out while the task was stuck. - The
vzdumpchild process was inDstate (uninterruptible I/O) on a FUSErequest_wait_answer— it could not be killed by a normal signal while blocked in the kernel. - Its temp directory
/mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/was left behind, and the guest's filesystem remained suspended for the whole ~24 h.
The failure mode is the important part: a backup task that cannot finish also cannot release the guest. Proxmox's snapshot mode suspends the guest filesystem while the snapshot is created and the data is read out. A stalled task therefore does not merely "fail" — it takes the guest down with it, silently, until someone notices.
Remediation (2026-11-04 04:4x AWST)
- Identified the stuck task on
pve:vzdumpparent PID 274590, child PID 274760 (stateD). kill -TERMon both PIDs — the parent died, the child remained blocked inDstate on the FUSE wait.kill -KILLon the remaining child — the kernel eventually released it once the NFS request timed out.- The guest unfroze immediately.
pct exec 299 -- /bin/echo ALIVE→ALIVE(previously exit 124 / timeout). - Verified end to end:
pct exec 299 -- systemctl is-active sftpgo→activepct exec 299 -- ss -ltn→LISTEN *:8080(WebAdmin) and:2022(SFTP)tailscale status→ peersftpgoOnline: true againtailscale ping 100.88.223.70→pong from sftpgo via 192.168.178.92:41641 in 0s(direct, not DERP)curl http://100.88.223.70:8080/→ HTTP 200
Post-remediation storage check
fred-qnap was healthy again by the time the task was cleared:
pvesm status→fred-qnap nfs active 14,380,169,216 KiB total / 12,084,987,904 used / 84.04 %ddwrite probe of 5 MiB withoflag=direct→ 28.6 MB/s, exit 0 — no stall.- The other backup task running concurrently (
vzdump: backup-sun-g1-100, started 2026-11-03 10:36 AWST) finished successfully at 2026-11-04 04:56:59 (TASK OK, 00:16:25) against the same storage.
So the NFS stall was transient, not a dead mount — it hit the one task that happened to hold a guest suspended.
Second freeze - 2026-11-04 08:30:04 to 2026-11-05 ~00:30 (~16 h)
The 11-03 freeze documented above was not the only one. A live probe of pve (100.96.100.82) on
2026-11-05 00:54:21 AWST, plus the node's task history, shows a second, separate wedge the next morning:
pvesh get /nodes/pve/taskslists two CT299vzdumptasks on 2026-11-04 - and neither is the 03:00 fleet job:2026-11-04 07:36:44-vzdump/ statusjob errors- a clean failure after ~11 minutes. Its log survives on the dump target:/mnt/ssd0/dump/vzdump-lxc-299-2026_11_04-07_36_44.log(597 B, mtime Nov 4 07:47).2026-11-04 08:30:04-vzdump/ statusunexpected status- the wedge.unexpected statusis what a killed task leaves behind; the 11-0303:00:04row reads the same.
- Three
push_filetasks from client100.118.5.51(CT460) ran at 07:17:27 / 07:27:18 / 07:33:07, immediately before the first attempt - so the sequence reads as operator/agent-initiated, not scheduled. - The abandoned temp dir
/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299is still on disk (the dump dir holds 8.8 G total; the only other artefact isopenweb-245-preupgrade-20261101-211545.tar). - Both attempts targeted the SSD that is bind-mounted into the guest (
--storage ssd0,/mnt/ssd0/dump) - the same hazard class as the 11-03 NFS target: the guest is snapshotted onto storage it is actively using.
Recovery state, verified live 2026-11-05 00:54:21 AWST: LOCK: 0 in /etc/pve/lxc/299.conf;
/sys/fs/cgroup/lxc/299/cgroup.freeze = 0 and ns/cgroup.freeze = 0; pct exec 299 -- /bin/echo ALIVE
returns ALIVE; systemctl is-active sftpgo -> active; 0 vzdump/lxc-freeze processes; SFTPGo
reachable end-to-end from CT460 at http://100.88.223.70:8080. CT299 is not frozen now.
Open: why a 299 dump was attempted twice on 11-04 outside any schedule, and whether the ssd0 target
should be used for this guest at all. See infrastructure/proxmox and the open items below. The abandoned
vzdumptmp3750030_299 directory should be reclaimed once nobody needs it.
Prevention / open items
- Nothing monitored the backup tasks. A
vzdumpthat hangs for 24 h on a guest-holding storage is an outage generator, and no page, cron job or dashboard covers it. A watchdog that alerts on avzdumptask older than ~2 h — or that checks forvzdumptasks holding a suspended guest — is the missing control. - NFS
hardmount semantics amplify this.fred-qnapis mountedhard,fatal_neterrors=none, so a stalled NFS server parks I/O in uninterruptible sleep (Dstate) indefinitely rather than failing fast. Asoft/intrmount or atimeout=tuned for the backup path would let the task die instead of freezing the guest. - The temp dir must be reclaimed manually.
/mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/was left on disk by the killed task; the existing cleanup cron skips a directory whose task is still registered. - Do not assume "guest offline" means "guest crashed." This guest was healthy and simply frozen. The first two
probes (ping, ssh) both time out in exactly the same way as a dead guest — the distinguishing signal is a
vzdumptask inDstate with an open snapshot on that guest.
Related
- infrastructure/proxmox — host, guest and storage inventory
- infrastructure/backups — Hermes-side config backups (a separate mechanism from PVE
vzdump) - incidents/2026-09-02-sftpgo-outage — the previous CT299 outage (swap exhaustion + cgroup pinning) — a different root cause with the same symptom
- incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure — prior NFS-layer failure in this estate
- systems/scheduled-tasks — cron fleet (no backup-task watchdog exists today)