Files

T

Tony0410 77250ea8c9 incident: SFTPGo outage 2026-09-02 — CT299 tailscaled death + stale lxc-attach recovery

- New incident page documenting CT299 SFTPGo outage (02:18 AWST)
- Root cause: swap exhaustion (43Gi, 100%) killed tailscaled; 5 stale lxc-attach PIDs blocked pct start
- Resolution: killed PIDs 3037951/3334928/3352203/3402511/4004148, pct stop/start 299
- Updated current-state.md with Recent Changes (2026-09-02), bumped updated to 2026-09-04
- Updated log.md with incident entry
- Updated index.md last-updated date

No secrets written. Verified end-to-end: tailscale direct, SFTPGo WebAdmin HTTP 401, SFTP banner SSH-2.0-SFTPGo_2.7.0.

2026-09-04 00:05:32 +08:00

3.4 KiB

Raw Permalink Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources

title

type

status

created

updated

verified_on

confidence

SFTPGo Outage — CT299 Offline (2026-09-02)

Summary

When: 2026-09-02, ~01:45 AWST Duration: ~30 minutes Affected: SFTPGo service on CT299 (LXC 299), dashboard monitoring Root cause: CT299's tailscaled died (likely OOM — CT299 swap was 100% full at 43Gi used). When attempting to restart the container, PVE's pre-start hook failed because 5 stale lxc-attach processes from earlier diagnostic probes had pinned CT299's cgroup.

Timeline

Time (AWST)	Event
~01:45	Dashboard reports SFTPGo service down
01:45–02:00	Investigation: CT299 unreachable on Tailscale (`offline, last seen 7h ago`), all TCP ports timeout
02:00–02:10	Root cause identified: CT299 swap exhausted (43Gi used, 100%), `tailscaled` inside CT299 had died; PVE `pct start 299` fails with pre-start hook error: `failed to remove directory /sys/fs/cgroup/lxc/299/ns/.lxc: Device or resource busy`
02:10–02:15	Stale `lxc-attach` PIDs identified: 3037951, 3334928, 3352203, 3402511, 4004148 — killed all five
02:15	`pct stop 299 --skiplock` succeeded
02:16	`pct start 299` succeeded — fresh PID 3471141
02:18	End-to-end verification: `tailscale ping sftpgo` → direct connection (192.168.178.92:41641), SFTPGo WebAdmin returns HTTP 401 (expected — auth required), SFTP banner `SSH-2.0-SFTPGo_2.7.0` confirmed
02:19	Wiki updated: `infrastructure/proxmox.md` and `current-state.md`

Root Cause Analysis

Primary: CT299 swap exhaustion

CT299 has 43Gi swap allocated on a 31Gi RAM host — excessive swap likely contributed to OOM conditions
tailscaled inside CT299 was killed (likely OOM), breaking tailnet connectivity
Without tailscaled, CT299 couldn't be reached via Tailscale, so dashboard showed red

Secondary: Stale `lxc-attach` processes

Earlier diagnostic probes using pct enter 299 or similar left lxc-attach processes running
These processes pinned CT299's cgroup namespace at /sys/fs/cgroup/lxc/299/ns/.lxc
PVE's pre-start hook couldn't remove the directory because it was still in use
Result: pct start 299 failed repeatedly until the stale PIDs were killed

Resolution

# Kill stale lxc-attach processes
kill 3037951 3334928 3352203 3402511 4004148

# Stop CT299 (skip lock since it's already down)
pct stop 299 --skiplock

# Start CT299 fresh
pct start 299

# Verify
tailscale ping sftpgo
curl -sS http://100.88.223.70:10080/healthz

Lessons Learned

Clean up lxc-attach probes — always exit cleanly and verify no lingering processes after diagnostic sessions
CT299 swap over-provisioned — 43Gi swap on 31Gi RAM is excessive; consider reducing swap allocation or trimming workload
Dashboard correctly flagged the outage — even though root cause was inside the LXC (tailscaled died), the dashboard's reachability check caught it

Files Updated

infrastructure/proxmox — CT299 entry now documents the incident and swap concern
current-state — Recent Changes (2026-09-02) section added
systems/scheduled-tasks — no changes (incident not cron-related)

infrastructure/proxmox
systems/sftpgo (if page exists — otherwise create)
incidents/index

3.4 KiB Raw Permalink Blame History Unescape Escape