--- title: SFTPGo Outage — CT299 Offline (2026-09-02) type: incident status: resolved created: 2026-09-04 updated: 2026-11-12 verified_on: 2026-11-12 confidence: high tags: [incident, sftpgo, ct299, tailscale, proxmox] sources: [session:20260902_014540_c193be] --- # SFTPGo Outage — CT299 Offline (2026-09-02) ## Summary **When:** 2026-09-02, ~01:45 AWST **Duration:** ~30 minutes **Affected:** SFTPGo service on CT299 (LXC 299), dashboard monitoring **Root cause:** CT299's `tailscaled` died (likely OOM — CT299 swap was 100% full at 43Gi used). When attempting to restart the container, PVE's pre-start hook failed because 5 stale `lxc-attach` processes from earlier diagnostic probes had pinned CT299's cgroup. ## Timeline | Time (AWST) | Event | |---|---| | ~01:45 | Dashboard reports SFTPGo service down | | 01:45–02:00 | Investigation: CT299 unreachable on Tailscale (`offline, last seen 7h ago`), all TCP ports timeout | | 02:00–02:10 | Root cause identified: CT299 swap exhausted (43Gi used, 100%), `tailscaled` inside CT299 had died; PVE `pct start 299` fails with pre-start hook error: `failed to remove directory /sys/fs/cgroup/lxc/299/ns/.lxc: Device or resource busy` | | 02:10–02:15 | Stale `lxc-attach` PIDs identified: 3037951, 3334928, 3352203, 3402511, 4004148 — killed all five | | 02:15 | `pct stop 299 --skiplock` succeeded | | 02:16 | `pct start 299` succeeded — fresh PID 3471141 | | 02:18 | End-to-end verification: `tailscale ping sftpgo` → direct connection (192.168.178.92:41641), SFTPGo WebAdmin returns HTTP 401 (expected — auth required), SFTP banner `SSH-2.0-SFTPGo_2.7.0` confirmed | | 02:19 | Wiki updated: `infrastructure/proxmox.md` and `current-state.md` | ## Root Cause Analysis ### Primary: CT299 swap exhaustion - CT299 has 43Gi swap allocated on a 31Gi RAM host — excessive swap likely contributed to OOM conditions - `tailscaled` inside CT299 was killed (likely OOM), breaking tailnet connectivity - Without `tailscaled`, CT299 couldn't be reached via Tailscale, so dashboard showed red ### Secondary: Stale `lxc-attach` processes - Earlier diagnostic probes using `pct enter 299` or similar left `lxc-attach` processes running - These processes pinned CT299's cgroup namespace at `/sys/fs/cgroup/lxc/299/ns/.lxc` - PVE's pre-start hook couldn't remove the directory because it was still in use - Result: `pct start 299` failed repeatedly until the stale PIDs were killed ## Resolution ```bash # Kill stale lxc-attach processes kill 3037951 3334928 3352203 3402511 4004148 # Stop CT299 (skip lock since it's already down) pct stop 299 --skiplock # Start CT299 fresh pct start 299 # Verify tailscale ping sftpgo curl -sS http://100.88.223.70:10080/healthz ``` ## Re-verified 2026-10-21 Live state checked: **CT299 `sftpgo` is `running`** on `pve` (100.96.100.82) per `pct list`; the node is `active` in `tailscale status` at `100.88.223.70`; the SFTPGo health endpoint `http://100.88.223.70:10080/healthz` returns **HTTP 401** (expected — auth required, service up). The 2026-09-02 outage remains resolved; no recurrence. ## Re-verified 2026-11-12 03:2x AWST (stocktake — page was 22d stale, the wiki's only stale page) Live state re-probed and **the residual claim still holds — no recurrence in 70 days.** All three sources checked independently: - `pct list` on `pve` (100.96.100.82) → **CT299 `sftpgo` `running`**; `pct config 299` confirms the guest (2 cores, 2048 MB RAM, **`swap: 512`** — the over-provisioned 43 Gi swap recorded in the root cause is gone, so that concern has since been addressed). - `tailscale status` peer record for `sftpgo` → **`Online: true`**, `100.88.223.70`, and `tailscale ping sftpgo` → **`pong from sftpgo (100.88.223.70) via 192.168.178.92:41641 in 0s`**. - SFTPGo health endpoint `http://100.88.223.70:10080/healthz` → **HTTP 401**, unchanged (auth required, service up). Note that this outage and the two later CT299 freezes are **separate faults**: the 2026-09-02 cause was `tailscaled` death plus stale `lxc-attach` cgroup pinning, whereas 2026-11-04/05 was a wedged `vzdump` freezing the guest → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]] (also resolved). ## Lessons Learned 1. **Clean up `lxc-attach` probes** — always exit cleanly and verify no lingering processes after diagnostic sessions 2. **CT299 swap over-provisioned** — 43Gi swap on 31Gi RAM is excessive; consider reducing swap allocation or trimming workload 3. **Dashboard correctly flagged the outage** — even though root cause was inside the LXC (tailscaled died), the dashboard's reachability check caught it ## Files Updated - [[infrastructure/proxmox]] — CT299 entry now documents the incident and swap concern - [[current-state]] — Recent Changes (2026-09-02) section added - [[systems/scheduled-tasks]] — no changes (incident not cron-related) ## Related - [[infrastructure/proxmox]] - [[systems/sftpgo]] (if page exists — otherwise create) - [[incidents/index]]