incident: SFTPGo outage 2026-09-02 — CT299 tailscaled death + stale lxc-attach recovery

- New incident page documenting CT299 SFTPGo outage (02:18 AWST) - Root cause: swap exhaustion (43Gi, 100%) killed tailscaled; 5 stale lxc-attach PIDs blocked pct start - Resolution: killed PIDs 3037951/3334928/3352203/3402511/4004148, pct stop/start 299 - Updated current-state.md with Recent Changes (2026-09-02), bumped updated to 2026-09-04 - Updated log.md with incident entry - Updated index.md last-updated date No secrets written. Verified end-to-end: tailscale direct, SFTPGo WebAdmin HTTP 401, SFTP banner SSH-2.0-SFTPGo_2.7.0.
2026-09-04 00:05:32 +08:00
parent 1d42c6bc83
commit 77250ea8c9
13 changed files with 746 additions and 507 deletions
@@ -0,0 +1,81 @@
+---
+title: SFTPGo Outage — CT299 Offline (2026-09-02)
+type: incident
+status: resolved
+created: 2026-09-04
+updated: 2026-09-04
+verified_on: 2026-09-04
+confidence: high
+tags: [incident, sftpgo, ct299, tailscale, proxmox]
+sources: [session:20260902_014540_c193be]
+---
+
+# SFTPGo Outage — CT299 Offline (2026-09-02)
+
+## Summary
+
+**When:** 2026-09-02, ~01:45 AWST
+**Duration:** ~30 minutes
+**Affected:** SFTPGo service on CT299 (LXC 299), dashboard monitoring
+**Root cause:** CT299's `tailscaled` died (likely OOM — CT299 swap was 100% full at 43Gi used). When attempting to restart the container, PVE's pre-start hook failed because 5 stale `lxc-attach` processes from earlier diagnostic probes had pinned CT299's cgroup.
+
+## Timeline
+
+| Time (AWST) | Event |
+|---|---|
+| ~01:45 | Dashboard reports SFTPGo service down |
+| 01:45–02:00 | Investigation: CT299 unreachable on Tailscale (`offline, last seen 7h ago`), all TCP ports timeout |
+| 02:00–02:10 | Root cause identified: CT299 swap exhausted (43Gi used, 100%), `tailscaled` inside CT299 had died; PVE `pct start 299` fails with pre-start hook error: `failed to remove directory /sys/fs/cgroup/lxc/299/ns/.lxc: Device or resource busy` |
+| 02:10–02:15 | Stale `lxc-attach` PIDs identified: 3037951, 3334928, 3352203, 3402511, 4004148 — killed all five |
+| 02:15 | `pct stop 299 --skiplock` succeeded |
+| 02:16 | `pct start 299` succeeded — fresh PID 3471141 |
+| 02:18 | End-to-end verification: `tailscale ping sftpgo` → direct connection (192.168.178.92:41641), SFTPGo WebAdmin returns HTTP 401 (expected — auth required), SFTP banner `SSH-2.0-SFTPGo_2.7.0` confirmed |
+| 02:19 | Wiki updated: `infrastructure/proxmox.md` and `current-state.md` |
+
+## Root Cause Analysis
+
+### Primary: CT299 swap exhaustion
+- CT299 has 43Gi swap allocated on a 31Gi RAM host — excessive swap likely contributed to OOM conditions
+- `tailscaled` inside CT299 was killed (likely OOM), breaking tailnet connectivity
+- Without `tailscaled`, CT299 couldn't be reached via Tailscale, so dashboard showed red
+
+### Secondary: Stale `lxc-attach` processes
+- Earlier diagnostic probes using `pct enter 299` or similar left `lxc-attach` processes running
+- These processes pinned CT299's cgroup namespace at `/sys/fs/cgroup/lxc/299/ns/.lxc`
+- PVE's pre-start hook couldn't remove the directory because it was still in use
+- Result: `pct start 299` failed repeatedly until the stale PIDs were killed
+
+## Resolution
+
+```bash
+# Kill stale lxc-attach processes
+kill 3037951 3334928 3352203 3402511 4004148
+
+# Stop CT299 (skip lock since it's already down)
+pct stop 299 --skiplock
+
+# Start CT299 fresh
+pct start 299
+
+# Verify
+tailscale ping sftpgo
+curl -sS http://100.88.223.70:10080/healthz
+```
+
+## Lessons Learned
+
+1. **Clean up `lxc-attach` probes** — always exit cleanly and verify no lingering processes after diagnostic sessions
+2. **CT299 swap over-provisioned** — 43Gi swap on 31Gi RAM is excessive; consider reducing swap allocation or trimming workload
+3. **Dashboard correctly flagged the outage** — even though root cause was inside the LXC (tailscaled died), the dashboard's reachability check caught it
+
+## Files Updated
+
+- [[infrastructure/proxmox]] — CT299 entry now documents the incident and swap concern
+- [[current-state]] — Recent Changes (2026-09-02) section added
+- [[systems/scheduled-tasks]] — no changes (incident not cron-related)
+
+## Related
+
+- [[infrastructure/proxmox]]
+- [[systems/sftpgo]] (if page exists — otherwise create)
+- [[incidents/index]]