---
title: SFTPGo Outage — CT299 Offline (2026-09-02)
type: incident
status: resolved
created: 2026-09-04
updated: 2026-09-04
verified_on: 2026-09-04
confidence: high
tags: [incident, sftpgo, ct299, tailscale, proxmox]
sources: [session:20260902_014540_c193be]
---

# SFTPGo Outage — CT299 Offline (2026-09-02)

## Summary

**When:** 2026-09-02, ~01:45 AWST
**Duration:** ~30 minutes
**Affected:** SFTPGo service on CT299 (LXC 299), dashboard monitoring
**Root cause:** CT299's `tailscaled` died (likely OOM — CT299 swap was 100% full at 43Gi used). When attempting to restart the container, PVE's pre-start hook failed because 5 stale `lxc-attach` processes from earlier diagnostic probes had pinned CT299's cgroup.

## Timeline

| Time (AWST) | Event |
|---|---|
| ~01:45 | Dashboard reports SFTPGo service down |
| 01:45–02:00 | Investigation: CT299 unreachable on Tailscale (`offline, last seen 7h ago`), all TCP ports timeout |
| 02:00–02:10 | Root cause identified: CT299 swap exhausted (43Gi used, 100%), `tailscaled` inside CT299 had died; PVE `pct start 299` fails with pre-start hook error: `failed to remove directory /sys/fs/cgroup/lxc/299/ns/.lxc: Device or resource busy` |
| 02:10–02:15 | Stale `lxc-attach` PIDs identified: 3037951, 3334928, 3352203, 3402511, 4004148 — killed all five |
| 02:15 | `pct stop 299 --skiplock` succeeded |
| 02:16 | `pct start 299` succeeded — fresh PID 3471141 |
| 02:18 | End-to-end verification: `tailscale ping sftpgo` → direct connection (192.168.178.92:41641), SFTPGo WebAdmin returns HTTP 401 (expected — auth required), SFTP banner `SSH-2.0-SFTPGo_2.7.0` confirmed |
| 02:19 | Wiki updated: `infrastructure/proxmox.md` and `current-state.md` |

## Root Cause Analysis

### Primary: CT299 swap exhaustion
- CT299 has 43Gi swap allocated on a 31Gi RAM host — excessive swap likely contributed to OOM conditions
- `tailscaled` inside CT299 was killed (likely OOM), breaking tailnet connectivity
- Without `tailscaled`, CT299 couldn't be reached via Tailscale, so dashboard showed red

### Secondary: Stale `lxc-attach` processes
- Earlier diagnostic probes using `pct enter 299` or similar left `lxc-attach` processes running
- These processes pinned CT299's cgroup namespace at `/sys/fs/cgroup/lxc/299/ns/.lxc`
- PVE's pre-start hook couldn't remove the directory because it was still in use
- Result: `pct start 299` failed repeatedly until the stale PIDs were killed

## Resolution

```bash
# Kill stale lxc-attach processes
kill 3037951 3334928 3352203 3402511 4004148

# Stop CT299 (skip lock since it's already down)
pct stop 299 --skiplock

# Start CT299 fresh
pct start 299

# Verify
tailscale ping sftpgo
curl -sS http://100.88.223.70:10080/healthz
```

## Lessons Learned

1. **Clean up `lxc-attach` probes** — always exit cleanly and verify no lingering processes after diagnostic sessions
2. **CT299 swap over-provisioned** — 43Gi swap on 31Gi RAM is excessive; consider reducing swap allocation or trimming workload
3. **Dashboard correctly flagged the outage** — even though root cause was inside the LXC (tailscaled died), the dashboard's reachability check caught it

## Files Updated

- [[infrastructure/proxmox]] — CT299 entry now documents the incident and swap concern
- [[current-state]] — Recent Changes (2026-09-02) section added
- [[systems/scheduled-tasks]] — no changes (incident not cron-related)

## Related

- [[infrastructure/proxmox]]
- [[systems/sftpgo]] (if page exists — otherwise create)
- [[incidents/index]]