- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json. mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping (CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online. omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout). 9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge (running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed. - incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared. - incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved 2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded. - Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits), work-outlook (himalaya still login not supported), omniroute-free-deepseek (good-free/free-stack both HTTP 200, non-deepseek resolution). - Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md. - wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
4.9 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SFTPGo Outage — CT299 Offline (2026-09-02) | incident | resolved | 2026-09-04 | 2026-11-12 | 2026-11-12 | high |
|
|
SFTPGo Outage — CT299 Offline (2026-09-02)
Summary
When: 2026-09-02, ~01:45 AWST
Duration: ~30 minutes
Affected: SFTPGo service on CT299 (LXC 299), dashboard monitoring
Root cause: CT299's tailscaled died (likely OOM — CT299 swap was 100% full at 43Gi used). When attempting to restart the container, PVE's pre-start hook failed because 5 stale lxc-attach processes from earlier diagnostic probes had pinned CT299's cgroup.
Timeline
| Time (AWST) | Event |
|---|---|
| ~01:45 | Dashboard reports SFTPGo service down |
| 01:45–02:00 | Investigation: CT299 unreachable on Tailscale (offline, last seen 7h ago), all TCP ports timeout |
| 02:00–02:10 | Root cause identified: CT299 swap exhausted (43Gi used, 100%), tailscaled inside CT299 had died; PVE pct start 299 fails with pre-start hook error: failed to remove directory /sys/fs/cgroup/lxc/299/ns/.lxc: Device or resource busy |
| 02:10–02:15 | Stale lxc-attach PIDs identified: 3037951, 3334928, 3352203, 3402511, 4004148 — killed all five |
| 02:15 | pct stop 299 --skiplock succeeded |
| 02:16 | pct start 299 succeeded — fresh PID 3471141 |
| 02:18 | End-to-end verification: tailscale ping sftpgo → direct connection (192.168.178.92:41641), SFTPGo WebAdmin returns HTTP 401 (expected — auth required), SFTP banner SSH-2.0-SFTPGo_2.7.0 confirmed |
| 02:19 | Wiki updated: infrastructure/proxmox.md and current-state.md |
Root Cause Analysis
Primary: CT299 swap exhaustion
- CT299 has 43Gi swap allocated on a 31Gi RAM host — excessive swap likely contributed to OOM conditions
tailscaledinside CT299 was killed (likely OOM), breaking tailnet connectivity- Without
tailscaled, CT299 couldn't be reached via Tailscale, so dashboard showed red
Secondary: Stale lxc-attach processes
- Earlier diagnostic probes using
pct enter 299or similar leftlxc-attachprocesses running - These processes pinned CT299's cgroup namespace at
/sys/fs/cgroup/lxc/299/ns/.lxc - PVE's pre-start hook couldn't remove the directory because it was still in use
- Result:
pct start 299failed repeatedly until the stale PIDs were killed
Resolution
# Kill stale lxc-attach processes
kill 3037951 3334928 3352203 3402511 4004148
# Stop CT299 (skip lock since it's already down)
pct stop 299 --skiplock
# Start CT299 fresh
pct start 299
# Verify
tailscale ping sftpgo
curl -sS http://100.88.223.70:10080/healthz
Re-verified 2026-10-21
Live state checked: CT299 sftpgo is running on pve (100.96.100.82) per pct list;
the node is active in tailscale status at 100.88.223.70; the SFTPGo health endpoint
http://100.88.223.70:10080/healthz returns HTTP 401 (expected — auth required, service up).
The 2026-09-02 outage remains resolved; no recurrence.
Re-verified 2026-11-12 03:2x AWST (stocktake — page was 22d stale, the wiki's only stale page)
Live state re-probed and the residual claim still holds — no recurrence in 70 days. All three sources checked independently:
pct listonpve(100.96.100.82) → CT299sftpgorunning;pct config 299confirms the guest (2 cores, 2048 MB RAM,swap: 512— the over-provisioned 43 Gi swap recorded in the root cause is gone, so that concern has since been addressed).tailscale statuspeer record forsftpgo→Online: true,100.88.223.70, andtailscale ping sftpgo→pong from sftpgo (100.88.223.70) via 192.168.178.92:41641 in 0s.- SFTPGo health endpoint
http://100.88.223.70:10080/healthz→ HTTP 401, unchanged (auth required, service up).
Note that this outage and the two later CT299 freezes are separate faults: the 2026-09-02 cause was
tailscaled death plus stale lxc-attach cgroup pinning, whereas 2026-11-04/05 was a wedged vzdump
freezing the guest → incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage (also resolved).
Lessons Learned
- Clean up
lxc-attachprobes — always exit cleanly and verify no lingering processes after diagnostic sessions - CT299 swap over-provisioned — 43Gi swap on 31Gi RAM is excessive; consider reducing swap allocation or trimming workload
- Dashboard correctly flagged the outage — even though root cause was inside the LXC (tailscaled died), the dashboard's reachability check caught it
Files Updated
- infrastructure/proxmox — CT299 entry now documents the incident and swap concern
- current-state — Recent Changes (2026-09-02) section added
- systems/scheduled-tasks — no changes (incident not cron-related)
Related
- infrastructure/proxmox
- systems/sftpgo (if page exists — otherwise create)
- incidents/index