Files
agent-estate-wiki/incidents/2026-09-02-sftpgo-outage.md
T
Hermes ab3652aa20 Stocktake 2026-11-12: tailscale live-vs-page drift corrected; sftpgo incident re-verified; pre-dispatch-filter signature found recurring on a second job/pin
- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json.
  mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping
  (CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online.
  omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout).
  9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge
  (running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table
  unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed.
- incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer
  Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared.
- incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on
  cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved
  2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded.
- Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits),
  work-outlook (himalaya still login not supported), omniroute-free-deepseek
  (good-free/free-stack both HTTP 200, non-deepseek resolution).
- Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md.
- wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
2026-11-12 03:19:12 +08:00

4.9 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
SFTPGo Outage — CT299 Offline (2026-09-02) incident resolved 2026-09-04 2026-11-12 2026-11-12 high
incident
sftpgo
ct299
tailscale
proxmox
session:20260902_014540_c193be

SFTPGo Outage — CT299 Offline (2026-09-02)

Summary

When: 2026-09-02, ~01:45 AWST Duration: ~30 minutes Affected: SFTPGo service on CT299 (LXC 299), dashboard monitoring Root cause: CT299's tailscaled died (likely OOM — CT299 swap was 100% full at 43Gi used). When attempting to restart the container, PVE's pre-start hook failed because 5 stale lxc-attach processes from earlier diagnostic probes had pinned CT299's cgroup.

Timeline

Time (AWST) Event
~01:45 Dashboard reports SFTPGo service down
01:45–02:00 Investigation: CT299 unreachable on Tailscale (offline, last seen 7h ago), all TCP ports timeout
02:00–02:10 Root cause identified: CT299 swap exhausted (43Gi used, 100%), tailscaled inside CT299 had died; PVE pct start 299 fails with pre-start hook error: failed to remove directory /sys/fs/cgroup/lxc/299/ns/.lxc: Device or resource busy
02:10–02:15 Stale lxc-attach PIDs identified: 3037951, 3334928, 3352203, 3402511, 4004148 — killed all five
02:15 pct stop 299 --skiplock succeeded
02:16 pct start 299 succeeded — fresh PID 3471141
02:18 End-to-end verification: tailscale ping sftpgo → direct connection (192.168.178.92:41641), SFTPGo WebAdmin returns HTTP 401 (expected — auth required), SFTP banner SSH-2.0-SFTPGo_2.7.0 confirmed
02:19 Wiki updated: infrastructure/proxmox.md and current-state.md

Root Cause Analysis

Primary: CT299 swap exhaustion

  • CT299 has 43Gi swap allocated on a 31Gi RAM host — excessive swap likely contributed to OOM conditions
  • tailscaled inside CT299 was killed (likely OOM), breaking tailnet connectivity
  • Without tailscaled, CT299 couldn't be reached via Tailscale, so dashboard showed red

Secondary: Stale lxc-attach processes

  • Earlier diagnostic probes using pct enter 299 or similar left lxc-attach processes running
  • These processes pinned CT299's cgroup namespace at /sys/fs/cgroup/lxc/299/ns/.lxc
  • PVE's pre-start hook couldn't remove the directory because it was still in use
  • Result: pct start 299 failed repeatedly until the stale PIDs were killed

Resolution

# Kill stale lxc-attach processes
kill 3037951 3334928 3352203 3402511 4004148

# Stop CT299 (skip lock since it's already down)
pct stop 299 --skiplock

# Start CT299 fresh
pct start 299

# Verify
tailscale ping sftpgo
curl -sS http://100.88.223.70:10080/healthz

Re-verified 2026-10-21

Live state checked: CT299 sftpgo is running on pve (100.96.100.82) per pct list; the node is active in tailscale status at 100.88.223.70; the SFTPGo health endpoint http://100.88.223.70:10080/healthz returns HTTP 401 (expected — auth required, service up). The 2026-09-02 outage remains resolved; no recurrence.

Re-verified 2026-11-12 03:2x AWST (stocktake — page was 22d stale, the wiki's only stale page)

Live state re-probed and the residual claim still holds — no recurrence in 70 days. All three sources checked independently:

  • pct list on pve (100.96.100.82) → CT299 sftpgo running; pct config 299 confirms the guest (2 cores, 2048 MB RAM, swap: 512 — the over-provisioned 43 Gi swap recorded in the root cause is gone, so that concern has since been addressed).
  • tailscale status peer record for sftpgo → Online: true, 100.88.223.70, and tailscale ping sftpgo → pong from sftpgo (100.88.223.70) via 192.168.178.92:41641 in 0s.
  • SFTPGo health endpoint http://100.88.223.70:10080/healthz → HTTP 401, unchanged (auth required, service up).

Note that this outage and the two later CT299 freezes are separate faults: the 2026-09-02 cause was tailscaled death plus stale lxc-attach cgroup pinning, whereas 2026-11-04/05 was a wedged vzdump freezing the guest → incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage (also resolved).

Lessons Learned

  1. Clean up lxc-attach probes — always exit cleanly and verify no lingering processes after diagnostic sessions
  2. CT299 swap over-provisioned — 43Gi swap on 31Gi RAM is excessive; consider reducing swap allocation or trimming workload
  3. Dashboard correctly flagged the outage — even though root cause was inside the LXC (tailscaled died), the dashboard's reachability check caught it

Files Updated