Files
agent-estate-wiki/incidents/2026-09-02-sftpgo-outage.md
T
Tony0410 c1d3b59260 🦉 wiki stocktake 2026-10-21: tailscale node table rebuilt, 2 missing Uber Eats reports recovered
Maintenance script ran clean (skills-index 346 -> 347, freshness audit, Quartz rebuilt
596 files and deployed, hermes-wiki-serve.service restarted).

infrastructure/tailscale.md — node table rebuilt from live `tailscale status --json`.
  The 2026-10-19 table had drifted: octop (100.92.163.122) and springroll
  (100.100.153.125) missing entirely, openmausbot / openmausbot-vm present in prose
  but absent from the table body, omarchymac now offline, all offline ages stale.
  Rebuilt keyed on DNS name rather than the JSON HostName — HostName is `ct260` for
  the freshrss node and BOTH OmniRoute nodes report the hostname `omniroute`, which
  collides silently. Now 51 non-funnel nodes. Container states cross-checked against
  live `pct list` on pve and m93p: CT255 stopped, CT601 stopped, CT299 running.

reports/uber-eats/ — 2026-08 and 2026-09 recovered. The generator wrote only to
  ~/wiki/reports/uber-eats/ and never to the canonical tree, so both months were
  missing from the published wiki. Root cause fixed in
  ~/.hermes/cron/scripts/generate-uber-eats-report.py: it now targets
  ~/workspace/agent-estate-wiki/reports/uber-eats/ and mirrors to the legacy tree,
  and now emits YAML frontmatter (without it the pages landed in the audit's
  "no updated DATE" bucket and Quartz could not read the page date). 2026-08
  previous-month column corrected from $0.00 to $264.51 / 11 orders.

incidents/2026-08-15-skills-index-drift.md — re-verified live; the named
  regenerate-skills-index.sh no longer exists on disk (regeneration now lives in
  wiki-maintain.sh step 1/3). Incident prevention confirmed working: 347 skills.
incidents/2026-09-02-sftpgo-outage.md — re-verified live: CT299 running, health
  endpoint HTTP 401 (auth required, service up), no recurrence.

reports/communities-wa-social-media-audit-2026.md — left stale (54d). Its source
  document no longer exists on disk, so the baseline cannot be re-verified against
  a live source. Flagged, not edited.

Freshness after this run: 1 stale, 94 fresh, 6 no-date of 101 pages.
2026-10-21 03:14:26 +08:00

3.8 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
SFTPGo Outage — CT299 Offline (2026-09-02) incident resolved 2026-09-04 2026-10-21 2026-10-21 high
incident
sftpgo
ct299
tailscale
proxmox
session:20260902_014540_c193be

SFTPGo Outage — CT299 Offline (2026-09-02)

Summary

When: 2026-09-02, ~01:45 AWST Duration: ~30 minutes Affected: SFTPGo service on CT299 (LXC 299), dashboard monitoring Root cause: CT299's tailscaled died (likely OOM — CT299 swap was 100% full at 43Gi used). When attempting to restart the container, PVE's pre-start hook failed because 5 stale lxc-attach processes from earlier diagnostic probes had pinned CT299's cgroup.

Timeline

Time (AWST) Event
~01:45 Dashboard reports SFTPGo service down
01:45–02:00 Investigation: CT299 unreachable on Tailscale (offline, last seen 7h ago), all TCP ports timeout
02:00–02:10 Root cause identified: CT299 swap exhausted (43Gi used, 100%), tailscaled inside CT299 had died; PVE pct start 299 fails with pre-start hook error: failed to remove directory /sys/fs/cgroup/lxc/299/ns/.lxc: Device or resource busy
02:10–02:15 Stale lxc-attach PIDs identified: 3037951, 3334928, 3352203, 3402511, 4004148 — killed all five
02:15 pct stop 299 --skiplock succeeded
02:16 pct start 299 succeeded — fresh PID 3471141
02:18 End-to-end verification: tailscale ping sftpgo → direct connection (192.168.178.92:41641), SFTPGo WebAdmin returns HTTP 401 (expected — auth required), SFTP banner SSH-2.0-SFTPGo_2.7.0 confirmed
02:19 Wiki updated: infrastructure/proxmox.md and current-state.md

Root Cause Analysis

Primary: CT299 swap exhaustion

  • CT299 has 43Gi swap allocated on a 31Gi RAM host — excessive swap likely contributed to OOM conditions
  • tailscaled inside CT299 was killed (likely OOM), breaking tailnet connectivity
  • Without tailscaled, CT299 couldn't be reached via Tailscale, so dashboard showed red

Secondary: Stale lxc-attach processes

  • Earlier diagnostic probes using pct enter 299 or similar left lxc-attach processes running
  • These processes pinned CT299's cgroup namespace at /sys/fs/cgroup/lxc/299/ns/.lxc
  • PVE's pre-start hook couldn't remove the directory because it was still in use
  • Result: pct start 299 failed repeatedly until the stale PIDs were killed

Resolution

# Kill stale lxc-attach processes
kill 3037951 3334928 3352203 3402511 4004148

# Stop CT299 (skip lock since it's already down)
pct stop 299 --skiplock

# Start CT299 fresh
pct start 299

# Verify
tailscale ping sftpgo
curl -sS http://100.88.223.70:10080/healthz

Re-verified 2026-10-21

Live state checked: CT299 sftpgo is running on pve (100.96.100.82) per pct list; the node is active in tailscale status at 100.88.223.70; the SFTPGo health endpoint http://100.88.223.70:10080/healthz returns HTTP 401 (expected — auth required, service up). The 2026-09-02 outage remains resolved; no recurrence.

Lessons Learned

  1. Clean up lxc-attach probes — always exit cleanly and verify no lingering processes after diagnostic sessions
  2. CT299 swap over-provisioned — 43Gi swap on 31Gi RAM is excessive; consider reducing swap allocation or trimming workload
  3. Dashboard correctly flagged the outage — even though root cause was inside the LXC (tailscaled died), the dashboard's reachability check caught it

Files Updated