From db6af668781af2c3635fbec0aed8335f9b6316c1 Mon Sep 17 00:00:00 2001 From: Tony0410 <47945770+Tony0410@users.noreply.github.com> Date: Fri, 30 Oct 2026 17:41:34 +0800 Subject: [PATCH] incident: m93p NFS mount loss took down all NFS-backed LXC containers (2026-10-30) --- ...30-m93p-nfs-mount-loss-lxc-boot-failure.md | 84 +++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100644 incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md diff --git a/incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md b/incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md new file mode 100644 index 0000000..77e2c58 --- /dev/null +++ b/incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md @@ -0,0 +1,84 @@ +--- +title: m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start +type: concept +status: resolved +created: 2026-10-30 +updated: 2026-10-30 +verified_on: 2026-10-30 +review_after: 2026-11-30 +confidence: high +tags: [proxmox, m93p, nfs, lxc, outage, freshrss] +sources: [live-probe, ssh-m93p, systemctl-status, pct-list, freshrss-script-output] +--- + +# m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start (2026-10-30) + +**Window:** 2026-10-30 ~15:55 AWST (host boot) → 17:20 AWST (manual recovery) +**Status:** **Resolved.** All 18 LXC containers on m93p are running; self-healing recovery unit installed. +**Impact:** Every LXC whose rootfs lives on the `nfs-ssd0-39` storage was stopped and unstartable. +This included **FreshRSS (CT 1260)**, Gitea (221), NTFY (1241), LiteLLM (104), openweb (245), +rclone (105), paseo (1261), social-dashboard (1263), homebridge (1007), docker-01 (1006), +vaultwarden (102), omakase (1264), and others. The FreshRSS midday digest cron failed with +`RuntimeError: Auth failed: ... urlopen error timed out` — a misleading error; the cause was +total service unavailability, not credentials. + +## Root cause + +Both Proxmox hosts rebooted within minutes of each other. The NFS server (`192.168.178.39`, +the PVE host) came back **after** m93p. m93p's boot-time NFS mount therefore failed, and +`pct start` for any CT with an NFS-backed rootfs failed with: + +``` +volume 'nfs-ssd0-39:1260/vm-1260-disk-0.raw' does not exist +``` + +There is no boot retry: m93p's mount attempt was one-shot and there is no Proxmox storage entry +for `ssd0-39` (`pvesm status` does not list it — the mount is a plain filesystem mount at +`/mnt/nfs/ssd0-39`). + +Secondary fault: **three** NFS mounts fail together on m93p, not one: + +| Source | Mount point | Purpose | +|---|---|---| +| `192.168.178.39:/mnt/ssd0` | `/mnt/nfs/ssd0-39` | FreshRSS + most service CT rootfs | +| `192.168.178.31:/Proxmox` | `/mnt/nfs/proxmox-31` | archive/dump storage | +| `192.168.178.54:/Proxmox` | `/mnt/nfs/proxmox-54` | archive/dump storage (same export as .31) | + +## Recovery performed + +``` +mount -t nfs -o vers=4,soft,intr 192.168.178.39:/mnt/ssd0 /mnt/nfs/ssd0-39 +mount -t nfs -o vers=4,soft,intr 192.168.178.31:/Proxmox /mnt/nfs/proxmox-31 +mount -t nfs -o vers=4,soft,intr 192.168.178.54:/Proxmox /mnt/nfs/proxmox-54 +``` + +then `pct start` for each stopped CT (all with `onboot=1`). All 18 containers are up and their +Tailscale peers are online again. + +## Permanent fix + +`m93p-nfs-recover.service` on m93p — enabled at `multi-user.target`, ordered after +`pve-guests.service`: + +- Script: `/usr/local/sbin/m93p-nfs-recover.sh` +- Retries the three NFS mounts up to 10 times, 20s apart +- Then starts any CT still `stopped` whose config has `onboot: 1` +- Log: `/var/log/m93p-nfs-recover.log` +- Verify: `systemctl status m93p-nfs-recover.service` + +## Runbook + +If a service on m93p is unreachable and `pct list` shows it `stopped`: + +1. `ssh root@100.104.51.98 'mount | grep /mnt/nfs/'` — expect three lines +2. If a line is missing: `systemctl start m93p-nfs-recover.service` (it remounts and restarts the CTs) +3. `pct list` to confirm `running`; then check the service from the Hermes host +4. FreshRSS specifically: `curl -sS -m 15 -o /dev/null -w '%{http_code}' http://100.83.60.93/` + +## Lessons + +- A Google Reader API `Auth failed ... timed out` error is an availability symptom first, an auth + problem last. Probe the endpoint with curl before touching credentials. +- The digest script needs up to 240s; a 180s timeout produced a spurious exit 124. +- FreshRSS on m93p is **HTTP only** — `https://freshrss.kangaroo-eel.ts.net` fails because no + Tailscale serve/funnel is configured for CT 1260.