incident: m93p NFS mount loss took down all NFS-backed LXC containers (2026-10-30)
This commit is contained in:
@@ -0,0 +1,84 @@
|
||||
---
|
||||
title: m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start
|
||||
type: concept
|
||||
status: resolved
|
||||
created: 2026-10-30
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
review_after: 2026-11-30
|
||||
confidence: high
|
||||
tags: [proxmox, m93p, nfs, lxc, outage, freshrss]
|
||||
sources: [live-probe, ssh-m93p, systemctl-status, pct-list, freshrss-script-output]
|
||||
---
|
||||
|
||||
# m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start (2026-10-30)
|
||||
|
||||
**Window:** 2026-10-30 ~15:55 AWST (host boot) → 17:20 AWST (manual recovery)
|
||||
**Status:** **Resolved.** All 18 LXC containers on m93p are running; self-healing recovery unit installed.
|
||||
**Impact:** Every LXC whose rootfs lives on the `nfs-ssd0-39` storage was stopped and unstartable.
|
||||
This included **FreshRSS (CT 1260)**, Gitea (221), NTFY (1241), LiteLLM (104), openweb (245),
|
||||
rclone (105), paseo (1261), social-dashboard (1263), homebridge (1007), docker-01 (1006),
|
||||
vaultwarden (102), omakase (1264), and others. The FreshRSS midday digest cron failed with
|
||||
`RuntimeError: Auth failed: ... urlopen error timed out` — a misleading error; the cause was
|
||||
total service unavailability, not credentials.
|
||||
|
||||
## Root cause
|
||||
|
||||
Both Proxmox hosts rebooted within minutes of each other. The NFS server (`192.168.178.39`,
|
||||
the PVE host) came back **after** m93p. m93p's boot-time NFS mount therefore failed, and
|
||||
`pct start` for any CT with an NFS-backed rootfs failed with:
|
||||
|
||||
```
|
||||
volume 'nfs-ssd0-39:1260/vm-1260-disk-0.raw' does not exist
|
||||
```
|
||||
|
||||
There is no boot retry: m93p's mount attempt was one-shot and there is no Proxmox storage entry
|
||||
for `ssd0-39` (`pvesm status` does not list it — the mount is a plain filesystem mount at
|
||||
`/mnt/nfs/ssd0-39`).
|
||||
|
||||
Secondary fault: **three** NFS mounts fail together on m93p, not one:
|
||||
|
||||
| Source | Mount point | Purpose |
|
||||
|---|---|---|
|
||||
| `192.168.178.39:/mnt/ssd0` | `/mnt/nfs/ssd0-39` | FreshRSS + most service CT rootfs |
|
||||
| `192.168.178.31:/Proxmox` | `/mnt/nfs/proxmox-31` | archive/dump storage |
|
||||
| `192.168.178.54:/Proxmox` | `/mnt/nfs/proxmox-54` | archive/dump storage (same export as .31) |
|
||||
|
||||
## Recovery performed
|
||||
|
||||
```
|
||||
mount -t nfs -o vers=4,soft,intr 192.168.178.39:/mnt/ssd0 /mnt/nfs/ssd0-39
|
||||
mount -t nfs -o vers=4,soft,intr 192.168.178.31:/Proxmox /mnt/nfs/proxmox-31
|
||||
mount -t nfs -o vers=4,soft,intr 192.168.178.54:/Proxmox /mnt/nfs/proxmox-54
|
||||
```
|
||||
|
||||
then `pct start` for each stopped CT (all with `onboot=1`). All 18 containers are up and their
|
||||
Tailscale peers are online again.
|
||||
|
||||
## Permanent fix
|
||||
|
||||
`m93p-nfs-recover.service` on m93p — enabled at `multi-user.target`, ordered after
|
||||
`pve-guests.service`:
|
||||
|
||||
- Script: `/usr/local/sbin/m93p-nfs-recover.sh`
|
||||
- Retries the three NFS mounts up to 10 times, 20s apart
|
||||
- Then starts any CT still `stopped` whose config has `onboot: 1`
|
||||
- Log: `/var/log/m93p-nfs-recover.log`
|
||||
- Verify: `systemctl status m93p-nfs-recover.service`
|
||||
|
||||
## Runbook
|
||||
|
||||
If a service on m93p is unreachable and `pct list` shows it `stopped`:
|
||||
|
||||
1. `ssh root@100.104.51.98 'mount | grep /mnt/nfs/'` — expect three lines
|
||||
2. If a line is missing: `systemctl start m93p-nfs-recover.service` (it remounts and restarts the CTs)
|
||||
3. `pct list` to confirm `running`; then check the service from the Hermes host
|
||||
4. FreshRSS specifically: `curl -sS -m 15 -o /dev/null -w '%{http_code}' http://100.83.60.93/`
|
||||
|
||||
## Lessons
|
||||
|
||||
- A Google Reader API `Auth failed ... timed out` error is an availability symptom first, an auth
|
||||
problem last. Probe the endpoint with curl before touching credentials.
|
||||
- The digest script needs up to 240s; a 180s timeout produced a spurious exit 124.
|
||||
- FreshRSS on m93p is **HTTP only** — `https://freshrss.kangaroo-eel.ts.net` fails because no
|
||||
Tailscale serve/funnel is configured for CT 1260.
|
||||
Reference in New Issue
Block a user