--- title: m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start type: concept status: resolved created: 2026-10-30 updated: 2026-10-30 verified_on: 2026-10-30 review_after: 2026-11-30 confidence: high tags: [proxmox, m93p, nfs, lxc, outage, freshrss] sources: [live-probe, ssh-m93p, systemctl-status, pct-list, freshrss-script-output] --- # m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start (2026-10-30) **Window:** 2026-10-30 ~15:55 AWST (host boot) → 17:20 AWST (manual recovery) **Status:** **Resolved.** All 18 LXC containers on m93p are running; self-healing recovery unit installed. **Impact:** Every LXC whose rootfs lives on the `nfs-ssd0-39` storage was stopped and unstartable. This included **FreshRSS (CT 1260)**, Gitea (221), NTFY (1241), LiteLLM (104), openweb (245), rclone (105), paseo (1261), social-dashboard (1263), homebridge (1007), docker-01 (1006), vaultwarden (102), omakase (1264), and others. The FreshRSS midday digest cron failed with `RuntimeError: Auth failed: ... urlopen error timed out` — a misleading error; the cause was total service unavailability, not credentials. ## Root cause Both Proxmox hosts rebooted within minutes of each other. The NFS server (`192.168.178.39`, the PVE host) came back **after** m93p. m93p's boot-time NFS mount therefore failed, and `pct start` for any CT with an NFS-backed rootfs failed with: ``` volume 'nfs-ssd0-39:1260/vm-1260-disk-0.raw' does not exist ``` There is no boot retry: m93p's mount attempt was one-shot and there is no Proxmox storage entry for `ssd0-39` (`pvesm status` does not list it — the mount is a plain filesystem mount at `/mnt/nfs/ssd0-39`). Secondary fault: **three** NFS mounts fail together on m93p, not one: | Source | Mount point | Purpose | |---|---|---| | `192.168.178.39:/mnt/ssd0` | `/mnt/nfs/ssd0-39` | FreshRSS + most service CT rootfs | | `192.168.178.31:/Proxmox` | `/mnt/nfs/proxmox-31` | archive/dump storage | | `192.168.178.54:/Proxmox` | `/mnt/nfs/proxmox-54` | archive/dump storage (same export as .31) | ## Recovery performed ``` mount -t nfs -o vers=4,soft,intr 192.168.178.39:/mnt/ssd0 /mnt/nfs/ssd0-39 mount -t nfs -o vers=4,soft,intr 192.168.178.31:/Proxmox /mnt/nfs/proxmox-31 mount -t nfs -o vers=4,soft,intr 192.168.178.54:/Proxmox /mnt/nfs/proxmox-54 ``` then `pct start` for each stopped CT (all with `onboot=1`). All 18 containers are up and their Tailscale peers are online again. ## Permanent fix `m93p-nfs-recover.service` on m93p — enabled at `multi-user.target`, ordered after `pve-guests.service`: - Script: `/usr/local/sbin/m93p-nfs-recover.sh` - Retries the three NFS mounts up to 10 times, 20s apart - Then starts any CT still `stopped` whose config has `onboot: 1` - Log: `/var/log/m93p-nfs-recover.log` - Verify: `systemctl status m93p-nfs-recover.service` ## Runbook If a service on m93p is unreachable and `pct list` shows it `stopped`: 1. `ssh root@100.104.51.98 'mount | grep /mnt/nfs/'` — expect three lines 2. If a line is missing: `systemctl start m93p-nfs-recover.service` (it remounts and restarts the CTs) 3. `pct list` to confirm `running`; then check the service from the Hermes host 4. FreshRSS specifically: `curl -sS -m 15 -o /dev/null -w '%{http_code}' http://100.83.60.93/` ## Lessons - A Google Reader API `Auth failed ... timed out` error is an availability symptom first, an auth problem last. Probe the endpoint with curl before touching credentials. - The digest script needs up to 240s; a 180s timeout produced a spurious exit 124. - FreshRSS on m93p is **HTTP only** — `https://freshrss.kangaroo-eel.ts.net` fails because no Tailscale serve/funnel is configured for CT 1260.