85 lines
3.6 KiB
Markdown
85 lines
3.6 KiB
Markdown
---
|
|
title: m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start
|
|
type: concept
|
|
status: resolved
|
|
created: 2026-10-30
|
|
updated: 2026-10-30
|
|
verified_on: 2026-10-30
|
|
review_after: 2026-11-30
|
|
confidence: high
|
|
tags: [proxmox, m93p, nfs, lxc, outage, freshrss]
|
|
sources: [live-probe, ssh-m93p, systemctl-status, pct-list, freshrss-script-output]
|
|
---
|
|
|
|
# m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start (2026-10-30)
|
|
|
|
**Window:** 2026-10-30 ~15:55 AWST (host boot) → 17:20 AWST (manual recovery)
|
|
**Status:** **Resolved.** All 18 LXC containers on m93p are running; self-healing recovery unit installed.
|
|
**Impact:** Every LXC whose rootfs lives on the `nfs-ssd0-39` storage was stopped and unstartable.
|
|
This included **FreshRSS (CT 1260)**, Gitea (221), NTFY (1241), LiteLLM (104), openweb (245),
|
|
rclone (105), paseo (1261), social-dashboard (1263), homebridge (1007), docker-01 (1006),
|
|
vaultwarden (102), omakase (1264), and others. The FreshRSS midday digest cron failed with
|
|
`RuntimeError: Auth failed: ... urlopen error timed out` — a misleading error; the cause was
|
|
total service unavailability, not credentials.
|
|
|
|
## Root cause
|
|
|
|
Both Proxmox hosts rebooted within minutes of each other. The NFS server (`192.168.178.39`,
|
|
the PVE host) came back **after** m93p. m93p's boot-time NFS mount therefore failed, and
|
|
`pct start` for any CT with an NFS-backed rootfs failed with:
|
|
|
|
```
|
|
volume 'nfs-ssd0-39:1260/vm-1260-disk-0.raw' does not exist
|
|
```
|
|
|
|
There is no boot retry: m93p's mount attempt was one-shot and there is no Proxmox storage entry
|
|
for `ssd0-39` (`pvesm status` does not list it — the mount is a plain filesystem mount at
|
|
`/mnt/nfs/ssd0-39`).
|
|
|
|
Secondary fault: **three** NFS mounts fail together on m93p, not one:
|
|
|
|
| Source | Mount point | Purpose |
|
|
|---|---|---|
|
|
| `192.168.178.39:/mnt/ssd0` | `/mnt/nfs/ssd0-39` | FreshRSS + most service CT rootfs |
|
|
| `192.168.178.31:/Proxmox` | `/mnt/nfs/proxmox-31` | archive/dump storage |
|
|
| `192.168.178.54:/Proxmox` | `/mnt/nfs/proxmox-54` | archive/dump storage (same export as .31) |
|
|
|
|
## Recovery performed
|
|
|
|
```
|
|
mount -t nfs -o vers=4,soft,intr 192.168.178.39:/mnt/ssd0 /mnt/nfs/ssd0-39
|
|
mount -t nfs -o vers=4,soft,intr 192.168.178.31:/Proxmox /mnt/nfs/proxmox-31
|
|
mount -t nfs -o vers=4,soft,intr 192.168.178.54:/Proxmox /mnt/nfs/proxmox-54
|
|
```
|
|
|
|
then `pct start` for each stopped CT (all with `onboot=1`). All 18 containers are up and their
|
|
Tailscale peers are online again.
|
|
|
|
## Permanent fix
|
|
|
|
`m93p-nfs-recover.service` on m93p — enabled at `multi-user.target`, ordered after
|
|
`pve-guests.service`:
|
|
|
|
- Script: `/usr/local/sbin/m93p-nfs-recover.sh`
|
|
- Retries the three NFS mounts up to 10 times, 20s apart
|
|
- Then starts any CT still `stopped` whose config has `onboot: 1`
|
|
- Log: `/var/log/m93p-nfs-recover.log`
|
|
- Verify: `systemctl status m93p-nfs-recover.service`
|
|
|
|
## Runbook
|
|
|
|
If a service on m93p is unreachable and `pct list` shows it `stopped`:
|
|
|
|
1. `ssh root@100.104.51.98 'mount | grep /mnt/nfs/'` — expect three lines
|
|
2. If a line is missing: `systemctl start m93p-nfs-recover.service` (it remounts and restarts the CTs)
|
|
3. `pct list` to confirm `running`; then check the service from the Hermes host
|
|
4. FreshRSS specifically: `curl -sS -m 15 -o /dev/null -w '%{http_code}' http://100.83.60.93/`
|
|
|
|
## Lessons
|
|
|
|
- A Google Reader API `Auth failed ... timed out` error is an availability symptom first, an auth
|
|
problem last. Probe the endpoint with curl before touching credentials.
|
|
- The digest script needs up to 240s; a 180s timeout produced a spurious exit 124.
|
|
- FreshRSS on m93p is **HTTP only** — `https://freshrss.kangaroo-eel.ts.net` fails because no
|
|
Tailscale serve/funnel is configured for CT 1260.
|