3.6 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start | concept | resolved | 2026-10-30 | 2026-10-30 | 2026-10-30 | 2026-11-30 | high |
|
|
m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start (2026-10-30)
Window: 2026-10-30 ~15:55 AWST (host boot) → 17:20 AWST (manual recovery)
Status: Resolved. All 18 LXC containers on m93p are running; self-healing recovery unit installed.
Impact: Every LXC whose rootfs lives on the nfs-ssd0-39 storage was stopped and unstartable.
This included FreshRSS (CT 1260), Gitea (221), NTFY (1241), LiteLLM (104), openweb (245),
rclone (105), paseo (1261), social-dashboard (1263), homebridge (1007), docker-01 (1006),
vaultwarden (102), omakase (1264), and others. The FreshRSS midday digest cron failed with
RuntimeError: Auth failed: ... urlopen error timed out — a misleading error; the cause was
total service unavailability, not credentials.
Root cause
Both Proxmox hosts rebooted within minutes of each other. The NFS server (192.168.178.39,
the PVE host) came back after m93p. m93p's boot-time NFS mount therefore failed, and
pct start for any CT with an NFS-backed rootfs failed with:
volume 'nfs-ssd0-39:1260/vm-1260-disk-0.raw' does not exist
There is no boot retry: m93p's mount attempt was one-shot and there is no Proxmox storage entry
for ssd0-39 (pvesm status does not list it — the mount is a plain filesystem mount at
/mnt/nfs/ssd0-39).
Secondary fault: three NFS mounts fail together on m93p, not one:
| Source | Mount point | Purpose |
|---|---|---|
192.168.178.39:/mnt/ssd0 |
/mnt/nfs/ssd0-39 |
FreshRSS + most service CT rootfs |
192.168.178.31:/Proxmox |
/mnt/nfs/proxmox-31 |
archive/dump storage |
192.168.178.54:/Proxmox |
/mnt/nfs/proxmox-54 |
archive/dump storage (same export as .31) |
Recovery performed
mount -t nfs -o vers=4,soft,intr 192.168.178.39:/mnt/ssd0 /mnt/nfs/ssd0-39
mount -t nfs -o vers=4,soft,intr 192.168.178.31:/Proxmox /mnt/nfs/proxmox-31
mount -t nfs -o vers=4,soft,intr 192.168.178.54:/Proxmox /mnt/nfs/proxmox-54
then pct start for each stopped CT (all with onboot=1). All 18 containers are up and their
Tailscale peers are online again.
Permanent fix
m93p-nfs-recover.service on m93p — enabled at multi-user.target, ordered after
pve-guests.service:
- Script:
/usr/local/sbin/m93p-nfs-recover.sh - Retries the three NFS mounts up to 10 times, 20s apart
- Then starts any CT still
stoppedwhose config hasonboot: 1 - Log:
/var/log/m93p-nfs-recover.log - Verify:
systemctl status m93p-nfs-recover.service
Runbook
If a service on m93p is unreachable and pct list shows it stopped:
ssh root@100.104.51.98 'mount | grep /mnt/nfs/'— expect three lines- If a line is missing:
systemctl start m93p-nfs-recover.service(it remounts and restarts the CTs) pct listto confirmrunning; then check the service from the Hermes host- FreshRSS specifically:
curl -sS -m 15 -o /dev/null -w '%{http_code}' http://100.83.60.93/
Lessons
- A Google Reader API
Auth failed ... timed outerror is an availability symptom first, an auth problem last. Probe the endpoint with curl before touching credentials. - The digest script needs up to 240s; a 180s timeout produced a spurious exit 124.
- FreshRSS on m93p is HTTP only —
https://freshrss.kangaroo-eel.ts.netfails because no Tailscale serve/funnel is configured for CT 1260.