Files
agent-estate-wiki/incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
T

3.6 KiB

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start concept resolved 2026-10-30 2026-10-30 2026-10-30 2026-11-30 high
proxmox
m93p
nfs
lxc
outage
freshrss
live-probe
ssh-m93p
systemctl-status
pct-list
freshrss-script-output

m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start (2026-10-30)

Window: 2026-10-30 ~15:55 AWST (host boot) → 17:20 AWST (manual recovery) Status: Resolved. All 18 LXC containers on m93p are running; self-healing recovery unit installed. Impact: Every LXC whose rootfs lives on the nfs-ssd0-39 storage was stopped and unstartable. This included FreshRSS (CT 1260), Gitea (221), NTFY (1241), LiteLLM (104), openweb (245), rclone (105), paseo (1261), social-dashboard (1263), homebridge (1007), docker-01 (1006), vaultwarden (102), omakase (1264), and others. The FreshRSS midday digest cron failed with RuntimeError: Auth failed: ... urlopen error timed out — a misleading error; the cause was total service unavailability, not credentials.

Root cause

Both Proxmox hosts rebooted within minutes of each other. The NFS server (192.168.178.39, the PVE host) came back after m93p. m93p's boot-time NFS mount therefore failed, and pct start for any CT with an NFS-backed rootfs failed with:

volume 'nfs-ssd0-39:1260/vm-1260-disk-0.raw' does not exist

There is no boot retry: m93p's mount attempt was one-shot and there is no Proxmox storage entry for ssd0-39 (pvesm status does not list it — the mount is a plain filesystem mount at /mnt/nfs/ssd0-39).

Secondary fault: three NFS mounts fail together on m93p, not one:

Source Mount point Purpose
192.168.178.39:/mnt/ssd0 /mnt/nfs/ssd0-39 FreshRSS + most service CT rootfs
192.168.178.31:/Proxmox /mnt/nfs/proxmox-31 archive/dump storage
192.168.178.54:/Proxmox /mnt/nfs/proxmox-54 archive/dump storage (same export as .31)

Recovery performed

mount -t nfs -o vers=4,soft,intr 192.168.178.39:/mnt/ssd0 /mnt/nfs/ssd0-39
mount -t nfs -o vers=4,soft,intr 192.168.178.31:/Proxmox /mnt/nfs/proxmox-31
mount -t nfs -o vers=4,soft,intr 192.168.178.54:/Proxmox /mnt/nfs/proxmox-54

then pct start for each stopped CT (all with onboot=1). All 18 containers are up and their Tailscale peers are online again.

Permanent fix

m93p-nfs-recover.service on m93p — enabled at multi-user.target, ordered after pve-guests.service:

  • Script: /usr/local/sbin/m93p-nfs-recover.sh
  • Retries the three NFS mounts up to 10 times, 20s apart
  • Then starts any CT still stopped whose config has onboot: 1
  • Log: /var/log/m93p-nfs-recover.log
  • Verify: systemctl status m93p-nfs-recover.service

Runbook

If a service on m93p is unreachable and pct list shows it stopped:

  1. ssh root@100.104.51.98 'mount | grep /mnt/nfs/' — expect three lines
  2. If a line is missing: systemctl start m93p-nfs-recover.service (it remounts and restarts the CTs)
  3. pct list to confirm running; then check the service from the Hermes host
  4. FreshRSS specifically: curl -sS -m 15 -o /dev/null -w '%{http_code}' http://100.83.60.93/

Lessons

  • A Google Reader API Auth failed ... timed out error is an availability symptom first, an auth problem last. Probe the endpoint with curl before touching credentials.
  • The digest script needs up to 240s; a 180s timeout produced a spurious exit 124.
  • FreshRSS on m93p is HTTP only — https://freshrss.kangaroo-eel.ts.net fails because no Tailscale serve/funnel is configured for CT 1260.