[SOLVED] Stale NFS mounts - VMs still running - nervous about what to do

[siege801]

Hi all,

Basic run down of my configuration:

• 3 PVE nodes
• unRAID (NAS-101) presenting NFS shares to Proxmox for VM / LXC storage and dumps
• a secondary NAS (NAS-201) also presenting NFS shares. This NAS is typically only used to hold backups, or to migrate VMs if i need to do work on NAS-101.

A week or so ago, unRAID had an issue and had to be rebooted. I can only guess that this was the initial problem that's led to where I am. Now I have found that all of the PVEs report a stale mount for the vmstore (/mnt/pve/NAS-101_vstore).

root@luc-pve-301:/mnt/pve# file NAS-101_vstore
NAS-101_vstore: cannot open `NAS-101_vstore' (Stale file handle)

Normally, to fix a stale NFS mount, one simply unmounts and remounts, but I'm unsure what impact this will have on the VMs and LXCc. Is their current state held in a disk image that is at risk of being lost?

I can't even run a PVE-level backup because the source can't be found:

ERROR: Backup of VM 103 failed - unable to activate storage 'NAS-101_vstore' - directory '/mnt/pve/NAS-101_vstore' does not exist or is unreachable

I'd love some advice before I do anything stupid in an attempt to fix this.

Thanks in advance

 

[waltar]

Do a "umount -l /mnt/pve/NAS-101_vstore" and wait up to 2 min and normally the mount should come back by pve (see "df").
Only if not (!!) - otherwise mount manually, try to migrate first node empty, reboot that, migrate the second empty, reboot, and third ...

 

[siege801]

Sensational! That worked and no VMs or LXCs were harmed in the process. @waltar , you're my superhero for the day. Thank you!