[SOLVED] After Proxmox VE 7 to 8 upgrade - NFS share enabled but not active.

[RogerMialkowski]

Hello all,

I have a small cluster of 3 machines with iSCSI and NFS storages. After upgrading one of the three (upgrade was successful), the PVE GUI put a question mark next to my NFS export. When I clicked on Summary for the NFS export under the node name, it said it was enabled but not active. When I clicked on VM Disks below that, it produced this error:

mount error: mount.nfs: mounting 10.100.0.245:/nfs/PVEIX2 failed, reason given by server: No such file or directory (500)

I decided to try to manually mount the export and when I did, it appears to have created a symlink and then I was able to see the storage in the PVE GUI and also see it by issuing a "df" via CLI. I rebooted that node and it still didn't automount it until I issued the mount command manually. Is anyone else having such a problem or is there a way to resolve it?

root@pve01:~# mount 10.100.0.245:/nfs/PVEIX2 /mnt/pve/PVEIX2/
Created symlink /run/systemd/system/remote-fs.target.wants/rpc-statd.service → /lib/systemd/system/rpc-statd.service.
root@pve01:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.2G 944K 3.2G 1% /run
/dev/mapper/pve-root 32G 5.1G 25G 18% /
tmpfs 16G 66M 16G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda2 1022M 344K 1022M 1% /boot/efi
/dev/fuse 128M 36K 128M 1% /etc/pve
tmpfs 3.2G 0 3.2G 0% /run/user/0
10.100.0.245:/nfs/PVEIX2 1.8T 585G 1.3T 33% /mnt/pve/PVEIX2
root@pve01:~#

Here is /var/log/syslog showing the errors and recovery once I manually mounted the NFS export. I'm using NFS v4 if that helps identify the issue.

2023-08-04T22:00:25.619396-04:00 pve01 pvestatd[888]: mount error: mount.nfs: mounting 10.100.0.245:/nfs/PVEIX2 failed, reason given by server: No such file or directory
2023-08-04T22:00:36.043329-04:00 pve01 pvestatd[888]: mount error: mount.nfs: mounting 10.100.0.245:/nfs/PVEIX2 failed, reason given by server: No such file or directory
2023-08-04T22:00:45.443286-04:00 pve01 pvestatd[888]: mount error: mount.nfs: mounting 10.100.0.245:/nfs/PVEIX2 failed, reason given by server: No such file or directory
2023-08-04T22:00:55.803374-04:00 pve01 pvestatd[888]: mount error: mount.nfs: mounting 10.100.0.245:/nfs/PVEIX2 failed, reason given by server: No such file or directory
2023-08-04T22:01:05.288753-04:00 pve01 systemd[1]: Reached target nss-lookup.target - Host and Network Name Lookups.
2023-08-04T22:01:05.334287-04:00 pve01 systemd[1]: Starting rpc-statd.service - NFS status monitor for NFSv2/3 locking....
2023-08-04T22:01:05.338480-04:00 pve01 rpc.statd[1906]: Version 2.6.2 starting
2023-08-04T22:01:05.338633-04:00 pve01 rpc.statd[1906]: Flags: TI-RPC
2023-08-04T22:01:05.338710-04:00 pve01 rpc.statd[1906]: Failed to read /var/lib/nfs/state: Success
2023-08-04T22:01:05.338754-04:00 pve01 rpc.statd[1906]: Initializing NSM state
2023-08-04T22:01:05.348446-04:00 pve01 systemd[1]: Started rpc-statd.service - NFS status monitor for NFSv2/3 locking..
2023-08-04T22:01:05.356649-04:00 pve01 systemd[1]: Reloading.
2023-08-04T22:01:05.817635-04:00 pve01 nfsrahead[1930]: setting /mnt/pve/PVEIX2 readahead to 128

Thanks,
Roger

 

[RogerMialkowski]

For my Iomega/Lenovo IX2-DL NFS backend, it turns out the path of the export is different depending on whether the NFS version is to 3 or 4. After manually mounting the NFS export in the previous email, today I did a "nfsstat -m" and found out that the export got mounted via version 3. The /etc/pve/storage.cfg was set to version 4. I then mounted the export using the "-o vers=4" and it couldn't find the export. I had to check in the NAS's config for the export names and it said for version 3 it's "/nfs/PVEIX2" while for version 4 it's "/PVEIX2". I then corrected my manual mount command accordingly and it mounted the export and "nfsstat -m" reports it as version 4. I updated the export path in /etc/pve/storage.cfg and I rebooted my 3 PVE nodes and the NFS storage was recognized and mount. That being said, I wonder if PVE 7.x handled NFS mount errors differently that PVE 8, whereas it would try the next lower version to see if the storage gets mounted then?