After upgrade all my nodes can't see any Disk images & containers files which are mounted NFS shares

[tatric]

i have multiple NFS shares on multiple proxmox nodes which are there own network and which was all working great until tonight upgrade. After i apt upgraded all my nodes, they all can't see any Disk images & containers types of files which are mounted NFS shares. On those same NFS shares i can see Dumps and ISOs and container templates so the mounts are working but the type of mounts are not showing for the actual disks in the WUI under storage - content on each node.

After rebooting and recreating another NFS share mount, which work fine, its proxmox that can't seem to see the disk images and containers folders.

All VMs and containers fail on 4 nodes when i try to start them with this error message like this: TASK ERROR: volume 'zfsnfs:128/vm-128-disk-0.qcow2' does not exist.
The file is there and has the same permissions as the others i can see them all if i make a mount to my laptop.

One node is off and has not been updated tonight so it will be interesting if that still works tomorrow.

Has anyone else got the same issues?

PS. I have been using proxmox for about 6 years and fixed strange issues like this from time to time and I think that the product is auresome.

 

[oguz]

hi,

please post your /etc/pve/storage.cfg file contents.

also pvesm status

 

[tatric]

Thanks for showing interest. i have other NFS shares on one other node aswell but these are the main ones and i don't want it to get to complicated. I have added two little screen shots of the WUI to show what i mean about the rootdir not showing to Proxmox.

nfs: MicroserverNFS
    export /rpool/nfs
    path /mnt/pve/MicroserverNFS
    server 192.168.9.10
    content images,vztmpl,snippets,backup,rootdir,iso
    maxfiles 4
    nodes hpserver,Octobox,homemade,microserver,proxmox1
    options vers=3

nfs: HPzfs
    export /rpool/nfs
    path /mnt/pve/HPzfs
    server 192.168.9.5
    content rootdir,iso,snippets,backup,vztmpl,images
    maxfiles 4
    options vers=3

root@microserver:~# pvesm status
Name                  Type     Status           Total            Used       Available        %
HPzfs                  nfs     active      2663680000      1883664384       780015616   70.72%
MicroserverNFS         nfs     active      5842751488      3471839232      2370912256   59.42%

 

[tatric]

chmod -R 777 /rpool/nfs/images/128
a ha, disk images show up in the storage and that one virt machines disk ....
looks like the perms on all the image folders have been changed.
so whats the right perms ?
drwxr----- 2 root root 3 Jun 9 2018 127
drwxrwxrwx 2 root root 3 Aug 26 21:39 128
or zfs set sharenfs=on defaults have changed with the update? like root squash?

root@microserver:~# pvesm scan nfs microserver
/rpool/nfs *
the star means all network hosts

i tried to export with options set as follows, but is doesn't make any difference.
root@microserver:~# zfs get sharenfs rpool/nfs
NAME PROPERTY VALUE SOURCE
rpool/nfs sharenfs no_root_squash,rw=192.168.9.0/24 local

who is mounting the shares in proxmox? is it root?
ok also can't upload new iso files says that cp cmd failed because of Permission denied.

root@microserver:~# ls -la /rpool/nfs/
total 62
drwxr-xr-x 8 root root 8 Mar 25 2019 .
drwxr-xr-x 7 root root 7 Apr 12 2018 ..
drwxr-xr-x 4 root root 4 Apr 3 2018 DATA
drwxr-xr-x 3 root root 798 Jan 26 11:16 dump
drwxr-xr-x 16 root root 16 Oct 2 22:41 images
drwxr-xr-x 2 root root 2 Apr 5 2018 private
drwxr-xr-x 2 root root 2 Mar 25 2019 snippets
drwxr-xr-x 4 root root 4 Apr 5 2018 template

 

[oguz]

how did you create the NFS? via gui?

also

All VMs and containers fail on 4 nodes when i try to start them with this error message like this: TASK ERROR: volume 'zfsnfs:128/vm-128-disk-0.qcow2' does not exist.

zfsnfs is not in your storage config. is that the old name? you'll have to change the configs to point to the right storage (and move the volumes there if you haven't)

who is mounting the shares in proxmox? is it root?

yes

chmod -R 777 /rpool/nfs/images/128
a ha, disk images show up in the storage and that one virt machines disk ....
looks like the perms on all the image folders have been changed.

try 744

 

[tatric]

ZFS NFS sharenfs=on does NOT map root user to nobody by default #9397
https://github.com/zfsonlinux/zfs/issues/9397
https://github.com/zfsonlinux/zfs/pull/9425

and one of the above pages... To restore the previous behavior use 'zfs set sharenfs="no_root_squash,..."'.

This explains it along with your spot above, and your conformation that the mounts are made as root, maybe i was not changing the right share ! can't check now, i will have to check later and let you know.

 

[tatric]

This fixed my issues adding no_root_squash to all my shares and some of my hosts needed to be rebooted to pickup the new setting.
The IP address here is my storage network
zfs set sharenfs="no_root_squash,rw=@192.168.9.0/24"