[SOLVED] Permission denied to mkdir ZFS shared over NFS between Proxmox nodes

[jsalas424]

I have added a node to a cluster. Both nodes are running PVE 6.4-8 and I am sharing my disks using ZFS over NFS. I have connected to the respective NFS shares and can see their disks on my node but apparently do not have the right permissions and I'm not certain what those would even entail.

When I try to backup the VM that I migrated, I get the following error

()
TASK ERROR: mkdir /Nextcloud.Storage/NC.VM.Backups/images/800: Permission denied at /usr/share/perl5/PVE/Storage/Plugin.pm line 728

When I try to delete:

TASK ERROR: unlink '/Nextcloud.Storage/NC.VM.Backups/dump/vzdump-qemu-800-2021_05_22-02_22_19.vma.zst' failed - Permission denied

Here's fstab on node:

# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
UUID=956D-B105 /boot/efi vfat defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

# NFS Share of new_ssd from Server
192.168.1.129:/new_ssd /new_ssd nfs defaults 0 0
192.168.1.129:/new_ssd/media /new_ssd/media nfs defaults 0 0
192.168.1.129:/Nextcloud.Storage /Nextcloud.Storage nfs defaults 0 0
192.168.1.129:/Storage.1 /Storage.1 nfs defaults 0 0

Expxorts that the node sees:

root@Server:~# showmount -e 192.168.1.129
Export list for 192.168.1.129:
/new_ssd/media                       *
/new_ssd/new_ssd.VMs                 *
/Nextcloud.Storage/NC.VM.Backups     *
/Nextcloud.Storage/User.Data.Backups *
/Nextcloud.Storage                   (everyone)
/Storage.1                           (everyone)
/new_ssd                             (everyone)
root@Server:~#

Here"s exports on host:

root@Server:~# cat /etc/exports
/new_ssd 192.168.0.0/16(rw,async,no_subtree_check,no_root_squash,insecure)
/Storage.1 192.168.1.24/32(rw,async,no_subtree_check,no_root_squash)
/Nextcloud.Storage 192.168.1.24/32(rw,async,no_subtree_check,no_root_squash)

I CAN backup to a parent folder NC.Disks.Dir whereas I can't backup to the subdirectory NC.VM.Backups, here's the config on the host:

dir: NC.VM.Backups.dir
        path /Nextcloud.Storage/NC.VM.Backups
        content vztmpl,rootdir,backup,images,iso,snippets
        is_mountpoint 1
        prune-backups keep-last=18
        shared 1

dir: NC.Disks.Dir
        path /Nextcloud.Storage
        content snippets,iso,images,backup,rootdir,vztmpl
        is_mountpoint 1
        prune-backups keep-last=10
        shared 1

 

[jsalas424]

Here's the reddit discussion about this: https://www.reddit.com/r/Proxmox/co...fs_shared_over_nfs_between/h12hz6a/?context=3

I tried to set unique fsid to see if that makes a difference, but it failed:

root@Server:~# cat /etc/exports
/new_ssd 192.168.0.0/16(rw,async,fsid=1,no_subtree_check,no_root_squash,insecure)
/Storage.1 192.168.1.24/32(rw,async,fsid=2,no_subtree_check,no_root_squash)
/Nextcloud.Storage 192.168.1.24/32(rw,async,fsid=3,no_subtree_check,no_root_squash)

root@Server:~# systemctl restart nfs-server

Attempt to erase:

TASK ERROR: unlink  '/Nextcloud.Storage/NC.VM.Backups/dump/vzdump-qemu-800-2021_05_23-02_21_45.vma.zst'  failed - Permission denied

Attempt to backup:

INFO: starting new backup job: vzdump 800 --compress zstd --mode snapshot --node TracheNodeA --storage NC.VM.Backups.dir --remove 0
ERROR: Backup of VM 800 failed - unable to create temporary directory '/Nextcloud.Storage/NC.VM.Backups/dump/vzdump-qemu-800-2021_06_08-16_02_13.tmp' at /usr/share/perl5/PVE/VZDump.pm line 842.
INFO: Failed at 2021-06-08 16:02:13
INFO: Backup job finished with errors
TASK ERROR: job errors

So I went digging in the logs:

Here's what I did:

root@TracheNodeA:~# touch /Nextcloud.Storage/NC.VM.Backups/touchtouch: cannot touch '/Nextcloud.Storage/NC.VM.Backups/touch': Permission denied

Here's what I saw in the logs of the NFS host:

Jun  8 16:22:13 TracheServ systemd[1]: Started Session 12836 of user root.
Jun  8 16:22:13 TracheServ systemd[1]: session-12836.scope: Succeeded.
Jun  8 16:22:13 TracheServ systemd[1]: Started Session 12837 of user root.
Jun  8 16:22:13 TracheServ systemd[1]: session-12837.scope: Succeeded.
Jun  8 16:22:13 TracheServ systemd[1]: Started Session 12838 of user root.
Jun  8 16:22:13 TracheServ systemd[1]: session-12838.scope: Succeeded.
Jun  8 16:22:14 TracheServ systemd[1]: Started Session 12839 of user root.
Jun  8 16:22:14 TracheServ systemd[1]: session-12839.scope: Succeeded.
Jun  8 16:22:14 TracheServ systemd[1]: Started Session 12840 of user root.
Jun  8 16:22:14 TracheServ systemd[1]: session-12840.scope: Succeeded.
Jun  8 16:22:15 TracheServ systemd[1]: Started Session 12841 of user root.
Jun  8 16:22:15 TracheServ systemd[1]: session-12841.scope: Succeeded.

 

[jsalas424]

Resolved, it was a typo.

I turned on debug logs with

rpcdebug -m nfsd all

ran mount -a on the node and then turned off the debug logs with

rpcdebug -m nfsd -c all

I saw and realized there was a typo

 rpc.mountd[41775]: refused mount request from 192.168.1.24 for /Nextcloud/Storage/User.Data.Backups (/): not exported