[SOLVED] Permission denied to mkdir ZFS shared over NFS between Proxmox nodes

jsalas424

Member
Jul 5, 2020
141
2
23
34
I have added a node to a cluster. Both nodes are running PVE 6.4-8 and I am sharing my disks using ZFS over NFS. I have connected to the respective NFS shares and can see their disks on my node but apparently do not have the right permissions and I'm not certain what those would even entail.

When I try to backup the VM that I migrated, I get the following error
Code:
()
TASK ERROR: mkdir /Nextcloud.Storage/NC.VM.Backups/images/800: Permission denied at /usr/share/perl5/PVE/Storage/Plugin.pm line 728

When I try to delete:
Code:
TASK ERROR: unlink '/Nextcloud.Storage/NC.VM.Backups/dump/vzdump-qemu-800-2021_05_22-02_22_19.vma.zst' failed - Permission denied

Here's fstab on node:
Code:
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
UUID=956D-B105 /boot/efi vfat defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

# NFS Share of new_ssd from Server
192.168.1.129:/new_ssd /new_ssd nfs defaults 0 0
192.168.1.129:/new_ssd/media /new_ssd/media nfs defaults 0 0
192.168.1.129:/Nextcloud.Storage /Nextcloud.Storage nfs defaults 0 0
192.168.1.129:/Storage.1 /Storage.1 nfs defaults 0 0

Expxorts that the node sees:
Code:
root@Server:~# showmount -e 192.168.1.129
Export list for 192.168.1.129:
/new_ssd/media                       *
/new_ssd/new_ssd.VMs                 *
/Nextcloud.Storage/NC.VM.Backups     *
/Nextcloud.Storage/User.Data.Backups *
/Nextcloud.Storage                   (everyone)
/Storage.1                           (everyone)
/new_ssd                             (everyone)
root@Server:~#

Here"s exports on host:
Code:
root@Server:~# cat /etc/exports
/new_ssd 192.168.0.0/16(rw,async,no_subtree_check,no_root_squash,insecure)
/Storage.1 192.168.1.24/32(rw,async,no_subtree_check,no_root_squash)
/Nextcloud.Storage 192.168.1.24/32(rw,async,no_subtree_check,no_root_squash)

I CAN backup to a parent folder NC.Disks.Dir whereas I can't backup to the subdirectory NC.VM.Backups, here's the config on the host:
Code:
dir: NC.VM.Backups.dir
        path /Nextcloud.Storage/NC.VM.Backups
        content vztmpl,rootdir,backup,images,iso,snippets
        is_mountpoint 1
        prune-backups keep-last=18
        shared 1

dir: NC.Disks.Dir
        path /Nextcloud.Storage
        content snippets,iso,images,backup,rootdir,vztmpl
        is_mountpoint 1
        prune-backups keep-last=10
        shared 1
 
Last edited:
Here's the reddit discussion about this: https://www.reddit.com/r/Proxmox/co...fs_shared_over_nfs_between/h12hz6a/?context=3

I tried to set unique fsid to see if that makes a difference, but it failed:
Code:
root@Server:~# cat /etc/exports
/new_ssd 192.168.0.0/16(rw,async,fsid=1,no_subtree_check,no_root_squash,insecure)
/Storage.1 192.168.1.24/32(rw,async,fsid=2,no_subtree_check,no_root_squash)
/Nextcloud.Storage 192.168.1.24/32(rw,async,fsid=3,no_subtree_check,no_root_squash)

root@Server:~# systemctl restart nfs-server

Attempt to erase:

Code:
TASK ERROR: unlink  '/Nextcloud.Storage/NC.VM.Backups/dump/vzdump-qemu-800-2021_05_23-02_21_45.vma.zst'  failed - Permission denied

Attempt to backup:

Code:
INFO: starting new backup job: vzdump 800 --compress zstd --mode snapshot --node TracheNodeA --storage NC.VM.Backups.dir --remove 0
ERROR: Backup of VM 800 failed - unable to create temporary directory '/Nextcloud.Storage/NC.VM.Backups/dump/vzdump-qemu-800-2021_06_08-16_02_13.tmp' at /usr/share/perl5/PVE/VZDump.pm line 842.
INFO: Failed at 2021-06-08 16:02:13
INFO: Backup job finished with errors
TASK ERROR: job errors

So I went digging in the logs:

Here's what I did:

Code:
root@TracheNodeA:~# touch /Nextcloud.Storage/NC.VM.Backups/touchtouch: cannot touch '/Nextcloud.Storage/NC.VM.Backups/touch': Permission denied

Here's what I saw in the logs of the NFS host:

Code:
Jun  8 16:22:13 TracheServ systemd[1]: Started Session 12836 of user root.
Jun  8 16:22:13 TracheServ systemd[1]: session-12836.scope: Succeeded.
Jun  8 16:22:13 TracheServ systemd[1]: Started Session 12837 of user root.
Jun  8 16:22:13 TracheServ systemd[1]: session-12837.scope: Succeeded.
Jun  8 16:22:13 TracheServ systemd[1]: Started Session 12838 of user root.
Jun  8 16:22:13 TracheServ systemd[1]: session-12838.scope: Succeeded.
Jun  8 16:22:14 TracheServ systemd[1]: Started Session 12839 of user root.
Jun  8 16:22:14 TracheServ systemd[1]: session-12839.scope: Succeeded.
Jun  8 16:22:14 TracheServ systemd[1]: Started Session 12840 of user root.
Jun  8 16:22:14 TracheServ systemd[1]: session-12840.scope: Succeeded.
Jun  8 16:22:15 TracheServ systemd[1]: Started Session 12841 of user root.
Jun  8 16:22:15 TracheServ systemd[1]: session-12841.scope: Succeeded.
 
Resolved, it was a typo.

I turned on debug logs with
Code:
rpcdebug -m nfsd all

ran mount -a on the node and then turned off the debug logs with
Code:
rpcdebug -m nfsd -c all

I saw and realized there was a typo
Code:
 rpc.mountd[41775]: refused mount request from 192.168.1.24 for /Nextcloud/Storage/User.Data.Backups (/): not exported
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!