In 7.0, I Can't backup a container to a NFS that worked in 6.0

[Lonnie]

I upgraded from Proxmox 6 to 7, by doing a clean install of Proxmox 7, mounting NFS shares, and then restoring all backups.

Now, I too am getting the same error talked about in this post:
https://forum.proxmox.com/threads/create-backup-fail-with-error-cannot-open-permission-de.32386/

INFO: including mount point rootfs ('/') in backup

INFO: creating vzdump archive '/mnt/pve/nfs-external2/dump/vzdump-lxc-104-2021_09_28-06_12_30.tar.zst'

INFO: tar: /mnt/pve/nfs-external2/dump/vzdump-lxc-104-2021_09_28-06_12_30.tmp: Cannot open: Permission denied

INFO: tar: Error is not recoverable: exiting now

ERROR: Backup of VM 104 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536

-- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-

include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--

directory=/mnt/pve/nfs-external2/dump/vzdump-lxc-104-2021_09_28-06_12_30.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--

directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var

/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/nfs-external2/dump/vzdump-lxc-104-2021_09_28-06_12_30.tar.dat' failed: exit

code 2

INFO: Failed at 2021-09-28 06:12:31

INFO: Backup job finished with errors

TASK ERROR: job errors

Note: Proxmox 6 had no issue reading and writing to this exact same NFS storage, but Proxmox 7 cannot write to it.


Others have mentioned the solution of changing permission on the mount point and dump folder. When I try to change the permission on these, at the Linux command prompt, they do NOT change!

The other NFS mounts have permissions drwxrwxr-x, but I cannot change this particular NFS mount point to have those same permissions; it stays drwx------ and chmod cannot modify the permission and chmod gives no error regarding the fact that it is not successfully changing those permissions.


I've tried removing and adding back this NFS storage using the Proxmox web interface. The issue remains.


One potentially weird thing is that this particular NFS storage's hard drive has a NTFS file system and is located on an external hard-drive that is connected to a separate non-proxmox-Linux-server hosting NFS, but Proxmox 6 had no issues writing backups to that. So, if Proxmox 7 is trying to set Linux-level file-permissions on a NFS shared NTFS file system, I can see how that would cause problems: NTFS doesn't have Linux permissions. Is that the difference between Proxmox 6 and 7? How do I get past this?


Please advise.

 

[Dunuin]

The problem shouldn't be PVE7. If you want to backup unprivileged LXCs the NFS share needs to grant UID 0 and 100000 full write access. I guess chmod wont help because your NFS server got its own right management and will revert the permissions of the shares root folder back to what is set in its config files.

 

[Lonnie]

@Dunuin Thanks for the reply.

I don't understand what is so special about privileged or unprivileged LXC containers, that they cannot be encapsulated into a file that gets transferred to a NFS share in the same manner as all other backups!

In my thinking, a backup is just a file that contains all that is needed to restore the container. How is this unlike any of the other Virtual Machines or Containers that are successfully being written to this NFS storage?

NFS is already granted with Proxmox nodes having full write privileges to this NTFS external hard drive that I'm attempting to write the backup to. "UID 0 and 100000" have no meaning to a NTFS file system.

 

[Dunuin]

I don't understand what is so special about privileged or unprivileged LXC containers, that they cannot be encapsulated into a file that gets transferred to a NFS share in the same manner as all other backups!

Because a unprivileged LXC isn't run as root because of the user-remapping. The root user (UID 0) inside the LXC is in reality a unprivileged user (UID 100000) on the host. If I understand it right the final archive file will be written by root (UID 0) to your backup storage but the files of your LXC will be first written to a temp folder as the LXCs root user (UID 100000) before they get zipped. If you don't tell PVE to use another vzdump temp directory it will create the temp folder on your backup storage. Thats why both UID 0 and 100000 need write access to it. If your NFS share got 700 rights only the owner of the share has access and NFS just won't allow your PVE to store stuff there.

NFS is already granted with Proxmox nodes having full write privileges to this NTFS external hard drive that I'm attempting to write the backup to. "UID 0 and 100000" have no meaning to a NTFS file system.

Is it a locally mounted NTFS partition or a NFS share? Your right that NTFS has no posix compatible rights but when you mount a NTFS partition to a linux system you tell linux how to handle the rights. If you mount that NTFS with the options "uid=0,gid=0,umask=700" it is owned by root and only root is allowed to use it. If you mount it as "uid=0,gid=0,umask=777" its still owned by root but everyone is allowed to use it. So rights and owners still matter even if you use NTFS.

 

[Lonnie]

Thank you. Your explanation, regarding this topic, is the best I've read so far. @Dunuin

Is it a locally mounted NTFS partition or a NFS share?

Both actually. This NTFS external drive is plugged into a (non-proxmox) Ubuntu 20.04 server.

Even though the drive is external, it is mounted via fstab with the following line:

/dev/disk/by-uuid/E5BC0CBACA0ABC05 /mnt/externalNFS ntfs-3g defaults,auto,uid=0,gid=0,umask=077 0 0

Then, in /etc/exports, this NFS entry permits access by IP:

/mnt/externalNFS/VMs    192.168.95.120(rw,sync,no_root_squash,no_subtree_check)

It is my understanding that this NFS share doesn't care at all about what internal Proxmox users may be writing to it; as long as the network packets are sourced from 192.168.95.120, permission to write is fully granted.

So, I don't think Promox would have any issue creating and writing to a temp directory that it creates (by default) onto this destination drive.

However, perhaps the issue is that this is a NTFS file system that is ultimately getting written to. If Proxmox is trying to set Linux file permissions onto the temp files it creates, on top of this NTFS file system, that's not possible (as far as I know). Could that be the root issue?

If so, is it possible to write these temp files to a tar container, instead of directly to the file system itself?

If that's possible, and efficient also, it seems that Proxmox could make its backups completely file system agnostic by writing directly to a tar container. I suspect you could indeed set Linux permission to each file placed directly into a tar container-destination and avoid ever having to deal with the permissions of whatever file system technology the NFS share is actually sharing at the partition level.

 

[Dunuin]

Even though the drive is external, it is mounted via fstab with the following line:

/dev/disk/by-uuid/E5BC0CBACA0ABC05 /mnt/externalNFS ntfs-3g defaults,auto,uid=0,gid=0,umask=077 0 0

Then, in /etc/exports, this NFS entry permits access by IP:

/mnt/externalNFS/VMs    192.168.95.120(rw,sync,no_root_squash,no_subtree_check)

It is my understanding that this NFS share doesn't care at all about what internal Proxmox users may be writing to it; as long as the network packets are sourced from 192.168.95.120, permission to write is fully granted.

So, I don't think Promox would have any issue creating and writing to a temp directory that it creates (by default) onto this destination drive.

However, perhaps the issue is that this is a NTFS file system that is ultimately getting written to. If Proxmox is trying to set Linux file permissions onto the temp files it creates, on top of this NTFS file system, that's not possible (as far as I know). Could that be the root issue?

Look like "umask" is inverted. So if you give a "umask=077" it will result in a "chmod 700". So "umask=000" should work for you because this will result in a "chmod 777". So right know with your "defaults,auto,uid=0,gid=0,umask=077" you tell linux that only root and no other user may use that NTFS partition. So UID 100000 has no rights to access it and backups wont work.
Normal case NFS should keep the users that want to access the share. So if you access NFS with UID 100000 it will try to access the NTFS partition as user UID 100000.

If so, is it possible to write these temp files to a tar container, instead of directly to the file system itself?

If that's possible, and efficient also, it seems that Proxmox could make its backups completely file system agnostic by writing directly to a tar container. I suspect you could indeed set Linux permission to each file placed directly into a tar container-destination and avoid ever having to deal with the permissions of whatever file system technology the NFS share is actually sharing at the partition level.

Thats basically what I told you to do as I said to edit the /etc/vzdump.conf and set the "tmpdir" to "/tmp" so that this temporarily backup folder is stored locally at /tmp (where everyone has full 777 access) so that vzdump only needs to write the final backup archive as UID 0 to your NFS share.