NFS Backups not working for all VMs

[thinkingfox]

I have an NFS mount, which I can write to when logged into the console as root and to which a number of VMs can back up to.

However, about half of the VMs fail to back up (although they have previously backed up to the same partition). The failure is about permissions, but that they haven't changed (and I have tried with chmod 777 and still the errors persist). Any pointers, please?

INFO: create storage snapshot 'vzdump'
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "snap_vm-113-disk-0_vzdump" created.
  WARNING: Sum of all thin volume sizes (203.00 GiB) exceeds the size of thin pool pve/data and the amount of free space in volume group (<16.00 GiB).
INFO: creating vzdump archive '/mnt/pve/vm-backups/dump/vzdump-lxc-113-2021_11_02-05_51_19.tar.zst'
INFO: tar: /mnt/pve/vm-backups/dump/vzdump-lxc-113-2021_11_02-05_51_19.tmp: Cannot open: Permission denied
INFO: tar: Error is not recoverable: exiting now
INFO: cleanup temporary 'vzdump' snapshot
  Logical volume "snap_vm-113-disk-0_vzdump" successfully removed
ERROR: Backup of VM 113 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/vm-backups/dump/vzdump-lxc-113-2021_11_02-05_51_19.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/vm-backups/dump/vzdump-lxc-113-2021_11_02-05_51_19.tar.dat' failed: exit code 2
INFO: Failed at 2021-11-02 05:51:20
INFO: Backup job finished with errors
TASK ERROR: job errors

 

[Dunuin]

I guess with "half of the VMs fail to back up" you actually mean VMs and LXCs. And you are using vzdump not PBS. Unprivileged LXCs use user remapping so you need to make sure the remapped root user of that unprivileged LXC has write access to the NFS share.
A easy solution would for example be to edit the /etc/vzdump.conf and set the "tmpdir" line to your local "/tmp" folder.

For more info you could search the forum. That is basically asked once per week.

 

[thinkingfox]

The LXC dumps were previously working without the tmpdir being set (as per my original request), guess there must have been a breaking change released at some point during one of the releases. tmpdir did fix.

@Dunuin how do we ask for this or one of the other answers to be made a sticky thread?

 

[Dunuin]

@Dunuin how do we ask for this or one of the other answers to be made a sticky thread?

I think most people wouldn't read sticky threads. But would be great if the staff members could write a FAQ with the top 100 most encounted problems somewhere, with html-anchors to each answer, or something like that. Would be way easier to post a link to a detailed answer instead of explaining the same problem again and again.