VM backup to CIFS share on TrueNAS VM fails at 100% completion

kernkraftzwerg

New Member
Apr 13, 2021
4
0
1
31
Hi folks,

I did set up a TrueNAS Core VM (version: TrueNAS-12.0-U2.1, FreeBSD 12.2-RELEASE-p3) with 2TB SSD passthrough from the Proxmox host (version: 6.3-6). Inside this VM I created a CIFS share in order to back up other VMs to. To test its functions I mounted the share on Linux and could successfully read, write, rename and delete files on it. Also I successfully added the CIFS share to my Proxmox datacenter, the folders 'dump' and 'template' were automatically created.
However, a backup of a VM (pfSense, in this case) to this share fails at 100% completion with the following message:

Code:
INFO: starting new backup job: vzdump 552 --node neodym --storage TrueNAS-VM --remove 0 --mode snapshot --compress zstd
INFO: Starting Backup of VM 552 (qemu)
INFO: Backup started at 2021-04-13 14:31:11
INFO: status = running
INFO: VM Name: pfSense
INFO: include disk 'virtio0' 'local-lvm:vm-552-disk-0' 16G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating vzdump archive '/mnt/pve/TrueNAS-VM/dump/vzdump-qemu-552-2021_04_13-14_31_11.vma.zst'
INFO: started backup task '617eafd4-a192-4fcf-a8c0-3f561b6e16a0'
INFO: resuming VM again
INFO:  34% (5.5 GiB of 16.0 GiB) in 3s, read: 1.8 GiB/s, write: 6.5 MiB/s
INFO:  59% (9.5 GiB of 16.0 GiB) in 6s, read: 1.3 GiB/s, write: 620.0 KiB/s
INFO:  63% (10.2 GiB of 16.0 GiB) in 9s, read: 230.1 MiB/s, write: 109.9 MiB/s
INFO:  67% (10.7 GiB of 16.0 GiB) in 12s, read: 195.1 MiB/s, write: 100.2 MiB/s
INFO:  70% (11.3 GiB of 16.0 GiB) in 15s, read: 203.1 MiB/s, write: 102.4 MiB/s
INFO:  76% (12.2 GiB of 16.0 GiB) in 18s, read: 311.4 MiB/s, write: 92.7 MiB/s
INFO:  88% (14.1 GiB of 16.0 GiB) in 21s, read: 651.6 MiB/s, write: 78.1 MiB/s
INFO: 100% (16.0 GiB of 16.0 GiB) in 23s, read: 952.9 MiB/s, write: 18.9 MiB/s
INFO: backup is sparse: 14.53 GiB (90%) total zero data
INFO: transferred 16.00 GiB in 23 seconds (712.3 MiB/s)
cannot unlink file for /mnt/pve/TrueNAS-VM/dump/vzdump-qemu-552-2021_04_13-14_31_11.tmp/qemu-server.conf: Permission denied at /usr/share/perl5/PVE/VZDump.pm line 1057.
cannot restore permissions to 0100755 for /mnt/pve/TrueNAS-VM/dump/vzdump-qemu-552-2021_04_13-14_31_11.tmp/qemu-server.conf:  at /usr/share/perl5/PVE/VZDump.pm line 1057.
cannot remove directory for /mnt/pve/TrueNAS-VM/dump/vzdump-qemu-552-2021_04_13-14_31_11.tmp: Permission denied at /usr/share/perl5/PVE/VZDump.pm line 1057.
ERROR: Backup of VM 552 failed - unable to rename '/mnt/pve/TrueNAS-VM/dump/vzdump-qemu-552-2021_04_13-14_31_11.vma.dat' to '/mnt/pve/TrueNAS-VM/dump/vzdump-qemu-552-2021_04_13-14_31_11.vma.zst'
INFO: Failed at 2021-04-13 14:31:34
INFO: Backup job finished with errors
TASK ERROR: job errors

Inside the share files were created (but not shown inside the Proxmox GUI). The TrueNAS CLI shows them as follows:

Code:
root@truenas[/mnt/Crucial2000GB/proxmox/dump]# ls -alh
total 708859
drwxrwx---+ 3 proxmox  wheel       5B Apr 13 05:31 .
drwxrwx---+ 4 root     wheel       4B Apr 13 03:31 ..
-rwxrwx---+ 1 proxmox  wheel     1.7K Apr 13 05:31 vzdump-qemu-552-2021_04_13-14_31_11.log
drwxrwx---+ 2 proxmox  wheel       3B Apr 13 05:31 vzdump-qemu-552-2021_04_13-14_31_11.tmp
-rwxrwx---+ 1 proxmox  wheel     692M Apr 13 05:31 vzdump-qemu-552-2021_04_13-14_31_11.vma.dat

To narrow down the problem I created another NFS share (on another ZFS dataset called 'prox-nfs') in the TrueNAS VM and added it to the Proxmox datacenter.
A Backup to this target does not fail:

Code:
INFO: starting new backup job: vzdump 552 --remove 0 --storage TrueNAS-VM-NFS --mode snapshot --compress zstd --node neodym
INFO: Starting Backup of VM 552 (qemu)
INFO: Backup started at 2021-04-13 14:54:07
INFO: status = running
INFO: VM Name: pfSense
INFO: include disk 'virtio0' 'local-lvm:vm-552-disk-0' 16G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating vzdump archive '/mnt/pve/TrueNAS-VM-NFS/dump/vzdump-qemu-552-2021_04_13-14_54_07.vma.zst'
INFO: started backup task '5fa6494b-d25b-4aec-9c77-40f66e2ccf84'
INFO: resuming VM again
INFO:  34% (5.5 GiB of 16.0 GiB) in 3s, read: 1.8 GiB/s, write: 6.4 MiB/s
INFO:  62% (9.9 GiB of 16.0 GiB) in 6s, read: 1.5 GiB/s, write: 41.2 MiB/s
INFO:  65% (10.5 GiB of 16.0 GiB) in 9s, read: 206.6 MiB/s, write: 111.2 MiB/s
INFO:  69% (11.1 GiB of 16.0 GiB) in 12s, read: 201.7 MiB/s, write: 101.0 MiB/s
INFO:  73% (11.7 GiB of 16.0 GiB) in 15s, read: 197.1 MiB/s, write: 101.2 MiB/s
INFO:  77% (12.4 GiB of 16.0 GiB) in 18s, read: 244.1 MiB/s, write: 106.2 MiB/s
INFO: 100% (16.0 GiB of 16.0 GiB) in 21s, read: 1.2 GiB/s, write: 35.6 MiB/s
INFO: backup is sparse: 14.53 GiB (90%) total zero data
INFO: transferred 16.00 GiB in 21 seconds (780.2 MiB/s)
INFO: archive file size: 691MB
INFO: Finished Backup of VM 552 (00:00:21)
INFO: Backup finished at 2021-04-13 14:54:28
INFO: Backup job finished successfully
TASK OK

Code:
root@truenas[/mnt/Crucial2000GB/prox-nfs/dump]# ls -alh
total 708870
drwxrwx---+ 2 proxmox  wheel     4B Apr 13 05:54 .
drwxrwx---+ 4 root     wheel     4B Apr 13 05:26 ..
-rw-r--r--+ 1 proxmox  wheel   1.6K Apr 13 05:54 vzdump-qemu-552-2021_04_13-14_54_07.log
-rwxrwx---+ 1 proxmox  wheel   692M Apr 13 05:54 vzdump-qemu-552-2021_04_13-14_54_07.vma.zst

For both datasets I used the (as far as I can tell) suitable ACL parameters:
  • user account in TrueNAS -> user: proxmox (with group: proxmox)
  • share type CIFS dataset: SMB -> ACL mode: Restricted, case sensitivity: Insensistive
  • share type NFS dataset: generic -> ACL mode: Passthrough, case sensitivity: Sensistive
  • ACL settings of both datasets: user root, group wheel & group proxmox are allowed to do everything (except delete dataset & child dataset)
  • NFS share: 'mapall user: proxmox' & 'mapall group: proxmox'
  • in both shares the Proxmox host's IP is the only 'authorized host'
A web search brought me to this thread from 2015, which discusses a lack of storage as a possible point of failure - however in my case I want to backup a 16GB VM (which is 90% empty) to a 1.76TB share (which is completely empty).

Since the NFS share works an an backup target, I can use it for the Proxmox host. However I have to create some CIFS shares and map them to a Windows VM, and the error message from the Proxmox backup utility indicates that I misconfigured the share. Of course I want to avoid data loss due to this.

Any help and insight in this matter is apreciated ;)

[This post is an english translation from this thread from the german forum, as per advised here]
 
Hi,

does a single backup to CIFS with specified --tmpdir work?
Code:
vzdump 552 --remove 0 --storage TrueNAS-VM-NFS --mode snapshot --compress zstd --node neodym --tmpdir /some/local/directory
 
Hi,

thanks for your time.
Since the problematic storage is 'TrueNAS-VM', not 'TrueNAS-VM-NFS' (which works fine), I did
root@neodym:/mnt/temp# vzdump 552 --remove 0 --storage TrueNAS-VM --mode snapshot --compress zstd --node neodym --tmpdir /mnt/temp/test/
(where /mnt/temp is an external USB-HDD)

Code:
INFO: starting new backup job: vzdump 552 --storage TrueNAS-VM --remove 0 --compress zstd --mode snapshot --node neodym --tmpdir /mnt/temp/test/
INFO: Starting Backup of VM 552 (qemu)
INFO: Backup started at 2021-04-19 15:57:11
INFO: status = running
INFO: VM Name: pfSense
INFO: include disk 'virtio0' 'local-lvm:vm-552-disk-0' 16G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating vzdump archive '/mnt/pve/TrueNAS-VM/dump/vzdump-qemu-552-2021_04_19-15_57_11.vma.zst'
INFO: started backup task '7c65855e-15a6-429f-9d12-029d77ee1f0c'
INFO: resuming VM again
INFO:  31% (5.0 GiB of 16.0 GiB) in 3s, read: 1.7 GiB/s, write: 10.0 MiB/s
INFO:  61% (9.9 GiB of 16.0 GiB) in 6s, read: 1.6 GiB/s, write: 19.6 MiB/s
INFO:  63% (10.2 GiB of 16.0 GiB) in 9s, read: 121.3 MiB/s, write: 94.4 MiB/s
INFO:  66% (10.7 GiB of 16.0 GiB) in 12s, read: 160.6 MiB/s, write: 91.2 MiB/s
INFO:  70% (11.3 GiB of 16.0 GiB) in 15s, read: 195.4 MiB/s, write: 95.0 MiB/s
INFO:  73% (11.8 GiB of 16.0 GiB) in 18s, read: 187.8 MiB/s, write: 92.3 MiB/s
INFO:  84% (13.5 GiB of 16.0 GiB) in 21s, read: 569.8 MiB/s, write: 75.7 MiB/s
INFO: 100% (16.0 GiB of 16.0 GiB) in 24s, read: 854.0 MiB/s, write: 28.7 MiB/s
INFO: backup is sparse: 14.52 GiB (90%) total zero data
INFO: transferred 16.00 GiB in 24 seconds (682.7 MiB/s)
ERROR: Backup of VM 552 failed - unable to rename '/mnt/pve/TrueNAS-VM/dump/vzdump-qemu-552-2021_04_19-15_57_11.vma.dat' to '/mnt/pve/TrueNAS-VM/dump/vzdump-qemu-552-2021_04_19-15_57_11.vma.zst'
INFO: Failed at 2021-04-19 15:57:35
INFO: Backup job finished with errors
job errors

I suspect the error has something to do with missing rename/delete rights in the storage folder /mnt/pve/TrueNAS-VM/ (due to my missconfiguration in TrueNAS).

I'll post when I've found the wrongly set settings.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!