PBS backup problem with failing VM backups, CT backups ok

[greybeard]

My setup has PBS installed on an OMV (OpenMediaVault) server with a separate PVE server, both physical servers.
I can backup CT's using PBS with no problems, however when I try and perform a PBS backup of a VM it fails with the following messages:

INFO: starting new backup job: vzdump 121 --mode snapshot --storage pbs --remove 0 --node pve
INFO: Starting Backup of VM 121 (qemu)
INFO: Backup started at 2021-04-09 09:21:48
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: OPNsense
INFO: include disk 'virtio0' 'local-zfs:vm-121-disk-0' 8G
INFO: include disk 'efidisk0' 'local-zfs:vm-121-disk-1' 1M
INFO: creating Proxmox Backup Server archive 'vm/121/2021-04-09T01:21:48Z'
INFO: starting kvm to execute backup task
ERROR: VM 121 qmp command 'backup' failed - backup register image failed: command error: ENODEV: No such device
INFO: aborting backup job
INFO: stopping kvm after backup task
ERROR: Backup of VM 121 failed - VM 121 qmp command 'backup' failed - backup register image failed: command error: ENODEV: No such device
INFO: Failed at 2021-04-09 09:21:54
INFO: Backup job finished with errors
TASK ERROR: job errors

I can backup the VM with the original PVE backup to the same OMV server on a NFS share.
Both machines are running current software updates.

I have the same error for any of the VM's I have tried this with.

Notes:
Running ZFS file system on my PVE server
Running mergerFS on my OMV server
Observing the target directory on my OMV server during the PBS backup I can see the target folder being created then deleted a few seconds later.
The PBS server (on the OMV machine) is using a folder created on the mergerFS then configured via the PBS gui as a folder based data store.

I suspect that the issue is something to do with the mergerFS (supported by the underlying OMV) however the successful backups of the CT's on the same setup has me doubting this conclusion.
The PBS documentation (section 5.2) mentions potential filesystem constraints and suggests xfs, ext4 or zfs. Are the requirements different between CT's and VM backups?

 

[voarsh]

I suspect that the issue is something to do with the mergerFS (supported by the underlying OMV) however the successful backups of the CT's on the same setup has me doubting this conclusion.

I have the same thing. LXC is fine....
It would/should be same to specify a mounted disk/folder instead of MergerFS path - which I might try since backups can take 3+ hours for me. :/

 

[voarsh]

Yeah, I used paths that were not in use by MergerFS (e.g direct mounted paths) and PBS worked just fine.

 

[greybeard]

Just tried bypassing mergerfs as you suggested to using a direct mounted path and it works fine.
If anyone is interested you can edit /etc/proxmox-backup/datastore.cfg to alter the path of an existing datastore. BUT you need to make sure you have collected all of the various files saved by MergerFS to the one direct path ie not leave them spread over different drives which mergerfs will do.
This worked for me but no guarantees that something won't break or change in the future.

 

[lucky001]

Hi, It should work if you remove cache.files=off or use cache.files=partial in mergerfs mount option. I have PBS docker image on OMV VM with dataset on mergerfs pool.

If you don't already know that you have a special use case then just start with one of the following option sets.

You need mmap (used by rtorrent and many sqlite3 base software)​

allow_other,use_ino,cache.files=partial,dropcacheonclose=true,category.create=mfs

You don't need mmap​

allow_other,use_ino,cache.files=off,dropcacheonclose=true,category.create=mfs

Do not use cache.files=off or direct_io if you expect applications (such as rtorrent) to mmap files. Shared mmap is not currently supported in FUSE w/ direct_io enabled. Enabling dropcacheonclose is recommended when cache.files=partial|full|auto-full or direct_io=false.