vzdump times out very often on zfs storage pool

[SRU]

Hello,
I am running vzdump from a bash shell against a zfs pool and get these messages:
"could not get storage information for 'storage-2': can't use storage type 'zfspool' for backup"
I need to loop up to 20 times with a 2 second wait between the iterations; somewhere in these 20 retries it works out.

Any hint to fix that ?

Here is the storage configuration:
root@pxmx-2:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content vztmpl,iso,backup

lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images

dir: pxmx-1-bck
path /mnt/data/backup
content backup,images
maxfiles 2
nodes pxmx-1
shared 1

dir: pxmx-1-hdd-img
path /mnt/data/hdd-img
content images,rootdir
nodes pxmx-1
shared 0

zfspool: storage-2
pool storage-2
content images,rootdir
mountpoint /storage-2
nodes pxmx-2

The pool consists of 5 Samsung EVO 860 SSDs:
root@pxmx-2:~# zpool status
pool: storage-2
state: ONLINE
scan: scrub repaired 0B in 0 days 00:04:04 with 0 errors on Sat Nov 14 10:18:51 2020
config:

NAME STATE READ WRITE CKSUM
storage-2 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
scsi-36c81f660ce7ca800274004d11356d925 ONLINE 0 0 0
scsi-36c81f660ce7ca800274004fe16134426 ONLINE 0 0 0
scsi-36c81f660ce7ca8002740051317542f54 ONLINE 0 0 0
scsi-36c81f660ce7ca8002740052b18b463f6 ONLINE 0 0 0
scsi-36c81f660ce7ca8002740054019fae5ff ONLINE 0 0 0

errors: No known data errors


Here are the package versions:
dpkg -l | cut -b 5-60 | grep ^pve
pve-cluster 6.2-1
pve-container 3.2-2
pve-docs 6.2-6
pve-edk2-firmware 2.20200531-1
pve-firewall 4.1-3
pve-firmware 3.1-3
pve-ha-manager 3.1-1
pve-i18n 2.2-2
pve-kernel-5.4 6.2-7
pve-kernel-5.4.34-1-pve 5.4.34-2
pve-kernel-5.4.65-1-pve 5.4.65-1
pve-kernel-helper 6.2-7
pve-lxc-syscalld 0.9.1-1
pve-manager 6.2-15
pve-qemu-kvm 5.1.0-6
pve-xtermjs 4.7.0-2
pve-zsync 2.0-3

best regards, SRU

 

[fiona]

Hi,
could you also share the vzdump command you use? Are you trying to use storage-2 as the backup target? PVE currently only supports VM and container images as contents of ZFS storages, but not backups.

 

[SRU]

Sure.

vzdump --quiet 1 --mailnotification failure --mailto some@mail.de $id 2>> /var/log/backup-error-log

root@pxmx-2:/etc# cat vzdump.conf
dumpdir: /storage-2/vzdump
storage: storage-2
mode: snapshot
maxfiles: 3
compress: zstd
zstd: 4

root@pxmx-2:~# zfs list
NAME                      USED  AVAIL     REFER  MOUNTPOINT
storage-2                4.12T  1.09T      425G  /storage-2
storage-2/data            170K  1.09T      170K  /storage-2/data
...
storage-2/vzdump         1.59T  1.09T     1.59T  /storage-2/vzdump

From where do you have the knowlegde that this is unsupported ?
If there is a special mode for backups, can we create a dataset of that type in zfs ?

We read https://pve.proxmox.com/wiki/Backup_and_Restore
and found:

Backup Storage​

Before a backup can run, a backup storage must be defined. Refer to the Storage documentation on how to add a storage. A backup storage must be a file level storage, as backups are stored as regular files.

ZFS is marked as file level storage here:
https://pve.proxmox.com/pve-docs/chapter-pvesm.html

Thus we considered this to be supported.

Thanks for the help.

 

[fiona]

Sadly the documentation is a bit contradictory here. ZFS is in principle file level, and the way PVE uses it is creating virtual block devices for VMs and file system datasets for containers. But the backup content type is not supported, see here (Table 8).

I'll take a closer look afterwards, but I'm already willing to bet that the reason the backup only works sometimes is because perl hashes are iterated in random order, so sometimes the dumpdir option of your configuration will take precedence over your storage option. Those two options are actually conflicting, but we currently only check that if they are both passed along via the command line, try:

vzdump <ID> --dumpdir /storage-2/vzdump --storage storage-2

The good news is that you can just comment out the storage option and it should always work. The bad news is that I'll have to write a patch now

 

[SRU]

Fabian,
thank you very much for looking into it and providing a solution.
I will observe if that fixes the issue for us.

I am sorry that this leads to the burden of writing a patch on your site - but we should take comfort from the fact that this will make the product even better .

Greetings, SRU