Backup fails when (additional) container disk is on ZFS ZVOL

Y

yummiweb

Active Member
Sep 29, 2019
24
2
43
55
Proxmox VE 9.1.4
Proxmox Backup Server 4.10

(The problem did not occur under PVE 8 / PBS 3)

Object: Container
Disk 0: "root"
Disk 1: "mp0" (/mnt/db_backups)
Disk locations: ZFS ZVOL

Problem:
Backup to PBS fails with the following error:
ERROR: Backup of VM 164001 failed - command 'mount -o ro -t zfs ZFS_POOL/zvol/subvol-164001-disk-1@vzdump /mnt/vzsnap0//mnt/db_backups' failed: exit code 2
Click to expand...

(Note the double "/" in the path after vzsnap0)

Details:

vzdump 164001 --mode snapshot --storage PBS_LOCAL --remove 0 --notes-template '{{guestname}}' --node base16 --notification-mode notification-system
INFO: Starting Backup of VM 164001 (lxc)
INFO: status = running
INFO: CT Name: admv1
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/mnt/db_backups') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: suspend vm to make snapshot
INFO: create storage snapshot 'vzdump'
INFO: resume vm
INFO: guest is online again after <1 seconds
ERROR: Backup of VM 164001 failed - command 'mount -o ro -t zfs ZFS_POOL/zvol/subvol-164001-disk-1@vzdump /mnt/vzsnap0//mnt/db_backups' Failed: Exit code 2
Click to expand...

In the end, an undeleted snapshot "vzsnap" of the container remains.

Current workaround:

I moved "mp0" (/mnt/db_backups) from ZVOL to the file-based area of ZFS_POOL – the backup works from there for now.

regards Yummiweb
 
Impact said:
Please share pct config 164001.

Can you elaborate on this? Why/how did you use a ZVOL?
Click to expand...
The "workaround" unfortunately has a significant drawback: (PVE) snapshots of the container can no longer be created if one of the drives is not located on a ZVOL.

Which is also the reason for storing the container on a ZVOL – to create snapshots of the container.

What makes the situation truly problematic at the moment is this:

If you choose to use a ZVOL, you have no backups (until this issue is resolved).

If you choose to use a workaround instead of a ZVOL, you have no snapshots.

I would like to report this directly as a bug, but I can't find where to do so.
 
Last edited:
Please share pct config 164001.
Click to expand...

This is the configuration at the time of the error (already under PVE9) - both drives on the ZVOL.
arch: amd64
cores: 2
cpulimit: 1
features: nesting=1
hostname: myhost
memory: 2048
mp0: POOL_A_ZVOL:subvol-164001-disk-1,mp=/mnt/db_backups,backup=1,mountoptions=discard;lazytime,size=8G
nameserver: 192.168.222.1
net0: name=eth0,bridge=vmbr1621,gw=192.168.222.1,hwaddr=DE:49:22:22:02:01,ip=192.168.222.12/32,tag=12,type=veth
ostype: debian
rootfs: POOL_A_ZVOL:subvol-164001-disk-0,mountoptions=discard;lazytime,size=16G
searchdomain: mydomain.internal
startup: order=4,down=180
swap: 4096
unprivileged: 1
Click to expand...
 
I think we have different opinions on what a ZVOL is. Can you share cat /etc/pve/storage.cfg in a code block?
 
Last edited:
Impact said:
I think we have different opinions on what a ZVOL is. Can you share cat /etc/pve/storage.cfg in a code block?
Click to expand...
This is a ZVOL, right?
zfspool: POOL_A_ZVOL
pool ZFS_POOL_A/zvol
blocksize 16k
content images,rootdir
mountpoint /ZFS_POOL_A/zvol
sparse 1
Click to expand...
And that is a "normal" Directory Storage on ZFS:
dir: POOL_A_FILES
path /mnt/MNTNAME/zfs_poo_a
content rootdir,images
prune-backups keep-all=1
shared 0
Click to expand...

What are yor diffrent opinion?

Remeber my first Post, please:
The problem persist since PVE9, on PVE8 there was no Problem with Backups
 
I'd consider that a dataset which makes this pretty confusing. Please share zfs list -ospace,type. Only volume is a ZVOL.
Can you share findmnt -T /mnt/MNTNAME/zfs_poo_a and findmnt -T /mnt/db_backups too? I need to get a grasp of your setup before I can consider what might be wrong.
Please use code blocks when sharing formatted text so this formatting is preserved.
 
Last edited:
Impact said:
I'd consider that a dataset which makes this pretty confusing. Please share zfs list -ospace,type. Only volume is a ZVOL.
Can you share findmnt -T /mnt/MNTNAME/zfs_pool_a
Click to expand...

Please note that the container is no longer located on the ZVOL:

Code: 
zfs list -ospace,type | grep "ZFS_POOL_A/zvol"
ZFS_POOL_A/zvol                             444G   389G        0B     96K             0B       389G  filesystem
ZFS_POOL_A/zvol/subvol-163601-disk-0       12.6G  3.37G        0B   3.37G             0B         0B  filesystem
ZFS_POOL_A/zvol/subvol-163601-disk-1       13.1G  6.64G     3.71G   2.93G             0B         0B  filesystem
ZFS_POOL_A/zvol/subvol-165201-disk-0       15.0G  6.35G     5.31G   1.04G             0B         0B  filesystem
ZFS_POOL_A/zvol/vm-162501-disk-0            444G  5.05G     2.56G   2.49G             0B         0B  volume
ZFS_POOL_A/zvol/vm-162501-disk-1            444G  1.17G      691M    503M             0B         0B  volume
ZFS_POOL_A/zvol/vm-162501-disk-2            444G  5.52G     2.40G   3.12G             0B         0B  volume
ZFS_POOL_A/zvol/vm-162501-disk-3            444G  16.7G     16.2G    573M             0B         0B  volume
ZFS_POOL_A/zvol/vm-165301-disk-0            444G  38.4G     34.9G   3.55G             0B         0B  volume
ZFS_POOL_A/zvol/vm-165301-disk-1            444G  9.10G     94.7M   9.01G             0B         0B  volume
ZFS_POOL_A/zvol/vm-165302-disk-0            444G  4.84G     1.78G   3.06G             0B         0B  volume
ZFS_POOL_A/zvol/vm-165302-disk-1            444G  9.23G      356K   9.23G             0B         0B  volume
ZFS_POOL_A/zvol/vm-165501-disk-0            444G  8.23G     5.16G   3.08G             0B         0B  volume
ZFS_POOL_A/zvol/vm-165501-disk-1            444G  81.2G     38.5G   42.7G             0B         0B  volume
ZFS_POOL_A/zvol/vm-165501-disk-2            444G  8.74G     3.80G   4.94G             0B         0B  volume
ZFS_POOL_A/zvol/vm-167301-disk-0            444G  3.75G     2.37G   1.37G             0B         0B  volume
ZFS_POOL_A/zvol/vm-167301-disk-1            444G  40.6G     5.18G   35.4G             0B         0B  volume
ZFS_POOL_A/zvol/vm-167301-disk-2            444G  1.70G     1.01G    703M             0B         0B  volume
ZFS_POOL_A/zvol/vm-167801-disk-0            444G  67.3G     25.3G   42.0G             0B         0B  volume
ZFS_POOL_A/zvol/vm-167801-disk-1            444G  2.11G     1.83G    286M             0B         0B  volume
ZFS_POOL_A/zvol/vm-168101-disk-0            444G  2.86G     1.56G   1.30G             0B         0B  volume
ZFS_POOL_A/zvol/vm-168101-disk-1            444G   100M     33.3M   66.8M             0B         0B  volume
ZFS_POOL_A/zvol/vm-168101-disk-2            444G  66.1G     35.5G   30.6G             0B         0B  volume
ZFS_POOL_A/zvol/vm-168101-disk-3            444G  15.8M     9.44M   6.38M             0B         0B  volume

163601 and 165201 are additional containers (only the "root" volume). They are labeled "Filesystem" because their volumes operate on a file-based basis, not a file system-based one.

I don't know why the file mount should matter when it comes to a ZVOL (and the backup problem only exists on the ZVOL), but here you go:

Code: 
findmnt -T /mnt/MNTNAME/zfs_pool_a
TARGET                  SOURCE                    FSTYPE OPTIONS
/mnt/MNTNAME/zfs_pool_a ZFS_POOL_A/files      zfs    rw,noatime,xattr,posixacl,casesensitive
and findmnt -T /mnt/db_backups too?
Click to expand...

The directory `/mnt/db_backups` doesn't exist; there's only this mount on the pool for files:

dir: POOL_A_FILES
path /mnt/MNTNAME/zfs_pool_a (an `l` was missing here)
content rootdir,images
prune-backups keep-all=1
shared 0

I need to get a grasp of your setup before I can consider what might be wrong.
Click to expand...

I understand wanting to get an overview, but I don't understand why the filestore should matter, nor why the configuration should matter at all. This problem only started with PVE9.

I find the detail of the double slashes in the error message much more interesting:

Code: 
ERROR: Backup of VM 164001 failed - command 'mount -o ro -t zfs ZFS_POOL/zvol/subvol-164001-disk-1@vzdump /mnt/vzsnap0//mnt/db_backups' failed: exit code 2

"vzdump" here is the snapshot that is created before the backup and from which the backup is ultimately read.
"vzsnap0" will be the subfolder, which is created by the PVE container itself.
"/mnt/db_backups" ist the mount path in the container config for "mp0".

Since the root filesystem can be backed up from ZVOL (I had disabled the backup of "mp0"), it's apparently not due to the configuration itself, but rather to the way PVE9 interprets the internal "mount points" of the containers – and I would strongly suspect this is where the error lies.

`vzdump` is the snapshot that is created before the backup and from which the backup is ultimately read. "/mnt/vzsnap0//mnt/db_backups" is in no way logical, unless the error message parsing is incorrect.
 
Last edited:
You must log in or register to reply here.
Top Bottom