Whenever I try to run a snapshot/zstd backup job on a specific VM (ID 100) with multiple ZFS volumes, it fails on that VM and the other one. If I edit the job and only run it on VM 101, which also has two volumes, it works fine. I've already made sure vzsnap isn't mounted and deleted any dangling vzdump zfs snapshots. I've also rebooted the VMs and the hostnode, but the error remains the same. Taking a snapshot manually works fine.
Failing backup (both VMs):
Successful backup (only VM 101):
Versions, clean target, and VM config files:
Failing backup (both VMs):
Code:
INFO: starting new backup job: vzdump --fleecing 0 --storage local --notification-mode notification-system --compress zstd --notes-template '{{guestname}}' --all 1 --node proxmox --mode snapshot
INFO: filesystem type on dumpdir is 'zfs' -using /var/tmp/vzdumptmp22405_100 for temporary files
INFO: Starting Backup of VM 100 (lxc)
INFO: Backup started at 2025-10-21 07:04:31
INFO: status = running
INFO: CT Name: game
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/var/lib/mysql') in backup
INFO: including mount point mp1 ('/var/backups') in backup
INFO: excluding volume mount point mp2 ('/mnt_tmp') from backup (disabled)
INFO: found old vzdump snapshot (force removal)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: suspend vm to make snapshot
INFO: create storage snapshot 'vzdump'
zfs_mount_at() failed: directory is not emptyumount: /mnt/vzsnap0/var/lib/mysql: not mounted.
command 'umount -l -d /mnt/vzsnap0/var/lib/mysql' failed: exit code 32
INFO: resume vm
INFO: guest is online again after <1 seconds
ERROR: Backup of VM 100 failed - command 'mount -o ro -t zfs data/subvol-100-disk-2@vzdump /mnt/vzsnap0//var/lib/mysql' failed: exit code 2
INFO: Failed at 2025-10-21 07:04:31
INFO: filesystem type on dumpdir is 'zfs' -using /var/tmp/vzdumptmp22405_101 for temporary files
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2025-10-21 07:04:31
INFO: status = running
INFO: CT Name: web
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/test') in backup
ERROR: Backup of VM 101 failed - mount point '/mnt/vzsnap0' not empty
INFO: Failed at 2025-10-21 07:04:31
INFO: Backup job finished with errors
WARN: 'root@pam' does not have a configured email address in the user configuration - not sending an email to this user
INFO: notified via target `mail-to-root`
TASK ERROR: job errors
Successful backup (only VM 101):
Code:
INFO: starting new backup job: vzdump 101 --node proxmox --mode snapshot --all 0 --notes-template '{{guestname}}' --compress zstd --notification-mode notification-system --fleecing 0 --storage local
INFO: filesystem type on dumpdir is 'zfs' -using /var/tmp/vzdumptmp136990_101 for temporary files
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2025-10-21 07:09:25
INFO: status = running
INFO: CT Name: web
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/test') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: suspend vm to make snapshot
INFO: create storage snapshot 'vzdump'
INFO: resume vm
INFO: guest is online again after <1 seconds
INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-lxc-101-2025_10_21-07_09_25.tar.zst'
INFO: Total bytes written: 46079897600 (43GiB, 324MiB/s)
INFO: archive file size: 35.63GB
INFO: adding notes to backup
INFO: cleanup temporary 'vzdump' snapshot
INFO: Finished Backup of VM 101 (00:02:21)
INFO: Backup finished at 2025-10-21 07:11:46
INFO: Backup job finished successfully
WARN: 'root@pam' does not have a configured email address in the user configuration - not sending an email to this user
INFO: notified via target `mail-to-root`
TASK OK
Versions, clean target, and VM config files:
Code:
root@proxmox:~# mount | grep vzsnap
root@proxmox:~# zfs list -t snapshot
no datasets available
root@proxmox:~# cat /etc/pve/lxc/100.conf
arch: amd64
cores: 12
cpuunits: 10000
features: nesting=1
hostname: game
memory: 32768
mp0: vms:subvol-100-disk-2,mp=/var/lib/mysql,backup=1,size=75G
mp1: vms:subvol-100-disk-3,mp=/var/backups,backup=1,size=50G
mp2: vms:subvol-100-disk-4,mp=/mnt_tmp,size=100G
onboot: 1
ostype: ubuntu
protection: 1
rootfs: vms:subvol-100-disk-1,size=25G
startup: order=1
swap: 32768
unprivileged: 1
root@proxmox:~# cat /etc/pve/lxc/101.conf
arch: amd64
cores: 8
cpuunits: 1000
features: nesting=1
hostname: web
memory: 16384
mp0: vms:subvol-101-disk-1,mp=/test,backup=1,size=8G
onboot: 1
ostype: ubuntu
rootfs: vms:subvol-101-disk-0,size=100G
swap: 16384
unprivileged: 1
lxc.cgroup2.cpuset.cpus: 0,1,4,5,6,7,10,11
root@proxmox:~# pveversion -v
proxmox-ve: 9.0.0 (running kernel: 6.14.11-4-pve)
pve-manager: 9.0.11 (running version: 9.0.11/3bf5476b8a4699e2)
proxmox-kernel-helper: 9.0.4
proxmox-kernel-6.14.11-4-pve-signed: 6.14.11-4
proxmox-kernel-6.14: 6.14.11-4
amd64-microcode: 3.20250311.1
ceph-fuse: 19.2.3-pve2
corosync: 3.1.9-pve2
criu: 4.1.1-1
frr-pythontools: 10.3.1-1+pve4
ifupdown2: 3.3.0-1+pmx10
intel-microcode: 3.20250512.1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.0
libproxmox-backup-qemu0: 2.0.1
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.3
libpve-apiclient-perl: 3.4.0
libpve-cluster-api-perl: 9.0.6
libpve-cluster-perl: 9.0.6
libpve-common-perl: 9.0.11
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.4
libpve-network-perl: 1.1.8
libpve-rs-perl: 0.10.10
libpve-storage-perl: 9.0.13
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 6.0.5-1
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
proxmox-backup-client: 4.0.16-1
proxmox-backup-file-restore: 4.0.16-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.0
proxmox-kernel-helper: 9.0.4
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.2
proxmox-widget-toolkit: 5.0.6
pve-cluster: 9.0.6
pve-container: 6.0.13
pve-docs: 9.0.8
pve-edk2-firmware: not correctly installed
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.3
pve-firmware: 3.17-2
pve-ha-manager: 5.0.5
pve-i18n: 3.6.1
pve-qemu-kvm: 10.0.2-4
pve-xtermjs: 5.5.0-2
qemu-server: 9.0.23
smartmontools: 7.4-pve1
spiceterm: 3.4.1
swtpm: 0.8.0+pve2
vncterm: 1.9.1
zfsutils-linux: 2.3.4-pve1
root@proxmox:~#
Last edited: