Problem backing up container

[struppu]

Yesterday I updated Proxmox from 8 to 9. Last night I encauntered errors backing up one of my Containers to a Proxmox Backup Server

Error Message:

NFO: starting new backup job: vzdump 101 --all 0 --prune-backups 'keep-daily=7,keep-monthly=12,keep-weekly=4,keep-yearly=999' --mode snapshot --storage duckpbs --notes-template '{{guestname}}' --node duckmox --fleecing 0 --notification-mode notification-system
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2025-12-29 12:59:10
INFO: status = running
INFO: CT Name: duckcloud
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/srv') in backup
INFO: found old vzdump snapshot (force removal)
  Logical volume "snap_vm-101-disk-0_vzdump" successfully removed.
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: suspend vm to make snapshot
INFO: create storage snapshot 'vzdump'
  Logical volume "snap_vm-101-disk-0_vzdump" created.
zfs_mount_at() failed: directory is not emptyumount: /mnt/vzsnap0/srv: not mounted.
command 'umount -l -d /mnt/vzsnap0/srv' failed: exit code 32
INFO: resume vm
INFO: guest is online again after <1 seconds
ERROR: Backup of VM 101 failed - command 'mount -o ro -t zfs zfs_storage/subvol-101-disk-0@vzdump /mnt/vzsnap0//srv' failed: exit code 2
INFO: Failed at 2025-12-29 12:59:10
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

Backups of the other containers work as expected it's just this one container that does not work, even though two more containers also have a mountpoint on the zfs storage

The container itself works just normal, only backing up is not possible

Anyone got an idea what the problem might be, because for me the error message does not make sense
Thanks in advance

 

[powersupport]

This error is caused by a leftover mount directory from a failed snapshot, which started happening after the Proxmox 9 upgrade. The key message is:

zfs_mount_at() failed: directory is not empty

This means Proxmox is trying to mount the snapshot, but the mount directory already contains files from a previous failed backup.

To Fix this error​

1. Stop the container

pct stop 101

1. Check and clean leftover snapshot mounts

ls -lah /mnt/vzsnap0


If you see directories like srv or old data inside:

umount -lf /mnt/vzsnap0/srv 2>/dev/null
rm -rf /mnt/vzsnap0/srv

1. Remove any leftover ZFS snapshots

zfs list -t snapshot | grep subvol-101
zfs destroy zfs_storage/subvol-101-disk-0@vzdump

1. Start the container again

pct start 101

1. Run the backup manually to test

vzdump 101 --mode snapshot --storage duckpbs

 

[struppu]

thank you for your reply

I tried your suggestions:
after stopping the container
ls -lah /mnt/vzsnap0
showed me
root@duckmox:/# ls -lah /mnt/vzsnap0/
total 8.0K
drwxr-xr-x 2 root root 4.0K Dec 27 21:54 .
drwxr-xr-x 3 root root 4.0K Dec 27 21:54 .

so i could neither delete or unmount anything in that folder.

zfs list -t snapshot | grep subvol-101
outputs
zfs_storage/subvol-101-disk-0@vzdump 384K - 10.3G -

after destroying the snapshot with:
zfs destroy zfs_storage/subvol-101-disk-0@vzdump
the output of
zfs list -t snapshot | grep subvol-101
is
no datasets available

after starting the container manually running the backup gives the following output.
I run it twice because in after the first run I got an zfs error
zfs error: could not find any snapshots to destroy; check snapshot names.

vzdump 101 --mode snapshot --storage duckpbs

INFO: starting new backup job: vzdump 101 --storage duckpbs --mode snapshot
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2025-12-29 16:40:03
INFO: status = running
INFO: CT Name: duckcloud
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/srv') in backup
INFO: found old vzdump snapshot (force removal)
  Logical volume "snap_vm-101-disk-0_vzdump" successfully removed.
zfs error: could not find any snapshots to destroy; check snapshot names.
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: suspend vm to make snapshot
INFO: create storage snapshot 'vzdump'
  Logical volume "snap_vm-101-disk-0_vzdump" created.
zfs_mount_at() failed: directory is not emptyumount: /mnt/vzsnap0/srv: not mounted.
command 'umount -l -d /mnt/vzsnap0/srv' failed: exit code 32
INFO: resume vm
INFO: guest is online again after 7 seconds
ERROR: Backup of VM 101 failed - command 'mount -o ro -t zfs zfs_storage/subvol-101-disk-0@vzdump /mnt/vzsnap0//srv' failed: exit code 2
INFO: Failed at 2025-12-29 16:40:10
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
job errors


vzdump 101 --mode snapshot --storage duckpbs

INFO: starting new backup job: vzdump 101 --mode snapshot --storage duckpbs
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2025-12-29 16:40:51
INFO: status = running
INFO: CT Name: duckcloud
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/srv') in backup
INFO: found old vzdump snapshot (force removal)
  Logical volume "snap_vm-101-disk-0_vzdump" successfully removed.
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: suspend vm to make snapshot
INFO: create storage snapshot 'vzdump'
  Logical volume "snap_vm-101-disk-0_vzdump" created.
zfs_mount_at() failed: directory is not emptyumount: /mnt/vzsnap0/srv: not mounted.
command 'umount -l -d /mnt/vzsnap0/srv' failed: exit code 32
INFO: resume vm
INFO: guest is online again after 1 seconds
ERROR: Backup of VM 101 failed - command 'mount -o ro -t zfs zfs_storage/subvol-101-disk-0@vzdump /mnt/vzsnap0//srv' failed: exit code 2
INFO: Failed at 2025-12-29 16:40:52
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
job errors

so this unfortunately did not solve the problem, do you have any other ideas?

 

[struppu]

I have one more question:
Why does proxmox want to mount the snapshot in /mnt/vzsnap0/srv
is it because my mountpoint mp0 ist mounted as /srv in the container?

The strange thing is that two of my other three containers have a mountpoint mounted under /srv but they get backed up without any problem

I also tried to create the srv directory under /mnt/vzsnap0 but then the backup fails with another error

INFO: starting new backup job: vzdump 101 --storage duckpbs --mode snapshot
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2025-12-30 00:15:42
INFO: status = running
INFO: CT Name: duckcloud
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/srv') in backup
INFO: found old vzdump snapshot (force removal)
  Logical volume "snap_vm-101-disk-0_vzdump" successfully removed.
ERROR: Backup of VM 101 failed - mount point '/mnt/vzsnap0' not empty
INFO: Failed at 2025-12-30 00:15:42
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
job errors

mount point '/mnt/vzsnap0' not empty

Now I tried to clone the container and run a backup on the clone and it worked just fine

now I am really confused for my understanding the clone should have the same problem as the original or am I wrong?

I had one more idea, since I tried the backup on the clone in stopped state, I tried it also with the container running with the result that the backup failed with the same error as on the original.

The problem seems that this container can only be backuped in stoped state - but this is only since I updatet proxmox from 8.4 to 9.1