Backup Error: fstat "xyz" failed - ENOENT: No such file or directory

[sdet00]

Hello everyone! I have a very unusual issue with getting backups to run with a specific Debian LXC. If any of my ZFS mount points are included as part of the backup, the backup to my Proxmox Backup Server will fail immediately.

What is interesting is that only this LXC has issues with this. I have other LXCs with mount points passed through and those appear to operate without issue. VMs are also backing up OK. The LXC configuration is virtually identical as well, which leads me to believe that something to do with the mount point itself is causing this issue, although I am not sure what exactly. This Debian LXC acts as a fileserver and I wonder if it has something to do with permissions that it applies to the folders in these mountpoints.

My Proxmox Backup Server is running on another Proxmox Node and has been setup as an LXC as well. Below is my log output when trying to initiate a backup. Any suggestions on how to solve this issue would be greatly appreciated!

INFO: starting new backup job: vzdump 400 --fleecing 0 --notes-template '{{cluster}}, {{guestname}}, {{node}}, {{vmid}}' --mode snapshot --all 0 --bwlimit 107520 --prune-backups 'keep-all=1' --node at-hv-07 --storage at-pbs-01-backups1
INFO: Starting Backup of VM 400 (lxc)
INFO: Backup started at 2025-01-06 19:02:19
INFO: status = running
INFO: CT Name: at-media-01
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('ZFS-JZ2H/media1') in backup
INFO: including mount point mp1 ('ZFS-JZ2H/at-org1') in backup
INFO: including mount point mp2 ('ZFS-JZ2H/at-setup1') in backup
INFO: including mount point mp3 ('ZFS-JZ2H/at-temp1') in backup
INFO: including mount point mp4 ('ZFS-JZ2H/3ded-org1') in backup
INFO: including mount point mp5 ('ZFS-JZ2H/md-user1') in backup
INFO: including mount point mp6 ('ZFS-JZ2H/sd-user1') in backup
INFO: backup mode: snapshot
INFO: bandwidth limit: 107520 KiB/s
INFO: ionice priority: 7
INFO: suspend vm to make snapshot
INFO: create storage snapshot 'vzdump'
INFO: resume vm
INFO: guest is online again after 4 seconds
INFO: creating Proxmox Backup Server archive 'ct/400/2025-01-06T09:02:19Z'
INFO: set max number of entries in memory for file-based backups to 1048576
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp714855_400/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --include-dev /mnt/vzsnap0/.ZFS-JZ2H/media1 --include-dev /mnt/vzsnap0/.ZFS-JZ2H/at-org1 --include-dev /mnt/vzsnap0/.ZFS-JZ2H/at-setup1 --include-dev /mnt/vzsnap0/.ZFS-JZ2H/at-temp1 --include-dev /mnt/vzsnap0/.ZFS-JZ2H/3ded-org1 --include-dev /mnt/vzsnap0/.ZFS-JZ2H/md-user1 --include-dev /mnt/vzsnap0/.ZFS-JZ2H/sd-user1 --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 400 --backup-time 1736154139 --entries-max 1048576 --repository backupuser@pbs@192.168.xx.xx:backups1
INFO: Error: fstat "/mnt/vzsnap0/.ZFS-JZ2H/media1" failed - ENOENT: No such file or directory
umount: /mnt/vzsnap0ZFS-JZ2H/sd-user1: no mount point specified.
command 'umount -l -d /mnt/vzsnap0ZFS-JZ2H/sd-user1' failed: exit code 32
ERROR: Backup of VM 400 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp714855_400/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --include-dev /mnt/vzsnap0/.ZFS-JZ2H/media1 --include-dev /mnt/vzsnap0/.ZFS-JZ2H/at-org1 --include-dev /mnt/vzsnap0/.ZFS-JZ2H/at-setup1 --include-dev /mnt/vzsnap0/.ZFS-JZ2H/at-temp1 --include-dev /mnt/vzsnap0/.ZFS-JZ2H/3ded-org1 --include-dev /mnt/vzsnap0/.ZFS-JZ2H/md-user1 --include-dev /mnt/vzsnap0/.ZFS-JZ2H/sd-user1 --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 400 --backup-time 1736154139 --entries-max 1048576 --repository backupuser@pbs@192.168xx.xx:backups1' failed: exit code 255
INFO: Failed at 2025-01-06 19:02:23
INFO: Backup job finished with errors
TASK ERROR: job errors

 

[Chris]

Hi,

INFO: Error: fstat "/mnt/vzsnap0/.ZFS-JZ2H/media1" failed - ENOENT: No such file or directory umount: /mnt/vzsnap0ZFS-JZ2H/sd-user1: no mount point specified.

please share the container config pct config 400 --current. Above error might indicate that there is an issue with the mountpoint. Also, do you get any errors when you run a pct mount 400?

 

[sdet00]

Found the issue - I was digging into the differences and it looks like my mountpoint naming convention was causing issues. For some reason I decided my mountpoints within my LXC should be mounted under the ZFS pool name (e.g. "/ZFS-JZ2H/directory-1") for example, and that was causing some weirdness.

I reconfigured mountpoints to mount under /mnt/ instead (e.g. "/mnt/directory-1") and the issue is now solved.