Backup failure help and configure

bphillips330

New Member
Dec 9, 2024
4
0
1
I have been a long time Unraid user and switching to Prox / Truenas. Truenas scale running as VM on Prox. Unrelated to this post. Kind of.


Currently i am running all my vm's on Proxmox. Using the built in backup system to backup to Unraid. I added directory to proxmox through Gui for Unraid_backup, NFS share. I can go to CLI and cd /mnt/pve/unraid_backup and I see all the files there. So i know the mount is good and the first backup works. But. I see in the morning that my nightly backup fails.


**cut all the data move***


INFO: backup is sparse: 2.74 GiB (1%) total zero data
INFO: transferred 250.00 GiB in 518 seconds (494.2 MiB/s)
INFO: archive file size: 172.95GB
INFO: adding notes to backup
INFO: prune older backups with retention: keep-last=9
ERROR: Backup of VM 101 failed - mkdir /mnt/pve/unraid_prox_bkcup: File exists at /usr/share/perl5/PVE/Storage/Plugin.pm line 1812.
INFO: Failed at 2024-12-09 08:46:03
command 'df -P -T -B 1 /mnt/pve/unraid_prox_bkcup/dump' failed: exit code 1

Did some searching on google and such and not found much help. I tried to run the backup manually by clicking backup now on Prox after umounting and mount -a. Everything seems to be find. I don't really see any error in /Plugin.pm at that line?

after I ran that backup. I look in the folder on unraid and everything is there. Backup ran fine. But still getting error. So it seems like backup is working. But why the failure then?

Side question. Trying to find best workflow. I am running truenas scale as a vm on prox. Should i run backup to Truenas first. Then just run a backup from truenas to unraid. Backups might be a day behind, but this is home lab stuff and really not super worried about it.
 
please post "pveversion -v" and the contents of "/etc/pve/storage.cfg" as well as the output of "mount". thanks!
 
here is the error from last nights backup.
TASK ERROR: could not activate storage 'unraid_prox_bkcup': mkdir /mnt/pve/unraid_prox_bkcup: File exists at /usr/share/perl5/PVE/Storage/Plugin.pm line 1811.



root@prox:~# pveversion -v
proxmox-ve: 8.3.0 (running kernel: 6.8.12-4-pve)
pve-manager: 8.3.1 (running version: 8.3.1/fb48e850ef9dde27)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-4
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.1
libpve-storage-perl: 8.2.9
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.0-1
proxmox-backup-file-restore: 3.3.0-1
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.3
pve-cluster: 8.0.10
pve-container: 5.2.2
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-1
pve-ha-manager: 4.0.6
pve-i18n: 3.3.2
pve-qemu-kvm: 9.0.2-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.0
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1




root@prox:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content backup,iso,vztmpl

zfspool: local-zfs
pool rpool/data
content images,rootdir
sparse 1

zfspool: nvme_raid0
pool nvme_raid0
content rootdir,images
mountpoint /mnt/pve/nvme_raid0
sparse 0

dir: vms
path /mnt/pve/vms
content backup,vztmpl,iso,images,rootdir
prune-backups keep-all=1
shared 0

cifs: smb_data_pics
path /mnt/pve/smb_data_pics
server 192.168.x.x
share data
content images
prune-backups keep-all=1
username brian

cifs: media-truenas
path /mnt/pve/media-truenas
server 192.168.x.x
share media
content images
prune-backups keep-all=1
username brian

nfs: vm_bkup
export /mnt/user/completed/truenas_backup
path /mnt/pve/vm_bkup
server 192.168.x.x
content backup,images
prune-backups keep-all=1

nfs: eve-ng
export /mnt/datastore/eve_files
path /mnt/pve/eve-ng
server 192.168.x.x
content images
prune-backups keep-all=1

dir: vm_nvme
path /nvme_raid0/vm_stuff
content images,iso
prune-backups keep-all=1
shared 0

nfs: unraid_prox_bkcup
export /mnt/user/completed/proxmox_backup/
path /mnt/pve/unraid_prox_bkcup
server 192.168.x.x
content backup,images
prune-backups keep-all=1

root@prox:~#



root@prox:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=32437912k,nr_inodes=8109478,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6494316k,mode=755,inode64)
rpool/ROOT/pve-1 on / type zfs (rw,relatime,xattr,posixacl,casesensitive)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11307)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
nvme_raid0 on /nvme_raid0 type zfs (rw,relatime,xattr,noacl,casesensitive)
rpool on /rpool type zfs (rw,relatime,xattr,noacl,casesensitive)
rpool/var-lib-vz on /var/lib/vz type zfs (rw,relatime,xattr,noacl,casesensitive)
rpool/ROOT on /rpool/ROOT type zfs (rw,relatime,xattr,noacl,casesensitive)
rpool/data on /rpool/data type zfs (rw,relatime,xattr,noacl,casesensitive)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
192.168.254.12:/mnt/datastore/eve_files on /mnt/pve/eve-ng type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.254.10,local_lock=none,addr=192.168.254.12)
//192.168.254.12/media on /mnt/pve/media-truenas type cifs (rw,relatime,vers=3.1.1,cache=strict,username=xxxxx,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.254.12,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,retrans=1,echo_interval=60,actimeo=1,closetimeo=1)
//192.168.254.12/data on /mnt/pve/smb_data_pics type cifs (rw,relatime,vers=3.1.1,cache=strict,username=xxxx,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.254.12,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,retrans=1,echo_interval=60,actimeo=1,closetimeo=1)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=6494312k,nr_inodes=1623578,mode=700,inode64)
/dev/sdd1 on /mnt/external type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
192.168.254.11:/mnt/user/completed/truenas_backup on /mnt/pve/vm_bkup type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.254.10,local_lock=none,addr=192.168.254.11)
192.168.254.11:/mnt/user/completed/proxmox_backup on /mnt/pve/unraid_prox_bkcup type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.254.10,local_lock=none,addr=192.168.254.11)
root@prox:~#
 
"export /mnt/user/completed/proxmox_backup/"

remove the trailing slash there, it seems to confuse the "is this storage mounted" check in our NFS plugin..
 
root@prox:/mnt/pve# ls -la
ls: cannot access 'unraid_prox_bkcup': Stale file handle
ls: cannot access 'vm_bkup': Stale file handle
total 12
drwxr-xr-x 11 root root 11 Dec 10 06:21 .
drwxr-xr-x 5 root root 5 Nov 27 12:38 ..
drwxr-xr-x 2 root root 2 Nov 27 10:05 data_pictures
drwxrwxrwx 3 root root 4 Dec 5 14:40 eve-ng
drwxr-xr-x 2 root root 0 Nov 30 11:21 media-truenas
drwxr-xr-x 2 root root 2 Nov 28 10:48 movies-truenas
drwxr-xr-x 2 root root 0 Dec 10 06:21 smb_data_pics
drwxr-xr-x 2 root root 2 Dec 10 06:21 Truenas_backup
d????????? ? ? ? ? ? unraid_prox_bkcup
d????????? ? ? ? ? ? vm_bkup
drwxr-xr-x 6 root root 6 Nov 26 12:57 vms

and the weird thing which i forgot to say yesterday. I umount unraid_prox_bkup and vm_bkup
and now
root@prox:/mnt/pve# ls -la
total 12
drwxr-xr-x 11 root root 11 Dec 10 06:21 .
drwxr-xr-x 5 root root 5 Nov 27 12:38 ..
drwxr-xr-x 2 root root 2 Nov 27 10:05 data_pictures
drwxrwxrwx 3 root root 4 Dec 5 14:40 eve-ng
drwxr-xr-x 2 root root 0 Nov 30 11:21 media-truenas
drwxr-xr-x 2 root root 2 Nov 28 10:48 movies-truenas
drwxr-xr-x 2 root root 0 Dec 10 06:21 smb_data_pics
drwxr-xr-x 2 root root 2 Dec 10 06:21 Truenas_backup
drwxrwxrwx 1 root root 26 Dec 6 16:39 unraid_prox_bkcup
drwxr-xr-x 3 root root 3 Nov 30 06:54 vm_bkup
drwxr-xr-x 6 root root 6 Nov 26 12:57 vms

It seems that during the backup process it tries to remount or something those folders. When i Umount them. everything works again? Maybe that helps?
 
please do what I suggested above..
 
I did that. It worked for a day. Then broke again. I have a backup script going to truenas scale and that is working fine. Once i see backup failed. I go to /mnt/pve/ and do ls -la and once again.

d????????? ? ? ? ? ? unraid_prox_bkcup

but umount that folder and everything is all good again.

drwxr-xr-x 2 root root 2 Dec 6 16:39 unraid_prox_bkcup


cat /etc/pve/storage......
nfs: unraid_prox_bkcup
export /mnt/user/completed/proxmox_backup
path /mnt/pve/unraid_prox_bkcup
server 192.168.254.11
content backup,images
prune-backups keep-last=9

nfs: Truenas_backup

it seems when it is trying to delete the last backup is where it fails. Doing some research. Might be permission issue on unraid?

NFO: 94% (47.1 GiB of 50.0 GiB) in 1m, read: 880.2 MiB/s, write: 837.5 MiB/s
INFO: 100% (50.0 GiB of 50.0 GiB) in 1m 3s, read: 982.7 MiB/s, write: 750.5 MiB/s
INFO: backup is sparse: 3.01 GiB (6%) total zero data
INFO: transferred 50.00 GiB in 63 seconds (812.7 MiB/s)
INFO: archive file size: 6.28GB
INFO: adding notes to backup
INFO: prune older backups with retention: keep-last=9
cannot remove directory for /mnt/pve/unraid_prox_bkcup/dump/vzdump-qemu-100-2024_12_16-00_00_01.tmp: Directory not empty at /usr/share/perl5/PVE/VZDump.pm line 1305.
ERROR: Backup of VM 100 failed - unable to activate storage 'unraid_prox_bkcup' - directory '/mnt/pve/unraid_prox_bkcup' does not exist or is unreachable

INFO: Failed at 2024-12-16 00:01:06
 
please check the system logs - that new error hints at the NFS server going down..
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!