I've got one LXC that's failing backup whenever I run it. I've tried setting the vzdump.conf temp dirs to my mergerfs pool in the thoughts that it was my main drive filling. But the error is really not giving me any pointers. And even watching it I could not see any drives full.
Here are the logs.
I watched the first step where the /mnt/mainpool/proxmox/dump/vzdump-lxc-102-2023_12_27-09_41_13.tar.dat was created, it got to about 37GB then the 2nd part seems to run which gets that error code 1.
Even though I updated vzdump.conf to set tmpdir to
It was putting something to /mnt/vzsnap0 which I thought might have been the problem, so I symlinked that dir back to the mainpool which has about 2TB free.
I've also tried suspend and stop backups to no avail.
Here are the logs.
Code:
INFO: starting new backup job: vzdump 102 --mode snapshot --node sauruman --remove 0 --compress zstd --storage sauruman-proxmox --notes-template '{{guestname}}'
INFO: Starting Backup of VM 102 (lxc)
INFO: Backup started at 2023-12-27 09:41:13
INFO: status = running
INFO: CT Name: fosco
INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp3 ('/mnt/mainpool') from backup (not a volume)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
WARNING: You have not turned on protection against thin pools running out of space.
WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
Logical volume "snap_vm-102-disk-0_vzdump" created.
WARNING: Sum of all thin volume sizes (170.00 GiB) exceeds the size of thin pool pve/data and the amount of free space in volume group (16.00 GiB).
INFO: creating vzdump archive '/mnt/mainpool/proxmox/dump/vzdump-lxc-102-2023_12_27-09_41_13.tar.zst'
INFO: tar: ./var/lib/docker/overlay2/847b9094a35568a92a01b66ec668a2985defae0bdb6715c995469721fdba0790/diff/app/lidarr/bin/libcoreclr.so: File shrank by 3117128 bytes; padding with zeros
INFO: Total bytes written: 62566983680 (59GiB, 33MiB/s)
INFO: cleanup temporary 'vzdump' snapshot
Logical volume "snap_vm-102-disk-0_vzdump" successfully removed.
ERROR: Backup of VM 102 failed - command 'set -o pipefail && tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/mainpool/dump/vzdumptmp1522559_102/' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp' '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd '--threads=1' >/mnt/mainpool/proxmox/dump/vzdump-lxc-102-2023_12_27-09_41_13.tar.dat' failed: exit code 1
INFO: Failed at 2023-12-27 10:11:37
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errorsI watched the first step where the /mnt/mainpool/proxmox/dump/vzdump-lxc-102-2023_12_27-09_41_13.tar.dat was created, it got to about 37GB then the 2nd part seems to run which gets that error code 1.
Even though I updated vzdump.conf to set tmpdir to
Code:
tmpdir: /mnt/mainpool/dump
#dumpdir: DIR
#storage: STORAGE_ID
#mode: snapshot|suspend|stop
#bwlimit: KBPS
#performance: [max-workers=N][,pbs-entries-max=N]
#ionice: PRI
#lockwait: MINUTES
#stopwait: MINUTES
#stdexcludes: BOOLEAN
#mailto: ADDRESSLIST
#prune-backups: keep-INTERVAL=N[,...]
#script: FILENAME
exclude-path: /tmp
#pigz: N
#notes-template: {{guestname}}It was putting something to /mnt/vzsnap0 which I thought might have been the problem, so I symlinked that dir back to the mainpool which has about 2TB free.
I've also tried suspend and stop backups to no avail.