Backup keeps failing for LXC container after increasing disk size

[linucksproxmox]

I recently increased disk size for a container from 40GB to 125GB (and currently using ~85GB). Since then, my scheduled backups are failing with different exit codes. I've seen exit codes 10, 11, and 23. I read that it's possible for /var/tmp to fill up, so I made that a symlink to another location with ample capacity. After that point, I'm only seeing exit code 23.
I ran the rsync command from the error output manually, adding the -v option so I could see what was causing problems. I'm seeing a lot of messages like this
rsync: [generator] set_acl: sys_acl_set_file(var/log/journal/72140ec7474641fa901709d4ba0ffd09/user-1000@b0565d7cec744d2ba77d1df12028c089-00000000000181df-0005fc3146da997d.journal, ACL_TYPE_ACCESS): Operation not supported (95)
I ran a backup with the container powered off and there were no issues, but currently the snapshot mode backups are having this issue. Any thoughts on what I can check at this point?

 

[fabian]

please post
- the full backup task log
- the container config
- the stoage config
- pveversion -v output

 

[linucksproxmox]

Backup log:

INFO: CT Name: dockercompose02
INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp0 ('/mnt/nextcloud') from backup (not a volume)
INFO: excluding bind mount point mp1 ('/mnt/server_backups') from backup (not a volume)
INFO: excluding bind mount point mp2 ('/mnt/unsorted_photos') from backup (not a volume)
INFO: excluding bind mount point mp3 ('/mnt/photos') from backup (not a volume)
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: dockercompose02
INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp0 ('/mnt/nextcloud') from backup (not a volume)
INFO: excluding bind mount point mp1 ('/mnt/server_backups') from backup (not a volume)
INFO: excluding bind mount point mp2 ('/mnt/unsorted_photos') from backup (not a volume)
INFO: excluding bind mount point mp3 ('/mnt/photos') from backup (not a volume)
INFO: starting first sync /proc/1729299/root/ to /var/tmp/vzdumptmp3324876_303
ERROR: Backup of VM 303 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' '--exclude=/mnt/nextcloud' '--exclude=/mnt/server_backups' '--exclude=/mnt/unsorted_photos' '--exclude=/mnt/photos' /proc/1729299/root//./ /var/tmp/vzdumptmp3324876_303' failed: exit code 23
INFO: Failed at 2023-06-20 08:31:04

Container config:

root@pve:~# cat /etc/pve/lxc/303.conf
arch: amd64
cores: 8
features: nesting=1
hostname: dockercompose02
memory: 16384
mp0: /tank/nextcloud,mp=/mnt/nextcloud
mp1: /tank/smbshare/server_backups/,mp=/mnt/server_backups,mountoptions=noatime,replicate=0
mp2: /tank/smbshare/photos/unsorted/,mp=/mnt/unsorted_photos,mountoptions=noatime,replicate=0
mp3: /tank/smbshare/photos/,mp=/mnt/photos,mountoptions=noatime,replicate=0
nameserver: 192.168.1.22
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,hwaddr=62:EC:07:FE:38:31,ip=192.168.1.30/24,type=veth
onboot: 1
ostype: ubuntu
rootfs: nvme1tb:303/vm-303-disk-0.raw,mountoptions=noatime,replicate=0,size=125G
swap: 1024
unprivileged: 1

Storage configuration: I believe this info is all in the container config, unless you're looking for other information. The rootfs is on an nvme disk and is an ext4 filesystem. The others are mount points from local ZFS volumes.

Pveversion -v:

root@pve:~# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.107-1-pve)
pve-manager: 7.4-13 (running version: 7.4-13/46c37d9c)
pve-kernel-5.15: 7.4-3
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.107-2-pve: 5.15.107-2
pve-kernel-5.15.107-1-pve: 5.15.107-1
pve-kernel-5.13.19-6-pve: 5.13.19-15
ceph-fuse: 14.2.21-1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.2-1
proxmox-backup-file-restore: 2.4.2-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-4
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-4
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

 

[fabian]

your tmpdir must support ACLs, else suspend mode won't work.

 

[linucksproxmox]

Do you mean the tmpdir on the proxmox host, at path /var/tmp?

 

[fabian]

yes, that is where the temporary copy of the data is made.

 

[linucksproxmox]

That makes sense, so I removed the symlink and moved the files back over, so that /var/tmp is on the root disk. My root disk is 1TB, but the root volume is only 96GB, and I have about 70GB free currently. I think I need more space for the tmpdir, but what's a good way to expand this or otherwise utilize more of the disk for tmp storage?

 

[fabian]

I don't know how your storage is structured, so I can't tell. you can override the tmpdir though in vzdump.conf and point it at any location you want

 

[linucksproxmox]

Thank you, I updated vzdump.conf to point tmpdir to another location on the disk and my backups are working now!