Not sure why but my lxc stopped being able to create a backup. I tried a manual backup (snapshot, suspend, stop), ZSTD compression. But all of the backups I tried stopped. The backup stopped working a month ago. All my other lxc's backup on the same storage and have no issues, I also tried backing up onto a different location (local) but that doesn't work either so must be something wrong with the LXC? I just have no idea what.
this is my storage.cfg content
content lxc config
Code:
()
INFO: starting new backup job: vzdump 104 --storage backups_proxmox --compress zstd --mode snapshot --node pve --remove 0
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp1824546_104 for temporary files
INFO: Starting Backup of VM 104 (lxc)
INFO: Backup started at 2022-06-15 18:59:31
INFO: status = running
INFO: CT Name: torrent
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
syncfs '/torrents' failed - No space left on device
Logical volume "snap_vm-104-disk-0_vzdump" created.
INFO: creating vzdump archive '/mnt/pve/backups_proxmox/dump/vzdump-lxc-104-2022_06_15-18_59_31.tar.zst'
INFO: tar: ./var/lib/docker/overlay2/1fb3e99ebc90d57552138dd4ccf6c20d91a75d89b94359a0d723bd7e7124c307/diff/usr/lib/mono/4.7.1-api/System.Web.Extensions.dll: File shrank by 35328 bytes; padding with zeros
INFO: Total bytes written: 26765168640 (25GiB, 130MiB/s)
INFO: cleanup temporary 'vzdump' snapshot
Logical volume "snap_vm-104-disk-0_vzdump" successfully removed
ERROR: Backup of VM 104 failed - command 'set -o pipefail && tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/tmp/vzdumptmp1824546_104' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/backups_proxmox/dump/vzdump-lxc-104-2022_06_15-18_59_31.tar.dat' failed: exit code 1
INFO: Failed at 2022-06-15 19:02:52
INFO: Backup job finished with errors
TASK ERROR: job errors
()
INFO: starting new backup job: vzdump 104 --mode suspend --node pve --remove 0 --storage backups_proxmox --compress zstd
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp1929216_104 for temporary files
INFO: Starting Backup of VM 104 (lxc)
INFO: Backup started at 2022-06-15 20:29:15
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: torrent
INFO: including mount point rootfs ('/') in backup
INFO: starting first sync /proc/15308/root/ to /var/tmp/vzdumptmp1929216_104
ERROR: Backup of VM 104 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/15308/root//./ /var/tmp/vzdumptmp1929216_104' failed: exit code 23
INFO: Failed at 2022-06-15 20:31:27
INFO: Backup job finished with errors
TASK ERROR: job errors
()
INFO: starting new backup job: vzdump 104 --compress zstd --storage backups_proxmox --mode stop --remove 0 --node pve
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp1932409_104 for temporary files
INFO: Starting Backup of VM 104 (lxc)
INFO: Backup started at 2022-06-15 20:33:57
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: torrent
INFO: including mount point rootfs ('/') in backup
INFO: stopping virtual guest
INFO: creating vzdump archive '/mnt/pve/backups_proxmox/dump/vzdump-lxc-104-2022_06_15-20_33_57.tar.zst'
INFO: tar: ./var/lib/docker/overlay2/1fb3e99ebc90d57552138dd4ccf6c20d91a75d89b94359a0d723bd7e7124c307/diff/usr/lib/mono/4.7.1-api/System.Web.Extensions.dll: File shrank by 35328 bytes; padding with zeros
INFO: Total bytes written: 26884700160 (26GiB, 137MiB/s)
INFO: restarting vm
explicitly configured lxc.apparmor.profile overrides the following settings: features:mount
INFO: guest is online again after 207 seconds
ERROR: Backup of VM 104 failed - command 'set -o pipefail && tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/tmp/vzdumptmp1932409_104' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/backups_proxmox/dump/vzdump-lxc-104-2022_06_15-20_33_57.tar.dat' failed: exit code 1
INFO: Failed at 2022-06-15 20:37:24
INFO: Backup job finished with errors
TASK ERROR: job errors
this is my storage.cfg content
Code:
root@pve:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content vztmpl,backup,iso
lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir
cifs: backups_proxmox
path /mnt/pve/backups_proxmox
server 192.168.1.50
share backups_proxmox
content iso,rootdir,snippets,backup,vztmpl,images
prune-backups keep-last=4
username mayne
content lxc config
Code:
root@pve:~# cat /etc/pve/lxc/104.conf
arch: amd64
cores: 2
features: mount=cifs
hostname: torrent
memory: 2048
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,hwaddr=9A:FE:12:15:31:8A,ip=192.168.1.60/24,type=veth
onboot: 1
ostype: ubuntu
rootfs: local-lvm:vm-104-disk-0,size=40G
startup: order=5
swap: 2048
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop: