[SOLVED] Backup of LXC fails sometimes

[DoesThisWork]

Hello,

I am currently trying to figure out, why one of my LXCs only backups once in a blue moon.

So far from the 20.07-26.07 the backup didnt work of that LXC. On the 19.07 and 27.07 the backup worked.

On this LXC is Nextcloud running with an additional mount point just for the nextcloud data. The size of the mount point is 100 GB.

I was wondering if the problem could be that my local storage of my Proxmox VE is only 100 GB and so that wouldn't be enough to store the 2 mount points locally in the tmp folder. It doesn't really explain why it works sometimes though..

In total the LXC has a capacity of 130 GB. Of those 130 GB roughly 30 GB are being used.

19.07.2022 => 903    nextcloud    OK    00:02:01    20.00GB    ct/903/2022-07-19T20:04:10Z

26.07.2022 => 903    nextcloud    FAILED    00:01:24    command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file 
--inplace --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' 
'--exclude=/var/run/?*.pid' /proc/1151489/root//./ /proc/1151489/root//./srv/nextcloud-data /var/tmp/vzdumptmp2045824_903' failed: exit code 23

If you need more information please let me know

Best regards

 

[oguz]

hi,

On this LXC is Nextcloud running with an additional mount point just for the nextcloud data. The size of the mount point is 100 GB.

I was wondering if the problem could be that my local storage of my Proxmox VE is only 100 GB and so that wouldn't be enough to store the 2 mount points locally in the tmp folder. It doesn't really explain why it works sometimes though..

could be the case, but hard to tell without logs.
rsync error code 23 implies "partial transfer due to error" which could have a different root cause.

if you can post:
* task log of a failed backup (from the GUI tasks log at the bottom)
* container configuration pct config 903
* pveversion -v
then it'll be easier to tell why it's happening

 

[DoesThisWork]

Thanks so much for the quick reply.

hi,


could be the case, but hard to tell without logs.
rsync error code 23 implies "partial transfer due to error" which could have a different root cause.

if you can post:
* task log of a failed backup (from the GUI tasks log at the bottom)
* container configuration pct config 903
* pveversion -v
then it'll be easier to tell why it's happening

I am also using the Proxmox Backup Server.

Task log

INFO: Starting Backup of VM 903 (lxc)
INFO: Backup started at 2022-07-21 22:03:48
INFO: status = running
INFO: CT Name: nextcloud
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/srv/nextcloud-data') in backup
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: nextcloud
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/srv/nextcloud-data') in backup
INFO: starting first sync /proc/1151489/root/ to /var/tmp/vzdumptmp1432131_903
INFO: first sync finished - transferred 21.67G bytes in 78s
INFO: suspending guest
INFO: starting final sync /proc/1151489/root/ to /var/tmp/vzdumptmp1432131_903
INFO: resume vm
INFO: guest is online again after 2 seconds
ERROR: Backup of VM 903 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --inplace --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/1151489/root//./ /proc/1151489/root//./srv/nextcloud-data /var/tmp/vzdumptmp1432131_903' failed: exit code 23
INFO: Failed at 2022-07-21 22:05:11

Container Config

arch: amd64
cores: 4
features: nesting=1
hostname: nextcloud
memory: 4096
mp0: NAS:903/vm-903-disk-1.raw,mp=/srv/nextcloud-data,backup=1,size=100G
nameserver: 192.168.100.1
net0: name=eth0,bridge=vmbr1,firewall=1,gw=192.168.100.1,hwaddr=DA:61:F8:69:91:FF,ip=192.168.100.60/24,type=veth
onboot: 1
ostype: debian
rootfs: NAS:903/vm-903-disk-0.raw,size=30G
swap: 1024
unprivileged: 1

pveversion

proxmox-ve: 7.2-1 (running kernel: 5.15.35-1-pve)
pve-manager: 7.2-4 (running version: 7.2-4/ca9d43cc)
pve-kernel-5.15: 7.2-3
pve-kernel-helper: 7.2-3
pve-kernel-5.15.35-1-pve: 5.15.35-3
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.1-8
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-2
libpve-storage-perl: 7.2-4
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.1-1
proxmox-backup-file-restore: 2.2.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-2
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

 

[fabian]

this might be related to PHP session files (they have special permissions in debian, and the second rsync run might not be able to overwrite the temp copy because of that).

see: https://forum.proxmox.com/threads/backup-failed.104678/#post-451763

 

[DoesThisWork]

this might be related to PHP session files (they have special permissions in debian, and the second rsync run might not be able to overwrite the temp copy because of that).

see: https://forum.proxmox.com/threads/backup-failed.104678/#post-451763

Wow! That was some quick help!!!

I added the

I added the exclude-path: /var/lib/php/sessions to

to the /etc/vzdump.conf file and after trying to backup the LXC I didn't get any errors. I will report back after the nightly backup!

Is the /etc/vzdump.conf replicated to the other hosts in the cluster? When I checked I didn't see the changes on the other hosts. Just wondering because when the LXC is being migrated to another host, the backup would fail again because the other host doesn't have the same vzdump.cfg

Can I exclude the path in the backup job or what is the best practice for that?

Thank you so much for you insanely quick help!

 

[fabian]

yeah, exclusion in vzdump.conf is per node, but you can add it to the job instead if you prefer that.

 

[DoesThisWork]

yeah, exclusion in vzdump.conf is per node, but you can add it to the job instead if you prefer that.

Perfect! Thank you so much for your help.