Unexpected backup failures

V

vaschthestampede

Well-Known Member
Oct 21, 2020
147
8
58
39
I have a PSB with 18 servers connected. The PBS server is a PowerEdge R740xd2 with two Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz and 128GB RAM.

Code: 
proxmox-backup: 4.2.0 (running kernel: 7.0.0-3-pve)
proxmox-backup-server: 4.2.0-1 (running version: 4.2.0)
proxmox-kernel-helper: 9.0.4
proxmox-kernel-7.0: 7.0.0-3
proxmox-kernel-7.0.0-3-pve-signed: 7.0.0-3
proxmox-kernel-6.17: 6.17.13-6
proxmox-kernel-6.17.13-6-pve-signed: 6.17.13-6
proxmox-kernel-6.17.2-1-pve-signed: 6.17.2-1
ifupdown2: 3.3.0-1+pmx12
libjs-extjs: 7.0.0-5
proxmox-backup-docs: 4.2.0-1
proxmox-backup-client: 4.2.0-1
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.3
proxmox-widget-toolkit: 5.1.9
pve-xtermjs: 5.5.0-3
smartmontools: 7.4-pve1
zfsutils-linux: 2.4.1-pve1

All these servers have a scheduled backup at 6:30 PM.

The problem is that, not always, not always for the same VMs and not always in the same servers, some backups fail and I can't understand why.

I uploaded the logs because the message wouldn't let me include them all.

Let me know what other information might be helpful.
 

Attachments

Last edited:
I assume your PBS datastores resides on a filesystem offered on an iscsi device. Is this filesystem shared with other datastores or other, unrelated data? Note that by default PBS performs an fssync at the end of a backup, to assure data is persisted to disk. Might be the case that this takes a lot on your filesystem, and the PVE client side runs into a timeout?
 
Last edited:
Chris said:
I assume your PBS datastores resides on a filesystem offered on an iscsi device
Click to expand...
Yes.

Chris said:
Is this filesystem shared with other datastores or other, unrelated data?
Click to expand...
No, it is connected via two-meter P2P fiber, without even a switch in between.

Chris said:
Might be the case that this takes a lot on your filesystem, and the PVE client side runs into a timeout?
Click to expand...
How can I check this?
 
I set:
Code: 
proxmox-backup-manager datastore update iSCSI --tuning 'sync-level=none'

I'll let it go to see if the errors persist.
Unfortunately, since they're very random, it could take weeks to see if it's resolved.
 
vaschthestampede said:
I set:
Code: 
proxmox-backup-manager datastore update iSCSI --tuning 'sync-level=none'

I'll let it go to see if the errors persist.
Unfortunately, since they're very random, it could take weeks to see if it's resolved.
Click to expand...
make sure to restart the PBS services as well, there is currently a bug which prevents the sync level from taking immediate effect. also, if acceptable performance wise i would recommend to rather use file instead of none. To be on the safe side.
 
Last edited:
Please share once again the backup task log from PVE and the corresponding one from PBS. And share the output of pveversion -v from the node the VM is running on.
 
Last edited:
I'm preparing the logs, but the command you tell me to show gives an error.
Code: 
root@Chimera:~# pve-version -v
-bash: pve-version: command not found

Do you mean this?
Code: 
proxmox-ve: 9.1.0 (running kernel: 6.17.4-2-pve)
pve-manager: 9.1.4 (running version: 9.1.4/5ac30304265fbd8e)
proxmox-kernel-helper: 9.0.4
proxmox-kernel-6.17.4-2-pve-signed: 6.17.4-2
proxmox-kernel-6.17: 6.17.4-2
proxmox-kernel-6.17.2-2-pve-signed: 6.17.2-2
proxmox-kernel-6.14.11-5-pve-signed: 6.14.11-5
proxmox-kernel-6.14: 6.14.11-5
proxmox-kernel-6.8: 6.8.12-17
proxmox-kernel-6.8.12-17-pve-signed: 6.8.12-17
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
ceph-fuse: 19.2.3-pve2
corosync: 3.1.9-pve2
criu: 4.1.1-1
frr-pythontools: 10.4.1-1+pve1
ifupdown2: 3.3.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.0
libproxmox-backup-qemu0: 2.0.1
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.5
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.0.7
libpve-cluster-perl: 9.0.7
libpve-common-perl: 9.1.4
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.2.4
libpve-rs-perl: 0.11.4
libpve-storage-perl: 9.1.0
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 6.0.5-3
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
proxmox-backup-client: 4.1.1-1
proxmox-backup-file-restore: 4.1.1-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.1
proxmox-kernel-helper: 9.0.4
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.3
proxmox-widget-toolkit: 5.1.5
pve-cluster: 9.0.7
pve-container: 6.0.18
pve-docs: 9.1.2
pve-edk2-firmware: 4.2025.05-2
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.17-2
pve-ha-manager: 5.1.0
pve-i18n: 3.6.6
pve-qemu-kvm: 10.1.2-5
pve-xtermjs: 5.5.0-3
qemu-server: 9.1.3
smartmontools: 7.4-pve1
spiceterm: 3.4.1
swtpm: 0.8.0+pve3
vncterm: 1.9.1
zfsutils-linux: 2.3.4-pve1
 
You must log in or register to reply here.
Top Bottom
Back