Unexpected backup failures

[vaschthestampede]

Do you have also another, unrelated datastore which is backed by S3 running on this instance?

No, I do not.

I'm asking since your backtrace includes proxmox-s3-client code paths and there was a recent bugfix for hanging proxy's [1]

I also keep PBS up to date with the no-subscription repository.

proxmox-backup: 4.2.0 (running kernel: 7.0.6-2-pve)
proxmox-backup-server: 4.2.1-1 (running version: 4.2.1)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0: 7.0.6-2
proxmox-kernel-7.0.6-2-pve-signed: 7.0.6-2
proxmox-kernel-7.0.2-6-pve-signed: 7.0.2-6
proxmox-kernel-6.17: 6.17.13-13
proxmox-kernel-6.17.13-13-pve-signed: 6.17.13-13
proxmox-kernel-6.17.2-1-pve-signed: 6.17.2-1
ifupdown2: 3.3.0-1+pmx12
libjs-extjs: 7.0.0-5
proxmox-backup-docs: 4.2.1-1
proxmox-backup-client: 4.2.1-1
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.3
pve-xtermjs: 6.0.0-1
smartmontools: 7.5-pve2
zfsutils-linux: 2.4.2-pve1

 

[vaschthestampede]

After days of relative calm (few, if any, backup failures), yesterday there were 10.

Any news on the proxy.backtrace file analysis?

 

[vaschthestampede]

Please help , failed backups are continuing.

 

[Chris]

Any news on the proxy.backtrace file analysis?

As stated previously, the proxy backtrace points towards lock contention on chunk insert being the issue here. Chunks being inserted into the datastore are synced up by using a chunk store lock. Please open an issue at bugzilla.proxmox.com for this, we should have a look on how to reduce lock contention.

For the time being I'm afraid you will have to workaround by rescheduling some of your backups for them to run without issues.