Unexpected backup failures

[vaschthestampede]

Do you have also another, unrelated datastore which is backed by S3 running on this instance?

No, I do not.

I'm asking since your backtrace includes proxmox-s3-client code paths and there was a recent bugfix for hanging proxy's [1]

I also keep PBS up to date with the no-subscription repository.

proxmox-backup: 4.2.0 (running kernel: 7.0.6-2-pve)
proxmox-backup-server: 4.2.1-1 (running version: 4.2.1)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0: 7.0.6-2
proxmox-kernel-7.0.6-2-pve-signed: 7.0.6-2
proxmox-kernel-7.0.2-6-pve-signed: 7.0.2-6
proxmox-kernel-6.17: 6.17.13-13
proxmox-kernel-6.17.13-13-pve-signed: 6.17.13-13
proxmox-kernel-6.17.2-1-pve-signed: 6.17.2-1
ifupdown2: 3.3.0-1+pmx12
libjs-extjs: 7.0.0-5
proxmox-backup-docs: 4.2.1-1
proxmox-backup-client: 4.2.1-1
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.3
pve-xtermjs: 6.0.0-1
smartmontools: 7.5-pve2
zfsutils-linux: 2.4.2-pve1

 

[vaschthestampede]

After days of relative calm (few, if any, backup failures), yesterday there were 10.

Any news on the proxy.backtrace file analysis?

 

[vaschthestampede]

Please help , failed backups are continuing.

 

[Chris]

Any news on the proxy.backtrace file analysis?

As stated previously, the proxy backtrace points towards lock contention on chunk insert being the issue here. Chunks being inserted into the datastore are synced up by using a chunk store lock. Please open an issue at bugzilla.proxmox.com for this, we should have a look on how to reduce lock contention.

For the time being I'm afraid you will have to workaround by rescheduling some of your backups for them to run without issues.

 

[Robert Obkircher]

Do you have also another, unrelated datastore which is backed by S3 running on this instance? I'm asking since your backtrace includes proxmox-s3-client code paths and there was a recent bugfix for hanging proxy's [1], so this might be related (not packaged yet at the time of writing).

[1] https://git.proxmox.com/?p=pro, so I would suggest upgrading.xmox-backup.git;a=commit;h=23400016322c7a6981f111558e8d22666e32ee8c

This patch is available in version 4.2.2-1. It won't reduce lock contention, but it should make the server more responsive.

Would you mind upgrading and sending a new version of the backtrace? As far as I can tell the one in the Bugzilla issue is still from an older version.

 

[vaschthestampede]

This patch is available in version 4.2.2-1

I'm already on that version...

proxmox-backup: 4.2.0 (running kernel: 7.0.12-1-pve)
proxmox-backup-server: 4.2.2-1 (running version: 4.2.2)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0: 7.0.12-1
proxmox-kernel-7.0.12-1-pve-signed: 7.0.12-1
proxmox-kernel-7.0.6-2-pve-signed: 7.0.6-2
proxmox-kernel-6.17: 6.17.13-13
proxmox-kernel-6.17.13-13-pve-signed: 6.17.13-13
proxmox-kernel-6.17.2-1-pve-signed: 6.17.2-1
ifupdown2: 3.3.0-1+pmx12
libjs-extjs: 7.0.0-5
proxmox-backup-docs: 4.2.2-1
proxmox-backup-client: 4.2.2-1
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.5
pve-xtermjs: 6.0.0-1
smartmontools: 7.5-pve2
zfsutils-linux: 2.4.2-pve1

Yesterday 19 backups failed.