[SOLVED] High IO wait during backups after upgrading to Proxmox 7

[fiona]

In Proxmox VE 7.3, it's possible to configure the number of workers used for VMs for vzdump/backup jobs. It's not yet exposed via UI, please see this post for how to configure it.

 

[e100]

I've upgraded to Proxmox 7.3 and added the new setting to /etc/vzdump.conf
Will report back in a few days if it resolved the problem, which I am confident it will.

Added to /etc/vzdump.conf:

performance: max-workers=1

Thanks @fiona

 

[Dunuin]

I also changed it to 1, as my small Atom CPU can't handle all the hashing/encryption/compression of PBS + ZFS overhead. Will report back after the next weekly backup, if it help when using weak CPUs. Average load jumping from 4 to 32 wasn't great when doing PBS backups and just using a weak quadcore CPU.

 

[e100]

It is also necessary to restart or live migrate the virtual servers so they use the updated kvm binary.
max-workers=1 seems to be working well so far.

On one node I restarted some, but not all VMs
In this IO wait graph the arrows are when vzdump began on a VM running the updated KVM, the spikes in IO wait are backups of VMs still running on the old KVM binary.