Very poor vzdump performance

[mutex]

My Proxmox cluster has been seeing incredibly slow VM backups, with vzdump speeds hovering at around 15 MB/s read.

I do all my backups to a locally hosted Proxmox Backup Server, which runs on a RAIDz of 4x 4TB CMR hard drives, with a ZIL and L2ARC SSD in the pool.
The vzdump config is completely default. The storage being used for VMs is ZFS on NVMe drives, on both nodes.

Here's one VM being backed up:


An iperf3 test from the Proxmox host backing up its VMs to the PBS VM:


The two-node cluster runs the same Proxmox versions across both nodes, those being kernel 6.5.11-8-pve and pve-manager/8.1.4/ec5affc9e41f1d79.

I'd appreciate any tips debugging this further; the VMs take forever to back up and are practically unusable while in the process. Frankly I'm out of ideas.

 

[esi_y]

An iperf3 test from the Proxmox host backing up its VMs to the PBS VM:
View attachment 63327

And do you have any idea why the Retr are non-zero?

 

[fiona]

Hi,
if there is high IO wait during the backup, please see: https://forum.proxmox.com/threads/t...oad-behavior-during-backup.118430/post-513106