Proxmox VZdump Backup bottleneck

[cornySlug]

Hi all,

I work with multiple PVE hosts (6.4, ssd-only) and want to speed up my backup transfer speeds without using PBS. All systems are connected via a 10Gibt network. The last months i used a nfs target with hdd-only storage for my vzdump backups. To speed things up I´ve set up an additional backup target with TureNAS Core, 10 Gbit/s and ssds for storage. My problem is that I can´t get more than ~220 MB/s (ZSTD) or ~300 MB/s (No-Compression) when backing up vms to the TureNAS system via nfs.

• iperf3 test (both directions) between TrueNAS and PVE host(s): ~9,5 GBit/s
• file transfer speed from PVE to TureNAS via nfs-mount: ~680 MB/s
• CPU/RAM usage is fine on both ends

Any suggestions to find the bottleneck and speed up my vzdump backups?
I´ve also started to integrate a PBS backup solution, but for legacy reasons I want to keep vzdumps for at least a few years.

 

[Dunuin]

Vzdump won't use dirty bitmaps so it needs to always read the full VM. If you got thin provisioning and most of your virtual disks are unused thats a real bottleneck. Lets say your SSDs can only read with 1000 MB/s and your virtual disk is 90% unused. Now you can't write with more then 100MB/s because of the 1000MB read per second only 100MB are actually data that needs to be backuped.

 

[cornySlug]

Vzdump won't use dirty bitmaps so it needs to always read the full VM. If you got thin provisioning and most of your virtual disks are unused thats a real bottleneck.

Thanks Dunuin for your answer. I read that QCOW2 use thin provisioning. To get this factor out of the equation i tested the following:

• Added an additional disk (30GB) to a linux VM (qcow2)
• Populated the disk with a ~30GB large random file
• Changed disks´backup to only backup the new 30gb disk with >99% random data
• Started a backup of the vm
  • Source and target are separate SATA SSDs with no raid and no other load within the same physical unit
    • ZSTD compression: 179 MiB/s
    • no compression: 345 MiB/s
  • Source is a SATA SSD with no other load, target is my TureNAS (nfs storage)
    • ZSTD compression: 177 MiB/s
    • no compression: 338 MiB/s

Can i rule out the thin/thick provisioing factor with this experiment?

 

[Dunuin]

Can i rule out the thin/thick provisioing factor with this experiment?

Yes, so the bottlenack is somewhere else.
Another bottleneck I could think of is read/write amplification. Depending on the blocksize, sync/async, and random/sequential reads/writes your write/read amplification and write/read performance may be very different.

I just did some benchmarks and performance/amplification reached from 1,87 MiB/s writes + factor 56,84 write amplification for 4K sync writes to 745 MiB/s writes + factor 3,88 write amplification for async 32K random writes. So depending on what you are doing the performance may be very different. (tested with 2 enterprise SATA SSDs as a zfs mirror with 4K blocksize on the host and test performed inside a guest using 4K ext4)

Would be interesting if the staff could tell us what kind of workload vzdump is producing.

Could be possible that your SSD is only capable to read 345 MiB/s for that kind of workload.

 

[fabian]

please post a backup log - but yeah, the numbers would indicate that your source only manages around 345MB/s of reads.

 

[cornySlug]

please post a backup log - but yeah, the numbers would indicate that your source only manages around 345MB/s of reads.

sorry for the delayed answer. The SSDs i used back then were customer grade Samsung SSDs which were probably the bottleneck. As i don´t work with proxomox I only wanted to mark your answer als probable solution to my problem.