Proxmox VE speed of backups

[MasinAD]

Since two days ago we backup to a Proxmox Backup Server. Setting it up is really a breeze, so kudos, Proxmox Team!

But when we received our first backup report e-mail (of a lx container) we noticed that the average speed was only at 27 MB/s. That's not really fast.

Of course, I first checked the network connection using iperf. It's at >940 MB/s, so that's not the culprit.

Then I dd'ed the logical volume into a file on the PVE server, which was at about 180 - 190 MB/s. I piped it through gz which resulted in an average speed of 37 MB/s. I repeated this with lzop and zstd which were at 440 and 340 MB/s respectively.

So, maybe gz is at fault? Nope, because I told the backup to use zstd.

I then repeated those tests on the PB server. As I didn't have any VM or LXC images there I copied one from the PVE server. At 110 MB/s. gz 42 MB/s, lzop 478 MB/s, zstd 312 MB/s.

But as soon as I use vzdump the throughput collapses to meager 27 MB/s. So, what's happening here? Why is the backup speed so low?

 

[dietmar]

What kind of backup storage do you use?

 

[MasinAD]

What kind of backup storage do you use?

PBS is configured to use a local volume. PVE, of course, is configured to backup to PBS.

I have to add that vzdump is that slow even when backing up locally.

 

[MasinAD]

More information:
PVE version 6.2-10 (yeah, I know, but I'd have to shutdown all VMs to update to 6.3)
Backup Server version 1.0-6

The PBS originally was a Debian Buster I extended by your PBS repos. I followed the installation guide. The storage is an encrypted LVM on HW RAID-5, IIRC. Because there are already other backups on these volumes I didn't want to freshly install PBS. As the problem also appears locally, I don't think it's caused by my setup.

 

[dietmar]

The storage is an encrypted LVM on HW RAID-5, IIRC.

Again, what kind of storage/disks? I guess you use slow spinning disks?

 

[MasinAD]

I scheduled a backup of most of our VMs and containers on Saturday at 00:15. I just read the result mail:

masin@SI-C001:/Users/masin/Projekte % grep -e "avgerage" -e "transferred" vzdump.log
100: 2021-01-16 00:16:42 INFO: root.pxar: had to upload 171.86 MiB of 3.36 GiB in 99.82s, avgerage speed 1.72 MiB/s).
101: 2021-01-16 00:17:01 INFO: root.pxar: had to upload 922.45 MiB of 947.26 MiB in 17.50s, avgerage speed 52.72 MiB/s).
102: 2021-01-16 00:17:14 INFO: root.pxar: had to upload 767.10 MiB of 792.02 MiB in 12.26s, avgerage speed 62.59 MiB/s).
110: 2021-01-16 00:17:53 INFO: transferred 4.00 GiB in 38 seconds (107.8 MiB/s)
120: 2021-01-16 00:20:52 INFO: transferred 20.00 GiB in 178 seconds (115.1 MiB/s)
122: 2021-01-16 00:28:12 INFO: transferred 50.00 GiB in 439 seconds (116.6 MiB/s)
123: 2021-01-16 03:55:50 INFO: transferred 1.52 TiB in 12457 seconds (127.6 MiB/s)
124: 2021-01-16 03:57:46 INFO: root.pxar: had to upload 7.05 GiB of 8.78 GiB in 115.35s, avgerage speed 62.55 MiB/s).
125: 2021-01-16 03:59:00 INFO: root.pxar: had to upload 5.06 GiB of 5.06 GiB in 71.70s, avgerage speed 72.22 MiB/s).
151: 2021-01-16 04:01:53 INFO: root.pxar: had to upload 13.40 GiB of 15.55 GiB in 170.94s, avgerage speed 80.29 MiB/s).
152: 2021-01-16 04:06:21 INFO: root.pxar: had to upload 23.53 GiB of 34.27 GiB in 265.40s, avgerage speed 90.79 MiB/s).
160: 2021-01-16 04:13:00 INFO: transferred 100.00 GiB in 395 seconds (259.2 MiB/s)
230: 2021-01-16 04:23:16 INFO: transferred 100.00 GiB in 615 seconds (166.5 MiB/s)
251: 2021-01-16 04:24:01 INFO: root.pxar: had to upload 4.94 GiB of 4.95 GiB in 44.09s, avgerage speed 114.75 MiB/s).
253: 2021-01-16 04:25:01 INFO: root.pxar: had to upload 2.75 GiB of 2.83 GiB in 57.98s, avgerage speed 48.55 MiB/s).
256: 2021-01-16 04:26:02 INFO: root.pxar: had to upload 2.37 GiB of 2.44 GiB in 59.56s, avgerage speed 40.81 MiB/s).
400: 2021-01-16 04:45:31 INFO: transferred 100.00 GiB in 1167 seconds (87.7 MiB/s)
424: 2021-01-16 04:47:00 INFO: root.pxar: had to upload 6.72 GiB of 8.46 GiB in 88.69s, avgerage speed 77.60 MiB/s).
800: 2021-01-16 04:50:09 INFO: root.pxar: had to upload 5.58 GiB of 5.62 GiB in 186.67s, avgerage speed 30.61 MiB/s).
911: 2021-01-16 05:24:51 INFO: transferred 200.00 GiB in 2077 seconds (98.6 MiB/s)
912: 2021-01-16 05:45:53 INFO: transferred 200.00 GiB in 1250 seconds (163.8 MiB/s)
913: 2021-01-16 06:05:08 INFO: transferred 200.00 GiB in 1151 seconds (177.9 MiB/s)
914: 2021-01-16 06:23:50 INFO: transferred 200.00 GiB in 1118 seconds (183.2 MiB/s)
999: 2021-01-16 06:24:12 INFO: root.pxar: had to upload 1.14 GiB of 1.16 GiB in 21.52s, avgerage speed 54.26 MiB/s).

Yeah, there's a typo ("avgerage"). Most the time, the transfer speed isn't that bad. It's noteworthy that VMs are much faster at backing up. Also, it's noteworthy that backup speed of containers are much more unsteady.

The lxc I made my tests with isn't part of the backup. It's a db container which I shutdown properly before backing up. All the other containers and VMs don't have any production databases running in them. id 100 is even slower than my db container id 150 (not seen here).

I find it interesting that PVE uses LVM thin volumes for both qemu and lxc but still uses different backup logic for them.

 

[MasinAD]

Again, what kind of storage/disks? I guess you use slow spinning disks?

Yes, they are HDDs, no SSDs or NVMe or M.2. But: The array is still fast enough as can be seen.