Slow backup to PBS server with compression

[x3ppy]

Hi!

I have two PVE servers and one PBS server. The PBS storage uses a dedicated 10G network. When I start a backup to this location, I can't select the type of compression or disable it - the ZSTD method is used. Copy speed 60 MB/s. If you make a backup copy to local storage without compression, the speed is 200+ MB/s.
I tried tweaking the /etc/vzdump.conf file but didn't get any speed boost when copying to the PBS storage.
Please, any ideas?

PBS:
# dd if=/dev/zero of=/root/test bs=1M count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB, 4.9 GiB) copied, 23.601 s, 222 MB/s

# proxmox-backup-client benchmark --repository common
Uploaded 286 chunks in 5 seconds.
Time per request: 17792 microseconds.
TLS speed: 235.73 MB/s
SHA256 speed: 138.35 MB/s
Compression speed: 208.74 MB/s
Decompress speed: 372.57 MB/s
AES256/GCM speed: 443.11 MB/s
Verify speed: 102.24 MB/s

 

[aaron]

What kind of storage do you use on the PBS and the PVE servers? What disks do you have in use?

Testing storage with dd, and especially from /dev/zero is not an adequate benchmark. Depending on the storage, you basically just benchmark how fast it can compress zeros.

If you want to get a feel on to how benchmark storage, have a look at the ZFS Benchmark whitepaper for example: https://forum.proxmox.com/threads/proxmox-ve-zfs-benchmark-with-nvme.80744/

 

[x3ppy]

PVE: raid 10, HDD 12*600 GB 10k SAS
Storage VM: local-lvm
Storage Backup: Proxmox Backup Server

PBS: raid 5, HDD 6*3 TB 7200 SAS

Backup to local storage (same raid 10) without compression:
INFO: 0% (807.4 MiB of 100.0 GiB) in 3s, read: 269.1 MiB/s, write: 220.9 MiB/s
INFO: 1% (1.4 GiB of 100.0 GiB) in 6s, read: 202.4 MiB/s, write: 192.6 MiB/s
INFO: 2% (2.0 GiB of 100.0 GiB) in 9s, read: 211.3 MiB/s, write: 208.1 MiB/s
INFO: 3% (3.0 GiB of 100.0 GiB) in 16s, read: 153.4 MiB/s, write: 150.2 MiB/s
INFO: 4% (4.2 GiB of 100.0 GiB) in 21s, read: 233.0 MiB/s, write: 230.4 MiB/s
INFO: 5% (5.2 GiB of 100.0 GiB) in 25s, read: 248.2 MiB/s, write: 244.1 MiB/s
INFO: 6% (6.1 GiB of 100.0 GiB) in 30s, read: 199.4 MiB/s, write: 196.8 MiB/s
INFO: 7% (7.1 GiB of 100.0 GiB) in 39s, read: 113.2 MiB/s, write: 112.1 MiB/s
INFO: 8% (8.2 GiB of 100.0 GiB) in 44s, read: 229.2 MiB/s, write: 223.4 MiB/s
INFO: 9% (9.1 GiB of 100.0 GiB) in 47s, read: 275.6 MiB/s, write: 271.4 MiB/s
INFO: 10% (10.2 GiB of 100.0 GiB) in 52s, read: 233.7 MiB/s, write: 229.9 MiB/s
INFO: 11% (11.1 GiB of 100.0 GiB) in 56s, read: 227.8 MiB/s, write: 222.5 MiB/s

Backup to local storage (same raid 10) with compression ZSTD:
INFO: 0% (340.8 MiB of 100.0 GiB) in 3s, read: 113.6 MiB/s, write: 94.8 MiB/s
INFO: 1% (1.0 GiB of 100.0 GiB) in 10s, read: 101.5 MiB/s, write: 86.6 MiB/s
INFO: 2% (2.0 GiB of 100.0 GiB) in 22s, read: 86.1 MiB/s, write: 84.1 MiB/s
INFO: 3% (3.0 GiB of 100.0 GiB) in 36s, read: 73.8 MiB/s, write: 72.3 MiB/s
INFO: 4% (4.0 GiB of 100.0 GiB) in 48s, read: 85.2 MiB/s, write: 84.2 MiB/s
INFO: 5% (5.1 GiB of 100.0 GiB) in 1m, read: 87.0 MiB/s, write: 86.0 MiB/s
INFO: 6% (6.0 GiB of 100.0 GiB) in 1m 13s, read: 74.5 MiB/s, write: 73.2 MiB/s
INFO: 7% (7.0 GiB of 100.0 GiB) in 1m 26s, read: 78.4 MiB/s, write: 77.6 MiB/s
INFO: 8% (8.0 GiB of 100.0 GiB) in 1m 39s, read: 81.7 MiB/s, write: 79.9 MiB/s
INFO: 9% (9.0 GiB of 100.0 GiB) in 1m 53s, read: 72.8 MiB/s, write: 71.4 MiB/s
INFO: 10% (10.0 GiB of 100.0 GiB) in 2m 5s, read: 85.5 MiB/s, write: 83.9 MiB/s
INFO: 11% (11.1 GiB of 100.0 GiB) in 2m 21s, read: 66.2 MiB/s, write: 64.9 MiB/s

Backup to local storage (same raid 10) with compression GZIP:
INFO: 0% (58.0 MiB of 100.0 GiB) in 3s, read: 19.3 MiB/s, write: 16.5 MiB/s
INFO: 1% (1.0 GiB of 100.0 GiB) in 56s, read: 18.3 MiB/s, write: 15.5 MiB/s

Backup to local storage (same raid 10) with compression GZIP pigz = 1:
INFO: 0% (453.1 MiB of 100.0 GiB) in 3s, read: 151.0 MiB/s, write: 132.3 MiB/s
INFO: 1% (1.0 GiB of 100.0 GiB) in 8s, read: 116.7 MiB/s, write: 96.0 MiB/s
INFO: 2% (2.0 GiB of 100.0 GiB) in 15s, read: 146.6 MiB/s, write: 143.1 MiB/s
INFO: 3% (3.0 GiB of 100.0 GiB) in 22s, read: 144.5 MiB/s, write: 141.4 MiB/s
INFO: 4% (4.0 GiB of 100.0 GiB) in 31s, read: 116.0 MiB/s, write: 114.7 MiB/s
INFO: 5% (5.2 GiB of 100.0 GiB) in 39s, read: 147.3 MiB/s, write: 145.0 MiB/s

Backup to PBS with compression ZSTD:
INFO: 2% (204.0 MiB of 8.2 GiB) in 3s, read: 68.0 MiB/s, write: 65.3 MiB/s
INFO: 4% (352.0 MiB of 8.2 GiB) in 6s, read: 49.3 MiB/s, write: 49.3 MiB/s
INFO: 6% (512.0 MiB of 8.2 GiB) in 9s, read: 53.3 MiB/s, write: 53.3 MiB/s
INFO: 7% (668.0 MiB of 8.2 GiB) in 12s, read: 52.0 MiB/s, write: 50.7 MiB/s
INFO: 10% (848.0 MiB of 8.2 GiB) in 15s, read: 60.0 MiB/s, write: 60.0 MiB/s
INFO: 11% (960.0 MiB of 8.2 GiB) in 18s, read: 37.3 MiB/s, write: 37.3 MiB/s
INFO: 13% (1.1 GiB of 8.2 GiB) in 21s, read: 60.0 MiB/s, write: 60.0 MiB/s
INFO: 15% (1.3 GiB of 8.2 GiB) in 24s, read: 49.3 MiB/s, write: 49.3 MiB/s
INFO: 17% (1.5 GiB of 8.2 GiB) in 27s, read: 68.0 MiB/s, write: 68.0 MiB/s
INFO: 19% (1.6 GiB of 8.2 GiB) in 30s, read: 56.0 MiB/s, write: 56.0 MiB/s

Backup (First\Full backup) other VM to PBS with compression ZSTD:
INFO: 1% (3.0 GiB of 301.0 GiB) in 55s, read: 47.3 MiB/s, write: 46.9 MiB/s
INFO: 2% (6.1 GiB of 301.0 GiB) in 1m 56s, read: 51.5 MiB/s, write: 51.0 MiB/s
INFO: 3% (9.0 GiB of 301.0 GiB) in 3m, read: 47.5 MiB/s, write: 47.1 MiB/s
INFO: 4% (12.1 GiB of 301.0 GiB) in 4m, read: 51.9 MiB/s, write: 51.1 MiB/s
INFO: 5% (15.1 GiB of 301.0 GiB) in 4m 59s, read: 52.1 MiB/s, write: 51.9 MiB/s
INFO: 6% (18.1 GiB of 301.0 GiB) in 6m 2s, read: 48.5 MiB/s, write: 48.4 MiB/s
INFO: 7% (21.1 GiB of 301.0 GiB) in 7m 3s, read: 50.2 MiB/s, write: 50.2 MiB/s
INFO: 8% (24.1 GiB of 301.0 GiB) in 8m 7s, read: 48.4 MiB/s, write: 47.8 MiB/s
INFO: 9% (27.1 GiB of 301.0 GiB) in 9m 5s, read: 53.4 MiB/s, write: 50.5 MiB/s
INFO: 10% (30.1 GiB of 301.0 GiB) in 10m 8s, read: 48.7 MiB/s, write: 48.7 MiB/s
INFO: 11% (33.1 GiB of 301.0 GiB) in 11m 10s, read: 50.1 MiB/s, write: 49.9 MiB/s