[SOLVED] Too slow backup speed from vzdump

[Sarge]

I have proxmox 7.4 in 3-node cluster and Infortrend as shared storage.
Connection via 10 gB cards with multipath.

It all started when I made a backup of a test VM to the PBS server:

I was surprised why such a low speed? 32 GiB in 118 seconds (277 MiB/s)!:-(

After measuring the speed of copying to a remote server, I realized that the problem is local(Backup to local Datastore is low speed too).
Started testing local storage speed from PVE:
rsync --progress -a test2.txt /dev/DATASTORE 5,368,709,120 100% 1.12GB/s 0:00:04 (xfr#1, to-chk=0/1)

fio --filename=/dev/mapper/mpatha --direct=1 --rw=read --bs=1m --size=20G --numjobs=200 --runtime=60 --group_reporting --name=file1

Run status group 0 (all jobs):
   READ: bw=1590MiB/s (1667MB/s), 1590MiB/s-1590MiB/s (1667MB/s-1667MB/s), io=93.3GiB (100GB), run=60079-60079msec

All is OK. Reading and writing speed is quite high.

I tried to make a backup from the console to the same storage, where the disks of the VMs.
it got a little better but far from normal speed:

vzdump 112 --node PVE1 --dumpdir /dev/DATASTORE --mode snapshot --remove 0
INFO:  88% (28.3 GiB of 32.0 GiB) in 56s, read: 1.6 GiB/s, write: 8.0 KiB/s
INFO:  89% (28.7 GiB of 32.0 GiB) in 59s, read: 152.3 MiB/s, write: 0 B/s
INFO:  90% (29.0 GiB of 32.0 GiB) in 1m 2s, read: 95.0 MiB/s, write: 0 B/s
INFO:  95% (30.7 GiB of 32.0 GiB) in 1m 5s, read: 575.3 MiB/s, write: 2.7 KiB/s
INFO: 100% (32.0 GiB of 32.0 GiB) in 1m 6s, read: 1.3 GiB/s, write: 8.0 KiB/s
INFO: backup is sparse: 20.42 GiB (63%) total zero data
INFO: transferred 32.00 GiB in 66 seconds (496.5 MiB/s)

How can i fix this problem?

 

[Chris]

I have proxmox 7.4 in 3-node cluster and Infortrend as shared storage.
Connection via 10 gB cards with multipath.

It all started when I made a backup of a test VM to the PBS server:
View attachment 51823
I was surprised why such a low speed? 32 GiB in 118 seconds (277 MiB/s)!:-(

After measuring the speed of copying to a remote server, I realized that the problem is local(Backup to local Datastore is low speed too).
Started testing local storage speed from PVE:
rsync --progress -a test2.txt /dev/DATASTORE 5,368,709,120 100% 1.12GB/s 0:00:04 (xfr#1, to-chk=0/1)

fio --filename=/dev/mapper/mpatha --direct=1 --rw=read --bs=1m --size=20G --numjobs=200 --runtime=60 --group_reporting --name=file1

Run status group 0 (all jobs):
   READ: bw=1590MiB/s (1667MB/s), 1590MiB/s-1590MiB/s (1667MB/s-1667MB/s), io=93.3GiB (100GB), run=60079-60079msec

All is OK. Reading and writing speed is quite high.

I tried to make a backup from the console to the same storage, where the disks of the VMs.
it got a little better but far from normal speed:

vzdump 112 --node PVE1 --dumpdir /dev/DATASTORE --mode snapshot --remove 0
INFO:  88% (28.3 GiB of 32.0 GiB) in 56s, read: 1.6 GiB/s, write: 8.0 KiB/s
INFO:  89% (28.7 GiB of 32.0 GiB) in 59s, read: 152.3 MiB/s, write: 0 B/s
INFO:  90% (29.0 GiB of 32.0 GiB) in 1m 2s, read: 95.0 MiB/s, write: 0 B/s
INFO:  95% (30.7 GiB of 32.0 GiB) in 1m 5s, read: 575.3 MiB/s, write: 2.7 KiB/s
INFO: 100% (32.0 GiB of 32.0 GiB) in 1m 6s, read: 1.3 GiB/s, write: 8.0 KiB/s
INFO: backup is sparse: 20.42 GiB (63%) total zero data
INFO: transferred 32.00 GiB in 66 seconds (496.5 MiB/s)

How can i fix this problem?

Hi, please check compute performance by running proxmox-backup-client benchmark on the PVE host. Maybe hashing or compression speed are the bottleneck in your case.

 

[Sarge]

Hi, please check compute performance by running proxmox-backup-client benchmark on the PVE host. Maybe hashing or compression speed are the bottleneck in your case.

proxmox-backup-client benchmark
SHA256 speed: 455.36 MB/s
Compression speed: 544.29 MB/s
Decompress speed: 794.90 MB/s
AES256/GCM speed: 1775.57 MB/s
Verify speed: 293.18 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ not tested         │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 455.36 MB/s (23%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 544.29 MB/s (72%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 794.90 MB/s (66%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 293.18 MB/s (39%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1775.57 MB/s (49%) │
└───────────────────────────────────┴────────────────────┘

Is it ok?

 

[Chris]

Yes, although it seems that you are truly limited by the sha256 computation speed, as you can see 455.36 MB/s (23%). What CPUs are you running your PVE cluster on?

 

[Sarge]

Yes, although it seems that you are truly limited by the sha256 computation speed, as you can see 455.36 MB/s (23%). What CPUs are you running your PVE cluster on?

Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           85
Model name:                      Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz
Stepping:                        7
CPU MHz:                         2900.000
CPU max MHz:                     3900.0000
CPU min MHz:                     1200.0000

 

[Sarge]

Is it possible for me to override the default encryption, so that there are no such speed restrictions?

 

[Chris]

No, sha256 is used for digest computation of the individual data chunks, so this is inherit to the backup process itself and not due to encryption.