Slow PBS performance despite fast hardware

[phil2987]

Hello,

I have PVE and PBS hosts connected to each other using 25Gbe Mellanox MCX4121A-ACAT NICS through a 100Gbe Mikrotik CRS504-4XQ switch. The backup server has 2 stores: 1 is a RAID1 mirror (2x 16TB Ironwolf Pro) on a Dell H730 controller and the other is a Lexar NVMe plugged into a PCIe card slot (no RAID or anything). I expected much better performance. When backing up a VM, I'm only getting around 55MiB/s, regardless of which store I send to. At the very least, I expected the NVMe store to be MUCH faster than it is, but it seems slow. iperf3 shows normal numbers.

EDIT: I forgot to mention: The SOURCE VM is also on a locally-attached NVMe drive.

Some numbers are below:


root@vhost1:~# iperf3 -c 10.200.200.199
Connecting to host 10.200.200.199, port 5201
[ 5] local 10.200.200.200 port 35828 connected to 10.200.200.199 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.70 GBytes 14.6 Gbits/sec 49 1.67 MBytes
[ 5] 1.00-2.00 sec 2.64 GBytes 22.7 Gbits/sec 145 1.46 MBytes
[ 5] 2.00-3.00 sec 2.73 GBytes 23.4 Gbits/sec 252 1.64 MBytes
[ 5] 3.00-4.00 sec 2.00 GBytes 17.2 Gbits/sec 112 1.21 MBytes
[ 5] 4.00-5.00 sec 2.28 GBytes 19.6 Gbits/sec 47 1.39 MBytes
[ 5] 5.00-6.00 sec 2.73 GBytes 23.4 Gbits/sec 259 1.36 MBytes
[ 5] 6.00-7.00 sec 2.72 GBytes 23.4 Gbits/sec 146 1.59 MBytes
[ 5] 7.00-8.00 sec 2.73 GBytes 23.4 Gbits/sec 185 1.61 MBytes
[ 5] 8.00-9.00 sec 2.73 GBytes 23.4 Gbits/sec 169 1.63 MBytes
[ 5] 9.00-10.00 sec 2.33 GBytes 20.0 Gbits/sec 92 1.57 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 24.6 GBytes 21.1 Gbits/sec 1456 sender
[ 5] 0.00-10.04 sec 24.6 GBytes 21.0 Gbits/sec receiver

iperf Done.
root@vhost1:~#


root@bkup:~# proxmox-backup-client benchmark --repository nvme1
Password for "root@pam": ********
Uploaded 380 chunks in 5 seconds.
Time per request: 13213 microseconds.
TLS speed: 317.44 MB/s
SHA256 speed: 270.31 MB/s
Compression speed: 358.82 MB/s
Decompress speed: 609.03 MB/s
AES256/GCM speed: 1188.88 MB/s
Verify speed: 196.06 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 317.44 MB/s (26%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 270.31 MB/s (13%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 358.82 MB/s (48%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 609.03 MB/s (51%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 196.06 MB/s (26%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1188.88 MB/s (33%) │
└───────────────────────────────────┴────────────────────┘
root@bkup:~#


()
INFO: starting new backup job: vzdump 1715 --mode snapshot --storage bkup-nvme --node vhost1 --remove 0 --notes-template '{{guestname}}'
INFO: Starting Backup of VM 1715 (qemu)
INFO: Backup started at 2023-03-17 10:36:23
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: s1715
INFO: include disk 'scsi0' 'shared-nvme1:1715/vm-1715-disk-0.qcow2' 40G
INFO: creating Proxmox Backup Server archive 'vm/1715/2023-03-17T14:36:23Z'
INFO: starting kvm to execute backup task
INFO: started backup task '0ebf4136-da83-453e-a5cb-b5343d7323e8'
INFO: scsi0: dirty-bitmap status: created new
INFO: 0% (196.0 MiB of 40.0 GiB) in 3s, read: 65.3 MiB/s, write: 61.3 MiB/s
INFO: 1% (492.0 MiB of 40.0 GiB) in 7s, read: 74.0 MiB/s, write: 73.0 MiB/s
INFO: 2% (860.0 MiB of 40.0 GiB) in 13s, read: 61.3 MiB/s, write: 61.3 MiB/s
INFO: 3% (1.2 GiB of 40.0 GiB) in 21s, read: 46.5 MiB/s, write: 46.5 MiB/s
INFO: 4% (1.6 GiB of 40.0 GiB) in 28s, read: 62.9 MiB/s, write: 62.9 MiB/s
INFO: 5% (2.0 GiB of 40.0 GiB) in 34s, read: 62.7 MiB/s, write: 62.7 MiB/s
INFO: 6% (2.4 GiB of 40.0 GiB) in 42s, read: 52.5 MiB/s, write: 52.5 MiB/s
INFO: 7% (2.8 GiB of 40.0 GiB) in 49s, read: 58.9 MiB/s, write: 58.9 MiB/s
INFO: 8% (3.3 GiB of 40.0 GiB) in 56s, read: 66.3 MiB/s, write: 66.3 MiB/s
INFO: 9% (3.6 GiB of 40.0 GiB) in 1m 3s, read: 53.1 MiB/s, write: 53.1 MiB/s
INFO: 10% (4.0 GiB of 40.0 GiB) in 1m 10s, read: 55.4 MiB/s, write: 55.4 MiB/s
INFO: 11% (4.4 GiB of 40.0 GiB) in 1m 18s, read: 51.0 MiB/s, write: 51.0 MiB/s
INFO: 12% (4.8 GiB of 40.0 GiB) in 1m 26s, read: 54.5 MiB/s, write: 54.5 MiB/s
INFO: 13% (5.2 GiB of 40.0 GiB) in 1m 34s, read: 53.5 MiB/s, write: 53.5 MiB/s
INFO: 14% (5.6 GiB of 40.0 GiB) in 1m 41s, read: 54.9 MiB/s, write: 54.9 MiB/s
INFO: 15% (6.0 GiB of 40.0 GiB) in 1m 48s, read: 56.6 MiB/s, write: 56.6 MiB/s
...

 

[ness1602]

Lexar nvme is not fast, it's just benchmark fast, otherwise use enterprise disks if you want enterprise speed.

 

[phil2987]

Oh come on. The Lexars are plenty fast. In other systems, they are at least several hundred megabytes per second. They may not be the fastest in the world, but they are certainly faster than spinning drives. I also tried with Samsung, as well as 25Gbps NFS stores for both source and destination, and the data rate remains the same. I'm stumped.

 

[Dunuin]

How did you benchmark those "several hundred megabytes per second"? PBS isn't doing sequential reads/writes. Its random IO with max 4MB file sizes (usually more like 1-2MB per chunk). So its more like reading/writing some TBs of jpegs in random order.

 

[ness1602]

PBS is an enterprise software,and if you want enterprise speeds, you need to buy enterprise equipment.
This is not mandatory,i have a few PBS for a client which uses random 2-4TB disks, and it works with thousands of snapshots(yes thousands). Okay, verification takes time, restore takes time, but it is a off-the-shelf backup solution,and it works.
If you want speed,and you don't want to buy enterprise equipment, spin up NAS, get NFS/SMB(usually OMV or Freenas),and there you have it.
You just loose deduplication.

 

[phil2987]

Not sure if this helps. I ran fio (read) on the vhost and fio(write) on the backup server (local nvme to local nvme). The results were:

vhost1 read: 3323 MiB/s
bkup write: 2382 MiB/s


why am I still getting bad rates when backing up a vm?


root@vhost1:/mnt/pve/nvme1# fio --ioengine=libaio --direct=1 --name=test --filename=.test --bs=4m --iodepth=64 --size=128G --readwrite=read
test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [R(1)][28.3%][r=3323MiB/s][r=830 IOPS][eta 00m:33s]

root@bkup:/mnt/datastore/nvme/vm# fio --ioengine=libaio --direct=1 --name=test --filename=.test --bs=4m --iodepth=64 --size=128G --readwrite=write
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [W(1)][22.8%][w=2382MiB/s][w=595 IOPS][eta 00m:44s]



INFO: VM Name: testtttt
INFO: include disk 'scsi0' 'vg-local-nvme:vm-997-disk-0' 40G
INFO: creating Proxmox Backup Server archive 'vm/997/2023-03-19T23:52:06Z'
INFO: starting kvm to execute backup task
INFO: started backup task 'fc5de407-adaf-407c-9d9d-523c49e9a772'
INFO: scsi0: dirty-bitmap status: created new
INFO: 0% (212.0 MiB of 40.0 GiB) in 3s, read: 70.7 MiB/s, write: 66.7 MiB/s
INFO: 1% (420.0 MiB of 40.0 GiB) in 6s, read: 69.3 MiB/s, write: 69.3 MiB/s
INFO: 2% (820.0 MiB of 40.0 GiB) in 12s, read: 66.7 MiB/s, write: 66.0 MiB/s

 

[narrateourale]

Try running the fio benchmark time based for at least 10 minutes (600 seconds) and check the results again. Consumer SSDs are good as long as they can cache, but one the cache is full, performance will drop. Testing for that long should make sure, that the SSD is getting its cache full.

Then compare the min, max and mean + stddev values to see in which range it is.

 

[Dunuin]

You can also try "randread" and "randwrite" instead of "read" and "write" when using fio. PBS data is deduplicated, so IO won't be purely sequential. And a bs of 1M or 2M would be more reasonable.

How is the CPU utilization? PBS will need a lot of CPU power to do the hashing, zstd compression and SSL encryption.

 

[ness1602]

This is how i usually test fio, mix of read and write:
sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75