PBS 2.2-3 slow datastore

[da-alb]

Hi,

I'm testing PBS alongisde PVE on the same host but I'm experiencing some perfomance issue (I'm guessing).

The PBS datastore is on raid0 pool of 2 Sata SSDs, for this test I'm using a pair of Samsung 860 EVO and the benchmark of it is the following:

Uploaded 345 chunks in 5 seconds.
Time per request: 14568 microseconds.
TLS speed: 287.91 MB/s
SHA256 speed: 364.14 MB/s
Compression speed: 409.32 MB/s
Decompress speed: 605.83 MB/s
AES256/GCM speed: 1173.18 MB/s
Verify speed: 246.36 MB/s

The server has a pair of Xeon E5-2660 v3 and 96 GiB of DDR4 RAM and a 10 GB fiber NIC.

Yet the backup and the restore times are way too high, I'm not hitting 10 gigs nor in read or write but a little more than 200 mib/s.

The PVE that I'm backing up from has a pair of Xeon Gold 5218, 192 GiB of DDR4 RAM and Ceph with nvme and a 10 GB fiber NIC (dedicated for the backup).

Any idea? Thanks

 

[dcsapak]

seems you're cpu bottlenecked here:

TLS speed: 287.91 MB/s
SHA256 speed: 364.14 MB/s
Compression speed: 409.32 MB/s
Decompress speed: 605.83 MB/s
AES256/GCM speed: 1173.18 MB/s
Verify speed: 246.36 MB/s

your SHA256 speed is not very high (e.g. on a ryzen 3700x here i get ~2GiB/s for sha256)
which seems to drag down your operations... everytime a chunk is read (or verified) we must calculate the sha256 checksum so that seems to be your bottleneck here
(i mean your compression/decompression speeds are also not that high, but at least the decompression seems reasonable)

 

[da-alb]

seems you're cpu bottlenecked here:


your SHA256 speed is not very high (e.g. on a ryzen 3700x here i get ~2GiB/s for sha256)
which seems to drag down your operations... everytime a chunk is read (or verified) we must calculate the sha256 checksum so that seems to be your bottleneck here
(i mean your compression/decompression speeds are also not that high, but at least the decompression seems reasonable)

So what are you suggesting? Try another server with high single core perfomance? Is PBS benefitting from higher core counts or higher perfomance in single core?

Thanks

 

[dcsapak]

Is PBS benefitting from higher core counts or higher perfomance in single core?

both, but a single backup/restore does not benefit much from more cores since those must mostly be serialized (e.g in qemu io to a single virtual disk is only single-threaded)
but you should see it scale up as you'd do multiple backups/restores in parallel (or gc in parallel to verify, and so on)