Proxmox VE Ceph Benchmark 2020/09 - hyper-converged with NVMe

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
4,617
433
88
On the part of encryption, here are some numbers. They weren't included in the paper.

Ceph uses aes-xts for its LUKS encrypted device.
cryptsetup benchmark
AlgorithmKeyEncryptionDecryption
aes-xts512b2201.4 MiB/s 2180.1 MiB/s

And the results of 3x simultaneous rados bench.
rados bench 600 write -b 4M -t 16 --no-cleanup
Single Namespace
Two Namespaces
Four Namespaces
Total time run
600.04​
600.02​
600.03​
Total writes made
426,318.00​
426,762.00​
426,444.00​
Write size
4,194,304.00​
4,194,304.00​
4,194,304.00​
Object size
4,194,304.00​
4,194,304.00​
4,194,304.00​
Bandwidth (MB/sec)
2,841.95​
2,844.97​
2,842.83​
Stddev Bandwidth
19.18​
23.57​
23.95​
Max bandwidth (MB/sec)
3,012.00​
3,048.00​
3,032.00​
Min bandwidth (MB/sec)
2,600.00​
2,584.00​
2,588.00​
Average IOPS
708.00​
710.00​
710.00​
Stddev IOPS
4.80​
5.89​
5.99​
Max IOPS
753.00​
762.00​
758.00​
Min IOPS
650.00​
646.00​
647.00​
Average Latency(s)
0.0676​
0.0675​
0.0675​
Stddev Latency(s)
0.0185​
0.0180​
0.0180​
Max latency(s)
0.2529​
0.2586​
0.2136​
Min latency(s)
0.0149​
0.0166​
0.0155​

rados bench 600 seq -t 16 (uses 4M from write)
Single Namespace
Two Namespaces
Four Namespaces
Total time run
240.91​
241.83​
240.33​
Total reads made
426,318.00​
426,762.00​
426,444.00​
Read size
4,194,304.00​
4,194,304.00​
4,194,304.00​
Object size
4,194,304.00​
4,194,304.00​
4,194,304.00​
Bandwidth (MB/sec)
7,087.03​
7,059.69​
7,098.41​
Average IOPS
1,771.00​
1,763.00​
1,773.00​
Stddev IOPS
29.51​
21.96​
19.70​
Max IOPS
2,132.00​
2,066.00​
2,070.00​
Min IOPS
1,595.00​
1,636.00​
1,645.00​
Average Latency(s)
0.0266​
0.0266​
0.0265​
Max latency(s)
0.1713​
0.1280​
0.1211​
Min latency(s)
0.0056​
0.0056​
0.0056​

1602752011937.png
Very likely that the bigger Epyc CPUs may perform better under encryption.
 

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
4,617
433
88
which of the tuning guides do you recommend for this use case scenario?
Go through all of them. ;) Each will have some information that might be useful to you.
 

velocity08

Member
May 25, 2019
204
10
18
45
Go through all of them. ;) Each will have some information that might be useful to you.
Thanks Alwin

there is a lot of documentation to go through it’s not a small task :(

did you notice any specific pointers for the single socket Epycs by any chance, we‘ve opted for the 7502p which are the single socket models.

I’ll try to slowly start going through the docs.

any pointers or comments would be greatly appreciate.

””Cheers
G
 

Byron

Member
Apr 2, 2019
19
1
8
41
This did not yield any benefit on that system. In atop you could observe that the write performance was divided by the namespaces. And for encryption, the Microns are faster then the aes-xts engine (with that cpu version). The rados bench tests maxed out at ~2.8 GB with any number of namespaces. I suspect it is the way the engine works on Epyc vs Xeon.
Does this depend on the type of drive or the CPU architecture or something else?

It would be super helpful to see CPU load during benchmarking, I'm considering using a 64C/128T single socket for 24 U.2 drives, at 5.3 threads per drive, would it be able to keep up?
 

Byron

Member
Apr 2, 2019
19
1
8
41
What do you mean by that?
We'd put 24 drives in a chassis with 128 CPU threads, giving a ratio of 5.33 CPU threads per NVMe. I've read before that 4 threads per NVMe is the recommended minimum, with 8+ seeming standard practice.

It looks like we'll run in to a CPU bottleneck as far as I can find, however it would be not really an issue to limit our servers to 16-20 drives if we start running into 100% CPU usage.

Any idea what the CPU load numbers were like?
 

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
4,617
433
88
We'd put 24 drives in a chassis with 128 CPU threads, giving a ratio of 5.33 CPU threads per NVMe. I've read before that 4 threads per NVMe is the recommended minimum, with 8+ seeming standard practice.
It's not only the NVMe's that need IO, with all the VM/CT, networking and other services/hardware the threads will be used up quickly. It will be quite a trail and error till you come close to the optimum.

It looks like we'll run in to a CPU bottleneck as far as I can find, however it would be not really an issue to limit our servers to 16-20 drives if we start running into 100% CPU usage.
Probably the memory bandwidth will be the deciding factor, first. And then, in regards to Ceph, the network bandwidth will put an upper limit on the a system.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!