Where will be the bottleneck?

[mrtx]

Hi there,
I am just buying hardware for a proxmox6-cluster.
I choose four identical nodes:
1 x AMD Epyc 7232
4 x 16 GB DDR4-3200 ECC-RAM
2 x SSD 240 GB Intel S4510 (One for Linux-OS, one for ceph-calculations)
3 x SSD 960 GB Intel S4510 (ceph-storage for virtual machines)
2 x 1 GBit-NICs (lan)
2 x 10 GBit-NICs (sync)

I connect them internally with two 16 Port 10 GBit Switches.
I use the recommended CEPH-Setting of 3 Replicas (how many pgs?)

Questions:
1. Is there something obviously wrong with that setup? (I am aware that one node should not be a monitor)
2. Is it better for the performance to use two SSD with 2 TB instead of three with one TB? (I found inconclusive information about that. I have more than enough storage, that is not important for the decision)
3. Is it smarter to use three nodes with 128 GB RAM each instead of four nodes with 64 GB RAM?
4. How much RAM will there be for virtual machines? How much would it be with 128 GB RAM?
5. How much storage will there be for virtual machines?

I am happy about hints from the practice.

mrtx

 

[LnxBil]

2. Is it better for the performance to use two SSD with 2 TB instead of three with one TB? (I found inconclusive information about that. I have more than enough storage, that is not important for the decision)

In general, the more devices the better but with SSD the limiting factor is the backplane, in your case 3 should be better because you will probably not saturate your backplane with only 3 disks, but this depends on the backplane. Do you have 12 GBit SAS?

1. Is there something obviously wrong with that setup?

I'd go with the default "everything redundant", so RAID1 on your boot drive - and what is a "ceph calcuclation" disk, and why only one? If you mean a OSD cache disk, then this is IMHO useless because you have the same type of drive as your data. You will not have a benefit from this besides separation. Normally you use this as a cache layer for slow spinners.

3. Is it smarter to use three nodes with 128 GB RAM each instead of four nodes with 64 GB RAM?

With respect to quorum, probably. Odd numbers are better than even ones.

5. How much storage will there be for virtual machines?

Depending on your replications desires, but If you want to have 2 replicas, you will end up with the space of one node in the three node setup.

 

[mrtx]

In general, the more devices the better but with SSD the limiting factor is the backplane, in your case 3 should be better because you will probably not saturate your backplane with only 3 disks, but this depends on the backplane. Do you have 12 GBit SAS?

Yes. I want to be able to add some storage later.

I'd go with the default "everything redundant", so RAID1 on your boot drive - and what is a "ceph calcuclation" disk, and why only one? If you mean a OSD cache disk, then this is IMHO useless because you have the same type of drive as your data. You will not have a benefit from this besides separation. Normally you use this as a cache layer for slow spinners.

Yes, I mean the OSD-cache. Is it ok to put the OSD-cache on the OS-disk? How much space do I need to plan? Does it even make sense to separate OSD-Cache?

Thank you for sharing!