Suggestion on PVE Cluster with CEPH

[Mike Toupin]

Good day community!

I want to setup a 10 nodes cluster with Ceph RBD + CephFS. I would like to run the VM/container in the Ceph(RBD) and using the CephFS for the shared Storage (files, pictures etc..).

Here is the config for my nodes (Domestic computers) :

10 nodes with same config:
Intel 8 cores HT 3.6Ghz, 64 G RAM non-ECC, 1T SSD M2, 2T Intel SSD, 1T HDD Seagate 7200RPM, UPS. 1x 10Gbit Infiniband + 2x 1Gbit Onboard NIC INtel.

I am not sure what will be the best approach for the cluster and we absolutely need HA and be able to scale to a bigger cluster in few years.

I was wondering if this will be best to using 5 Nodes just for the Ceph (Production HA config) (no Proxmox) and using the other 5 nodes with proxmox or running both proxmox+Ceph on every nodes. I understand that Ceph will likely be faster when using it on more than 5 nodes but is my config will be sufficent for running both Ceph and Proxmox on every nodes? I don't expect a lot of VM/Containers to using all ressource per nodes. Ceph will be used on different network (10Gbit) than proxmox network. I expect to use Infiniband but not sure if the nodes can support the ressource for the OSD/MON....

 

[Mike Toupin]

Nobody can answer that one or give some feedback. Basically I only want to know if it is better to run my Ceph on seperate nodes (without proxmox) and using the rest of my node for Proxmox cluster / or put Ceph on every single nodes (x10 in total).

Which one will be easier for maintain / performance...

Thx!

 

[PigLover]

Its more of a philosophical question than a technical one. There is no single answer to the question. Both approaches are valid and it will really come down to preference more than technology. You can make strong arguments why the fully hyperconverged approach is better - and equally valid strong arguments preferring the segregated approach.

 

[sb-jw]

Do you really need this high amount of servers? It might be a better solution to scale vertical instead of horizontal.

I would choose only 5 nodes but connect them with two power supply, two 10GbE ports (no infiniband). Choose smaller SSDs but more, don't mix so many different types. Choose 1TB enterprise grade SATA SSDs and no spinning disks, normally you do not need an M2 SSD, but this one you can use for the PVE itself or choose two 120GB SSDs for higher availability.

I would choose hardware which is more mainstream, what is easier to maintain and more universal. So infiniband is a bit special, you need Adapter, Switches and Cables - sure for other standards too, but you can use the server in every datacenter in a shared rack if you do not have some special shit in it. And you can reduce spare parts, if you use not so many different parts. The troubleshooting is much easier, if you have many different hardware.

 

[AlexLup]

Do you really need this high amount of servers? It might be a better solution to scale vertical instead of horizontal.

I would choose only 5 nodes but connect them with two power supply, two 10GbE ports (no infiniband). Choose smaller SSDs but more, don't mix so many different types. Choose 1TB enterprise grade SATA SSDs and no spinning disks, normally you do not need an M2 SSD, but this one you can use for the PVE itself or choose two 120GB SSDs for higher availability.

I would choose hardware which is more mainstream, what is easier to maintain and more universal. So infiniband is a bit special, you need Adapter, Switches and Cables - sure for other standards too, but you can use the server in every datacenter in a shared rack if you do not have some special shit in it. And you can reduce spare parts, if you use not so many different parts. The troubleshooting is much easier, if you have many different hardware.

I agree with this fully, however the more nodes he has, the faster ceph will be. Also, make sure to factor in one NVME for the ceph journal and maybe a NVMEs for cache tier ?

 

[guletz]

I am not sure what will be the best approach for the cluster and we absolutely need HA

Why do you think that you need this? Could you describe for your case one or two cases that in your opinion need a HA?