Configuration suggestion for Proxmox + CEPH

[marcelovvm]

Dear friends, I have a question on assembling a Proxmox + CEPH (I am a newbie in Proxmox... but a have experience in use VMWare) cluster with 7 servers. My servers are a bit old, HPE DL360 and DL380 both generation 7 with P410i RAID controller (SAS 6Gb/s) that does not allow passthrough mode, ie for CEPH I will have to configure each disk to be RAID 0 (or all disks in a large RAID 0 volume) for CEPH can create OSD.

My configuration question would be (I can only put four 2.5 inch disks per server):

configuration 1: 1 x 200GB SAS SSD HPE + 3 X 900GB 10K SAS HDD HPE on each server. So I will have 1.4TB cache (7 X 200GB) and 18.9TB capacity (7 X 900GB X 3);

configuration 2: 1 X Intel Server D3-S4510 240GB SSD + 3 X Intel Server D3-S4510 960GB SSD (SATA 3 - but powered on a SAS 6Gb/s controller that only supports SATA 3Gb/s). So I will have 1.6TB of cache (7 X 240GB) and 20.1TB of capacity (7 X 960GB X 3);

For configuration 1 would be all SAS 6Gb / s disks, but the capacity layer would be spinning disks.

For configuration 2 would be all SSD disks (SATA), but the interface would be 3Gb/s.

Which setting above (or other) do you suggest?

Live long and prosper,
Marcelo Magalhães
Rio de Janeiro - BR

 

[Alwin]

Please see our Ceph precondition section and the Ceph requirements (linked in the docs).
https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_precondition

 

[marcelovvm]

Hi Alwin, I know the requirements to not to use RAID between CEPH FS and hard disk. But server RAID controllers do not allow HBA mode, and the servers no longer have a PCI slot to add an extra controller. So ... what's less worse ... a RAID 0 for each disk on each server or a RAID 0 with all disks on a single server?

Live long and prosper,
Marcelo Magalhães
Rio de Janeiro - BR

 

[Dragon19]

But server RAID controllers do not allow HBA mode

Yes they do. The raid controller (LSI) built directly into my supermicro motherboards support HBA and my physical LSI raid cards support HBA. You may need to flash the firmware but usually the raid card manufacture has that available on their website.

Connect the four drives to your motherboards SATA ports and be done with it...

 

[Alwin]

So ... what's less worse ... a RAID 0 for each disk on each server or a RAID 0 with all disks on a single server?

Neither. A RAID controller simply is build for a different purpose. I can tell you from my own experience, you may have sleepless nights ahead.

 

[marcelovvm]

Yes they do. The raid controller (LSI) built directly into my supermicro motherboards support HBA and my physical LSI raid cards support HBA. You may need to flash the firmware but usually the raid card manufacture has that available on their website.

Connect the four drives to your motherboards SATA ports and be done with it...

Hi Dragon19, note that the controller I mention does NOT have HBA mode. It is not an LSI, but an HPE P410i.

Live long and prosper,
Marcelo Magalhães
Rio de Janeiro - BR

 

[marcelovvm]

Neither. A RAID controller simply is build for a different purpose. I can tell you from my own experience, you may have sleepless nights ahead.

Ok.... and what is your suggestion? Note I Can’t install a HBA controller.

Live long and prosper,
Marcelo Magalhães
Rio de Janeiro - BR

 

[Dragon19]

Hi Dragon19, note that the controller I mention does NOT have HBA mode. It is not an LSI, but an HPE P410i.

Live long and prosper,
Marcelo Magalhães
Rio de Janeiro - BR

Okay? I'm responding to what you said. You said server raid controllers do not have HBA and that's simply not true.

I understand English is not your native language so what you said implied all server raid controllers don't allow HBA passthrough. What you meant was that your server raid controllers don't allow HBA passthrough.

Good luck! And I would suggest removing the raid card from the equation and connecting the four hard drives to the motherboards SATA ports.

 

[Alwin]

Ok.... and what is your suggestion? Note I Can’t install a HBA controller.

Buy new hardware if you want to use Ceph on them. The mentioned hardware was not made for this kind of use.

7 X 900GB X 3);

Aside from the controller, the above calculation is not complete, ((((3 * 900) * 7) / 3) * 0.95) = 5985 GB.
One node will have 3x 900 GB disks, each of it will be an OSD. Totaling in 21 OSDs throughout the cluster. But by default Ceph uses a replica of 3 and an OSD can only be filled to 95%. The "cache" disks, if used for the OSDs DB/WAL, will not be visible as storage.