Hardware for Proxmox Cluster with CEPH Storage

[abdulwahab]

Dears, I'm preparing to buy 3 Dell servers for our Proxmox Cluster. I am planning to use CEPH storage for the Cluster. I would like to know whether CEPH storage can work with a RAID controller, but the disks for CEPH will be configured in NON-RAID mode. I want to use RAID controller because I need to use RAID-1 for the OS partition. So, two SAS SSDs will be configured in RAID-1. Four SAS SSDs will be used in NON-RAID mode (pass through mode in RAID controller) for CEPH storage. Let me know whether this will work. Below are the hardware specifications for each server.

1x Dell PowerEdge R740 8x2.5” Rack Server
2x Intel Xeon Gold 6152 Processors
256GB Memory
1x PERC H730P Controller
2x Dell 240GB 6G SATA RI 2.5” SSDs
4x Dell 3.84TB 12G SAS MU 2.5” SSDs
1x Intel X520 2x 10Gb SFP+

 

[gurubert]

I would like to know whether CEPH storage can work with a RAID controller, but the disks for CEPH will be configured in NON-RAID mode.

Yes, this will work.

 

[deepcloud]

Dears, I'm preparing to buy 3 Dell servers for our Proxmox Cluster. I am planning to use CEPH storage for the Cluster. I would like to know whether CEPH storage can work with a RAID controller, but the disks for CEPH will be configured in NON-RAID mode. I want to use RAID controller because I need to use RAID-1 for the OS partition. So, two SAS SSDs will be configured in RAID-1. Four SAS SSDs will be used in NON-RAID mode (pass through mode in RAID controller) for CEPH storage. Let me know whether this will work. Below are the hardware specifications for each server.

1x Dell PowerEdge R740 8x2.5” Rack Server
2x Intel Xeon Gold 6152 Processors
256GB Memory
1x PERC H730P Controller
2x Dell 240GB 6G SATA RI 2.5” SSDs
4x Dell 3.84TB 12G SAS MU 2.5” SSDs
1x Intel X520 2x 10Gb SFP+

Yes it will work but Ceph hates Raid cards and likes the disks exposed directly. not sure if your card is going to pass these disks as native disks like in IT Mode. I would keep all of them in IT Mode (convert the card to IT mode and not in RAID Mode, enough articles available google for it - you will get the procedure) and use ZFS RAID 1 for the OS Disks.

 

[alexskysilk]

but Ceph hates Raid cards

the H730p can be set to jbod (passthrough) mode. works fine (I have many in operation.)

 

[jdancer]

...
1x PERC H730P Controller
...

When I migrated 13th-gen Dells to Proxmox Ceph, I swapped out the PERC for a Dell HBA330 (which is a pure IT-mode controller).

Then I used ZFS RAID-1 to mirror Proxmox. Rest of drives are OSDs.

I don't think the HW RAID PERC can be configured in both HW RAID mode and pass-through mode at the same time. It's either/or.

 

[alexskysilk]

I don't think the HW RAID PERC can be configured in both HW RAID mode and pass-through mode at the same time

Correct. and you wouldnt want to even if that was possible (IBM LSI firmware allows this) because that would require the card to process IO through its controller (and cache) which would introduce all manner of issues for file systems such as ceph.

 

[abdulwahab]

Thanks All. I've received below recommendation from our hardware vendor when I requested the quote. That's the reason why I asked this question.

///

For ZFS and CEPH they both prefer disc to be presented as HBA, so I think the HBA 330 is the way to go here.

///

So, I think I'll stick with HBA330 in this case and use ZFS RAID-1 for OS partitions.

 

[deepcloud]

Thanks All. I've received below recommendation from our hardware vendor when I requested the quote. That's the reason why I asked this question.

///

For ZFS and CEPH they both prefer disc to be presented as HBA, so I think the HBA 330 is the way to go here.

///

So, I think I'll stick with HBA330 in this case and use ZFS RAID-1 for OS partitions.

Thats the right way to go