Ceph on Raid0

[hoinz]

Dear Proxmox Team,

we have 3 Dell Servers running Proxmox 6.0. Unfortunately we encountered an issue with the setup of Ceph OSD's on individual drives. The main problem is that the given Perc H710 mini adapter does not allow "IT-Mode" / JBOD Passthrough of individual drives, and so we're stuck with putting each Drive in it's own Raid1 or Raid0 configuration.
This seems to stop us from creating OSD's on the harddrives, since we assume the hardware details of the drives are masked by the raid controller. We are aware of potential performance-problems with intermediate raid-caches and differing block sizes, but would wish for a solution, since the drives themselves are individually adressed afterall.

If not generally applicable, we could patch the raid checks locally aswell. Then it would be really helpful to know where this check happens?

Thank you in advance!

 

[Alwin]

We use ceph-volume underneath, you can directly run it to create OSDs.

For everyone else reading this, look at our precondition and aviod RAID.
https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_precondition

 

[node1]

I run my home and work lab cluster like this. I set raid-0 for each drive, then quick init each one. If you don't initialize them after setting them up in the perc controller they won't be recognized for use by ceph.

 

[Jannoke]

For more adventurous - H710 Minis can be flashed to IT mode, and it seems it is not too hard to achieve.

 

[snpz]

Can you please share how to do that?
We have Dell 720xd servers with perc h710 mini, but there is no option to enable pass-through for disks!

 

[Jannoke]

It seems this is the guide:
https://fohdeesha.com/docs/perc/

There also seems to be video info:
https://www.youtube.com/watch?v=J82s_WYv3nU

I have not done this on dell cards, but have done it Supermicro cards, that are same family LSI chips. All that said - supermicro cards are just easy to flash since there is no limitation on what cards to use. Dell does lock the cards out that he thinks is not HP pci id cards. The guides work around the problem.

NB! You should have physical access to server since It is good idea to remove the battery since it could interfere with flashing process and in IT mode it will not be used anyway and of course it will have minimal wear out if lying on the shelf instead in the working server.

 

[snpz]

Thanks!
Will test this as soon as possible!

 

[snpz]

Thanks!
Will test this as soon as possible!

Worked as expected and servers now using Ceph like they should in JBOD mode - thanks again!

 

[bzg]

Worked as expected and servers now using Ceph like they should in JBOD mode - thanks again!

May I ask if a HDD fails how can you figure out in which slot is the failed HDD? We tested this solution, but as you replace the firmare to IT then the LEDs are not working, so if we had a failed disk we wasn't able to figure iut which one has to be replaced.

 

[snpz]

May I ask if a HDD fails how can you figure out in which slot is the failed HDD? We tested this solution, but as you replace the firmare to IT then the LEDs are not working, so if we had a failed disk we wasn't able to figure iut which one has to be replaced.

Hi!
As far as i remember, leds were working! I sent those servers over to the client, so i have no option to take a look at them any more!

 

[bzg]

Weird, as I know if the IT firmware is installed then the LEDs are not working, we tried on tens of DL360/380 Gen8/9 servers and the same. Thats why I asked. Do wou remember what Gen were those servers?

 

[snpz]

DELL PowerEdge R720xd servers were used!
Srsly - cant remember about leds

 

[Jannoke]

AFAIk the leds only blink (activity) and will not show "dead" disks, because now it's HBA and does not monitor your disk statuses. You might still be able to call identify , but that's about it.

You get led action at all? even for disk read-write?

Was just hit reply then found this: https://forums.servethehome.com/ind...0-h810-mini-full-size-to-it-mode.27459/page-4