Ceph Hyperconverged on Blade servers

[Paspao]

Hello,

I was considering a Dell M1000E enclosure + blade servers, but I see blade server support 2 disks only.

If I start with a cluster of 4 nodes I will have only 4 Ceph OSDs (4 disks are for boot).

Do you have experience or suggestions for Hyperconvergence on blade servers?

Thank you.
P.

 

[navigator]

Unfortunetly this didn't get any answer. But i'll tell you my case today.

We just bought a bunch of Huawei E9000 that looks equivalent to this setup. We have 14 blade per/chassis with the same setup 2.5in disks.

One possibility is to use NVMe drives as here in this documentation is allegedly affirmed:

Running multiple OSDs on a single SAS / SATA driveis NOT a good idea. NVMe drives, however, can achieve improved performance by being split into two or more OSDs.

So now i am trying to figure it out how this can be done. I'll buy some NVMe drive for testing this. And build a basic 3 node Cluster for testing while the other 11 nodes will be set with local storage and replication temporarly.

Another topic found was this one where someone affirms that this type of configuration is not adequate. So i am undecided.

Finally, if 1OSD/NVMe shows to be more performant i'll simply go that way because with 14 blade per Cluster I really won't care if one node or even 4 or 5 are down because HA would migrate everything and then thing should be pretty consistent.

Anyway hope this topic gets up because today server on the edge are mostlikely to be chassis servers with bladed due to the backplane switching and highperformance and i haven't seen any discussion here about it and really gets my attention on how people are doing this.

Anyways hope this helps you eventought some years latter.

Please if you got any experience with this let me no.

 

[aaron]

For now, if you want to create multiple OSDs / disk, you need to create them with ceph-volume lvm batch --osds-per-device X /dev/nvme….

If you want to build a Ceph cluster with blades, keep in mind that failure domains might be a bit different, depending on what is locally per blade and what is handled in the chassis. Power, network switches and so on.

If you run a size/min_size of default 3/2, Ceph will only be fine as long as it is losing one node at a time with enough time to recover until the next node loss.

 

[navigator]

Aaron, in the case scenario:

E9000 Huawei with 7Blades CH121 (2Disks per Blade (one disk for Proxmox OS + one disk for CephOSD)) wich would be an appropriate size/min settings?

PowerSupply and Switch is centralized for this type of system (6 PSU + 2Switches) I supouse 1 node failure is to risk, so lets say 2 node failure in 7 available.

Can you help me on that? Im so new to this type of equipment, always worked with consumer desktops pc in my systems. Have more doubts and questions than certain & confidance. Sorry

 

[jsterr]

So basically you need to find out, how many power supplys you can loose so everything is still working on all nodes? What happens when one of the 6 psus dies? Is a blade offline, are two blades offline? You need to find that out, so you know how many blades you would loose on PSU death or "switch"death.

With 3:2 on a 7 node setup, you can only loose one node at a time. If you complete recover then from that node-loss, you can loose another one. an so one, until you only have 3 nodes left or one of the disks rearches 90-95% (look up mon osd full ratio).

You can increase the number of blades you can loose by increasing the SIZE in ceph. 4:2 for examples tolerates 2 blade failures at the same time. You can easily think about it SIZE - MINSIZE = Server / Blades you can loose at the same time without downtime.

Edit: What are you hoping to archive with this blade setup? Why not using normal servers?

 

[navigator]

Beacuse those server were already bought and they offer 150 Watts consumption/Blade Total 2.2Kw Chassi for sppining up 512RAM/Blade (Total 7.2Tb RAM) and come with 2x24 Cores CPUS by Blade, total of 672 cores 3.2Ghz. Energy efficency is really good and also We have two NFS outside for Backups and Large Storage.

Gonna addopt those weights recomended by you

 

[alexskysilk]

Just fyi, each of your cpus are probably ~135W under load. and ddr4 ram consumes approx 2W/8g for LR, 3W for rdimms (more for quad rank.) and we havent even counted any NICs, HBAs, South bridges, etc.

Even if your blades take NO Power (which is impossible), you're not at 150w. Just sayin.

And these are meant for high density compute, not for storage. ideally storage is provided from outside the chassis.

 

[navigator]

YEs alex, i also have 2 CH222 with 14 Drives Each. I could spinup the ceph server there?

Yes de consumption is just idle.

 

[alexskysilk]

i also have 2 CH222 with 14 Drives Each. I could spinup the ceph server there?

I know next to nothing about Huawei devices, but as long as you can set the raid controller into hba or passthrough mode yes you could. Bear in mind that you want a minimum of 3 CH222 nodes to have sufficient targets for a 3:2 replication pool.