Ceph 3 Nodes / Mixed Drives

[ShiftCTRL]

Hey all,

I'm setting up Proxmox at a religious institution where they received 3 HP Proliant servers as donations - they're a bit old but suffice for their needs. Basically these will be running their UniFI SDN some VMs/CTs, OpenVAS, etc.. nothing too crazy.

I haven't setup a Ceph cluster before, though reading up on it, its something I'd like to play with and in the process try to earn some points with the One upstairs. I digress.

The servers disks area as follows:
HP Proliant G6 - (4) 146GB SAS Drives
HP Proliant G4 - (2) 146GB SATA Drives
HP Proliant G4 - (2) 146GB SATA Drives

(Node 4 is on the way I believe it's a relatively new HP Proliant server)

Questions:

1) If you were in my shoes, would you go with Ceph or just stick with hardware raid (or etc...)?

[If Yes:] a) Seeing as how two of the servers have 2 drives - would that be cause for concern?

Thanks

 

[sb-jw]

Ouh man...
Normally, i would say, throw it away, but, i will try it.

1.) I do not recommend to mix SAS and SATA Drives for CEPH
2.) You need an HBA (real HBA in IT Mode, not IR Mode with LD/LV)
3.) You need more Drives, you should seperate the OS from the CEPH OSDs
4.) If it is possible i would recommend to use SSDs only

In generall, you should use the same Hardware and not different types with different Sockets and Architectures. If you are not able to put in some more Hardware, i would not recommend to use CEPH on these Nodes.
With Bluestore you need more RAM, so for every TB of Space (RAW) which you have, you should configure 1,5 - 2GB RAM for it. And at this point it think the G4 with a maximum of 12GB RAM should be a Problem.

Do you have some more Informations for us? How many Storage you need? Which Replica you want to use? How many Pools you need? How many IOPS you need? How many VMs shall run on this? What OS/Services shall run in the VMs? Did you plan to expand in the forseable future?

But i doubt you get an acceptable performance out of these Servers - but try it first.

 

[ShiftCTRL]

@sb-jw Thanks for your reply.

Quite frankly I agree with you. Fact is, as this setup is for a non-profit I was trying to save as much money as possible, but in all reality after giving it some thought, the G4s truly are trash. In fact, anyone interested in them and can pick up from NYC - let me know.

The setup:
Node 1- G6, 4 x 146GB SAS, 36GB ECC, Dual Quad-Core Xenon.
Node 2- G6, 8 x 300GB SATA, 8GB ECC [To be upgraded] Dual Quad-Core Xenon.

So what I'm thinking now is - I'll swap out the 4 x 146 with 4 x 300 SATA and just purchase one G6 off e-bay or somewhere.

This still sets me up with a mixed bag, but whatever. I highly doubt the org will be using these much. A few linux VMs for the dev team, various controllers (i.e. UniFi SDN), Traefik, Ansible, OpenVAS (this is the only beast which is expected to be running) and a backup of their media... that's probably about it. Nothing too intense.

It's either that, or I submit a request to purchase 3 almost-new good servers and it is what it is.

I'm being super indecisive here being that the money is coming from member donations and don't really want to buy something they might not need. On the flip side, if they realize this as a great investment - then all this work will be down the drain in the near future and I'll have to do everything from scratch again (not to mention all the migrations).

Thoughts?

 

[sb-jw]

The setup:
Node 1- G6, 4 x 146GB SAS, 36GB ECC, Dual Quad-Core Xenon.
Node 2- G6, 8 x 300GB SATA, 8GB ECC [To be upgraded] Dual Quad-Core Xenon.

So what I'm thinking now is - I'll swap out the 4 x 146 with 4 x 300 SATA and just purchase one G6 off e-bay or somewhere.

Thats sounds better. I would leave 2x 146GB SAS drives per Node for PVE itself and Swap etc. If it is possible try to balance the RAM and Hard Drives over all of your Nodes, so every Node has the same amount.

It's either that, or I submit a request to purchase 3 almost-new good servers and it is what it is.

I think you can work with this. So check the Performance let it run over weeks or month and if it is not enough, then have to buy some newer Hardware.
But i run a CEPH Cluster too on Dell R710 which based on the same socket (2x X5650 and 18x 8GB DDR3 ECC Reg), i run it with an H310 (HBA, flashed with LSI IT FW) and currently 4x 1TB SSDs per Node and 2x 120GB SSDs for OS and only with 4x 1GbE Backend with VLANs and 802.3ad. Currently i running it with 1,8TB of data with an replica of 3 and avg 500 IOPS Write, this morning we reached out the 10k IOPS Read and 3,7k Write at the same time without having any issues. Recovery runs at around 257,3MBs and around 70 Objects per Sec.

then all this work will be down the drain in the near future and I'll have to do everything from scratch again (not to mention all the migrations).

Dont worry, if you have CEPH running you can add the new Nodes to the Cluster, move the VMs from it and remove the OSDs from Crushmap and delete the PVE Node and then shutdown the old Server. But keep in mind, you have to setupt the VMs yet to do this in the Future, so do not use the "host" CPU and some other special stuff.

This will cause a little trouble in the Cluster, but the whole environment should not came to stop and you do not have really more work than initially.

 

[ShiftCTRL]

I would leave 2x 146GB SAS drives per Node for PVE itself and Swap etc. If it is possible try to balance the RAM and Hard Drives over all of your Nodes, so every Node has the same amount.

I like this. The ram def. needs to be upgraded and balancing them shouldn't be too much of an issue.
I have a ton 128GB SSDs going to waste which I'll use for the OS.

But i run a CEPH Cluster too on Dell R710 which based on the same socket (2x X5650 and 18x 8GB DDR3 ECC Reg), i run it with an H310 (HBA, flashed with LSI IT FW) and currently 4x 1TB SSDs per Node and 2x 120GB SSDs for OS and only with 4x 1GbE Backend with VLANs and 802.3ad. Currently i running it with 1,8TB of data with an replica of 3 and avg 500 IOPS Write, this morning we reached out the 10k IOPS Read and 3,7k Write at the same time without having any issues. Recovery runs at around 257,3MBs and around 70 Objects per Sec.

Interesting... only out of curiosity, what's a setup like that run you (roughly)?

Dont worry, if you have CEPH running you can add the new Nodes to the Cluster, move the VMs from it and remove the OSDs from Crushmap and delete the PVE Node and then shutdown the old Server. But keep in mind, you have to setupt the VMs yet to do this in the Future, so do not use the "host" CPU and some other special stuff.

This will cause a little trouble in the Cluster, but the whole environment should not came to stop and you do not have really more work than initially.

I'm really glad you mentioned this, it totally skipped my mind this was an option.
If in the near future there is a justification for it, I'm pretty sure they'll buy whatever I recommend.

Thanks @sb-jw your help has been much appreciated.