Sizing for an HA cluster with Ceph

[Falk R.]

@pille99
From my point of view, vSAN does not make sense. With vSphere8, it has once again become significantly more performant, but still does not quite come close to Microsoft S2D.
In addition, vSAN is far too expensive. If vSAN were not so expensive, it would be a great product.

I'm also curious about @tholderbaumRTS reasons, because everyone has different requirements and therefore different test results.

 

[pille99]

i got an offer from vmware. a 3 node cluster with esxi, vcenter, vsan and nsx = 130.000 dollar.

 

[Falk R.]

i got an offer from vmware. a 3 node cluster with esxi, vcenter, vsan and nsx = 130.000 dollar.

And what hardware was in the offer?
CPU, RAM, disk and network?

I think a Proxmox cluster costs less than half.

 

[tholderbaumRTS]

hello, can i ask about the considerations ? for what reason ? in your point of view, what is the disadvantages ? with what solution you are going ? what was your main point for the solution you are going ?
dont get me wrong / i dont want yo question your decissions. as you manage bigger environment its always nice to learn something new and see some points on different views. did you check vsan ?

The issues is that CEPH really works when you have more hosts. Because we were looking at a phased approach, at the beginning, we would only have 3 hosts. CEPH cannot give you the fault tolerance we need at that size.

Phase 1 = 3 hosts
Phase 2 = 7 Hosts

So, CEPH is still in the plans, we just want to wait until we have enough hosts to properly support it. Which for us, should be 7 hosts.

There is value in our setup for some of our VMs to use a replicated VM instead of shared storage. So we will have both capabilities in our hosts.

 

[tholderbaumRTS]

hello, can i ask about the considerations ? for what reason ? in your point of view, what is the disadvantages ? with what solution you are going ? what was your main point for the solution you are going ?
dont get me wrong / i dont want yo question your decissions. as you manage bigger environment its always nice to learn something new and see some points on different views. did you check vsan ?

I didn't answer your second question.

We are implementing an infrastructure refresh. We are currently Hyper-V, but we will be transitioning to Proxmox/linux KVM. I had thought that we would be using VMware, but the rough calculation was that over 7 nodes, VMware would run is the better part of a $250,000 after we get done licensing everything we would need.

Simply put, VMware is too expensive and does not present enough features that would be worth investing a quarter mil on. I can buy a lot of hardware for that.

 

[tholderbaumRTS]

i got an offer from vmware. a 3 node cluster with esxi, vcenter, vsan and nsx = 130.000 dollar.

They quoted me too. 7 node cluster, and vsan. $217,000 plus maintenance.

 

[pille99]

standard dell with dal xeon, 256 ram, and some ssd,

They quoted me too. 7 node cluster, and vsan. $217,000 plus maintenance.

they are pretty sick with the prices. i mean, 200.000 dollar, you get 20 good dell server which yo can build a very sweet and high performant cluster with KVM (proxmox is based on kvm, and kvm is the only hypervisor software which runs almost like native, as the speed of the available hardware)

 

[bbgeek17]

There is an old saying that gets applied to various companies depending on the vertical, the first time when I came across it was about storage:
"No one ever got fired for buying EMC". Through series of acquisitions it can be extrapolated to "No one ever got fired for buying EMC, Dell, VMware".

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[qhash]

So since you guys helped me out, I present to you my phase 1 proof of concept reference design.

I have this already built in a virtual lab (obviously without the hardware)

I am further exploring the networking setup, I am planning on each client having an isolated VM.

Notice how I have dedicated NICs for cluster traffic? That is a trick I picked up from VMWare land.

Phase 1 will be three nodes in N+1 for about 1/3 of my load. Then we will expand. Full deployment will be 7 nodes as N+2. Max deployment is 15 nodes. Then I build another rack.

the VMware bill for this would be $350,000. That is money that is going straight to my pocket. We are going with the top end support license to support the project.

With Broadcom taking over VMware and with Microsoft just flat out ripping people off with the per core datacenter license, I see a bright future for Proxmox.

I like this approach. We are looking to replace and upgrade our old VMWare cluster. I do not fully understand your storage. I mean, I understand how it keeps data replicated between the nodes, but how are drives in a single node joined to create storage for VMs? or they are not? And what is capacity storage for? Backup or also file/mail/this type servers? Do you stack your switches or they are totally seperate networks?

 

[Falk R.]

For Ceph you should run switches in the stack or with MLAG.
The disks (OSD) each have their own Deamon running. The traffic runs via these services and not via a central service as with vSAN, for example.
With the pool, you define how the data is distributed and you cut out parts of the large block pool for the VM disks as with SAN storage LUNs.
For VMs, I only use NVMe disks and the network should have a lot of bandwidth and low latencies.
So not less than 25GBit for productive. I personally only configure 100GBit networks for Ceph.

 

[qhash]

Thank you. Wrote too fast - by stacking I meant MLAG. I never stack iSCSI switches and I would not stack ceph data network either. You wrote "stacking", however. Seeing you do 100Gbps networks, dont think you can't afford MLAG. Why stacking then?