Three node Ceph Hyperconverged

[jfenning]

What advice would you give me on running a three node setup? We are a small campus network of 145 employees. I'm currently running two, three node inverted pyramids of doom. One cluster is for disaster recovery at my other data center. I would like to migrate to a Hyperconverged Ceph. I have 33 KVM virtual servers. All but four are Windows so KVM is it for me - no containers. Two are MSSQL and one MS Exchange 2013 but I would not consider this a high production environment. Hopefully I'll be able to migrate to O365 at some point and get rid of local Exchange. I've tested Ceph with my 10k SAS drives and was ok but abysmal performance compared to the NAS storage. Flash seams to be the best way to go but I only had four Samsung sm863's to test with. Further complicating things is that we are implementing a new ERP that is not in production. So I have no real performance numbers for what I'll need. Most of the Windows VM's are 4 core, 8-16G of memory. I have 256GB in each host and have 6TB or storage in use on my Ubuntu NAS's. Sorry for the dump just letting my brain spew.

Since Ceph is better at five + nodes this would be expensive for the Windows licensing. I've been considering just going with local ZFS and replication to keep it simple. I can tolerate the 15 minute lag of ZFS replication. Since the minimum Ceph setup is the max I want to run this gives me some apprehension going down this path. Thanks in advance of any advice anyone can offer.

Currently a subscriber and gladly support Proxmox. It's been good to me for the last 10 years.
Jeff

 

[wolfgang]

Hi,

If you like to go the Ceph way with three nodes, yo need SSD Disk or better NVMe.
Also, I would recommend you a 25GBit Full Mesh Network for Ceph communication.
The main problem at small setups is the latency that why NVMe, in combination with 25GBit, improves the performance enormously even you do not use the Bandwith.
With such a setup, you can stay on three nodes and get excellent performance.
See the Benchmark paper [1].

With ZFS, you can use SAS 10K spinners, but I would recommend you for MSSQL Server to involve an NVMe ZIL (write cache).
The ZIL doesn't to be huge. 2-4GB is enough.

You have to know if you like a sync FS or and async FS is just fine for your use case.
The async variant is less expensive and not as complicated as the sync variant.

1.) https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/

 

[jfenning]

Thanks for the input. Would going with more devices like smaller 960G enterpise sata SSD's and say 8-14 drivers per node work? Like the same effect of more spindles on on a raid set? I would think this would help with rebuilt times as well. Is there a number of OSD's per node that scaling up is of no benefit? I'm going to replace my existing setup with new servers so NVME would be available. The spec's on SAS 12G SSD's look good too. It's just both are more money than SATA and I'm alwasy looking for the "bang for the buck".

Thanks
Jeff

 

[wolfgang]

Would going with more devices like smaller 960G enterpise sata SSD's and say 8-14 drivers per node work?

Yes, this works. You need about one core per OSD.

Like the same effect of more spindles on on a raid set?

Yes, it will be faster if you have more OSD.

I would think this would help with rebuilt times as well.

Sure, and you have less traffic on the network at rebuild time.
And at rebalancing, this is the main problem.

Is there a number of OSD's per node that scaling up is of no benefit?

The limitation is the CPU core count, and the interconnect from Disk to Mainboard.

If you ask me about NVMe or SAS, I would go the NVMe way.
NVMe provides lower latency. Protokoll is designed for parallel workload, and if the Disk support namespaces, you can use it.
The price is the same for NVMe or SAS NAND Disks.