New cluster. What can you recommend?

[Miktash]

We are planning to install a new proxmox cluster for hosting VM's.
I can get a couple Dell R730XD servers but I'm unsure what to buy exactly.
We do want the VMs to be fast so we're going enterprise SSD's only.

For one. I'm not sure which storage option to choose from.

Ceph sounds interesting but it needs at least 3 servers for quorum, increasing the total power usage.
It also requires a 10Gb network, at least.

We will be running maybe 50 VM's which can certainly be run on one of these servers with SSD's.
But we need some kind of fail-over / HA to keep VM's running if there's a hardware failure.

ZFS with replication sounds interesting too because this way we could get away with only 2 servers, right?
One complete server less is a lot cheaper on the electricity bill in the datacentre.
I think VM i/o speeds will be better compared to Ceph?
If we add enough SSD's on both servers we can enable replication and VM's on server1 will be replicated to server2 and vice-versa?
So if one server fails, we can run them on the other server while we're repairing the defective one.
Question: Is live migration possible in such a scenario?


Has anyone got something to recommend?

 

[wolfgang]

Hi,

ZFS with replication sounds interesting too because this way we could get away with only 2 servers, right?

if you like to have HA you need a third quorum member. This could be an qdevice.
https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support

I think VM i/o speeds will be better compared to Ceph?

This depends on the HW that is used.
Because also for zfs asynchrony replication you need 10Gbit as a replication network if you have about 50 VM's.
The pros on ZFS asynchrony replica are:
less complex
less CPU usage.
No need for a 25Gbit network for high IO cluster.
fewer nodes are needed
The cons:
- no delta live-migration when replica runs.
you have to delete the replica and make a full storage migration to use the live mig.
- data are not Synchron on all nodes.

Here is a benchmark about ceph performance.

https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark

If we add enough SSD's on both servers we can enable replication and VM's on server1 will be replicated to server2 and vice-versa?

Yes