Advice needed: Cluster FS

[nuclearhosting]

Hello guys,

preparing a new cluster setup and considering what FS / storage to use. There will be 3 servers: 2 nodes (let's say node1,node2) with the same disks (2xSSD + 4xHDD) and one will have only 2xSSD (node3).

The goal is to have HA load-balanced VM. So, for this VM I need to have this VM on the node1 and node2 with shared storage mounted (1 disk where shared images are). The node3 will contain another VM for MySQL only. After a lot of readings, I am considering to use DRBD with LVM + NFS or Ceph. The questions are:

• is possible to use ceph (never used before) on setup where not all 3 nodes will have the same disks, and only two of them will?
• is possible to mount ceph on the two clients (vm on node1 and vm on node2) at the same time?
• what about Ceph speed? It will be high-load setup, the shared storage is considered to store a lot of images (small file), so I am worrying not to cause bottle-neck
• if ceph is not suitable, I am thinking of DRBD+LVM on some another VM (in HA) which will provide NFS for VMs on node1 and node2. What do you think about this option? What about the speed?
• do you have any other possible advice on how to fulfil described setup demands?

The problem is, there is load-balancing demand. If only HA I will do that by using ZFS but the client expects big traffic increasement so we need to have more resources ready in load-balancing setup. Also, there is not much space for testing and then choosing the right solution. Time is running out.

Thanks in advance.

 

[Dominic]

Hi,

We did a Ceph benchmark in September, which you can find on our website.
Ceph is already integrated into Proxmox VE and thus relatively easy to get somewhat working. Thus, I suggest that you create a virtual nested cluster on some (spare) PVE machine that simulates your setup. Of course, you can also try to install DRDB there. I think this would clear up a lot of questions.

 

[nuclearhosting]

Hi Dominic,

because I had no free HW to provide testing prior, I have decided to use NVMe drives instead of standard HDD because I was afraid about lack of performance using Ceph/DRBD. After providing some testing on new NVMe-only HW using ceph cluster (plan B was DRBD+NFS) we are fully satisfied with the performance and we keep using Ceph in this setup.

Anyway, many thanks for your benchmark test. There is a lot of very useful information! Good job!

 

[Dominic]

Great that it works for you! NVME instead of spinning disks was certainly a good decision.

And thank you for the kind words