Help with cluster/ceph/zfs

[netpana]

Hi guys, i need some advice/clarification about CEPH or ZFS.

I had for 3 years 2 servers with VM’s on it running Cluster with NFS (OMV) without any problem at all. Use the NFS as a Disk image, Container, with that system, can move the VM from one node to the other without any problem. But can’t have HA.

Recently I had the chance to add another server, I was thinking about set up HA, is not really I need it. Can tolerate lose one server, connect to the other and start the machine without any problems until repare one server.

But now I really would like to do HA. And here it comes the headakes.

This are my 3 servers:

PVE 1 HP PROLIANT DL 360:

40 vCpu

96 Gb (RAM)

6 HDD:

HDD 1 - HDD 2 raid1 2 for the Proxmox OS ( 74GB ). (Enterprise disk)

HDD3 – 1Tb (Enterprise disk)

HDD4 – 1Tb (Enterprise disk)

HDD5 – 1Tb (Enterprise disk)

HDD6 – 1Tb (Enterprise disk)



PVE 2 PowerEdge R430:

12 vCPU

64GB RAM

4 HD:

HDD1 - HDD2 raid1 2 for the Proxmox OS ( 74GB ). (Enterprise disk)

HDD 2 – 2 GB (7200 RPM)

HDD 3 – 2 GB (7200 RPM)

DELL POWEREDGE T610

40 vCpu

96 Gb (RAM)

3 HDD:

HDD 1 for the Proxmox OS ( 74GB ) (Enterprise disk)

HDD2 – 1Tb (Enterprise disk) (7200 RPM)

HDD3 – 1Tb (Enterprise disk) (7200 RPM)

The scenario is like this right now:





So far I try CEPH with this scenario and Im getting bad performance, probably because the difference between servers and disks.

Without the 10Gb links, it was giving me a really bad performance on de VM’s, now whit the link between servers at 10G the performance is a bit better but not how it sould be.

( I have an spare server R220 with 32 Gb and 8 vCPU and the VM loads faster on it than the scenario above) (This can not be included on the Cluster because I don’t have more 10 Gb cards and also don’t have a 10Gb port switch)

¿Will you think this scenario can run properly?

On the other hand, I can build ZFS, try it by installing the os and the data together making appropriate raid with Proxmox OS on each node, but I don’t like this way, I want the OS on separated disks.

By knowing this, if I create a ZFS volume lets say with PEV 1 ( 4 x 1 TB DISKS)

¿how can I create a volume ZFS with 4 Gb? Cant figure out how to do the raid

¿how will be the best scenario for enable HA on the Cluster and have the same amount of storage on each server?

¿How can I share all the

I have to add than I have another server HP proliant G5 mini (OMV Nas) Where are all the backups



I know if I do raid0 if one disk fails will lose data but I don’t worry about it because have the backups.

 

[alexskysilk]

¿Will you think this scenario can run properly?

Properly is an indeterminate quality. A 3 node ceph cluster can work, but in your case it will be slow no matter what you do- you have very few osd's and they're HDDs. You also have a problem with lopsided osd distribution; in this configuration (4TB/4TB/2TB) your file system will only be able to provide ~1.6TB of usable space.

¿how can I create a volume ZFS with 4 Gb? Cant figure out how to do the raid

zpool create mypool mirror hdd3 hdd4 mirror hdd5 hdd6

make sure to configure ashift properly for your disks, and use the device uuids for the disks.

¿how will be the best scenario for enable HA on the Cluster and have the same amount of storage on each server?

Not sure what you're asking. Do you mean with zfs storage? you can use zfs replication but this is only good for two servers.

If I were in your shoes, I would pool all the disks to one "filer" and use it as shared storage. Alternatively, you might want to rethink the budget and minimum requirements for this project.

 

[netpana]

thanks for your reply,

alexskysilk​

I thing im going back to my original setup with 2 nodes and one NAS will be easy.

I have this setup for years with no problem and I like to sleep on the nigth