[SOLVED] Plan for 3 Proxmox clusters for high availability and Ceph?

[danman]

Hi

I need a little bit help regarding a set up which will contain 3 proxmox server for high availability.

I've currently running 1 pve as a main server which contains the following hardware:

1 Proxmox
- 128GB Ram
- AMD Ryzen 5 2600 Six-Core Processor
- 512GB NVMe
- 2 x 2TB Seagates HDD (barracuda)
- 2 x 4TB Seagates HDD (IronWolf)
- 2 x 8TB Seagates HDD (barracuda via USB)

2 Proxmox
The second one will arrive sometime next week. It is a Dell Optiplex 7040 micro
- 32GB Ram
- i5-6500T Four-Core Processor
- 4TB NVMe

3 Proxmox
The third one is still in plan actually. PVE 1 & 2 are in one room and the 3rd one will be in the garage.
The idea is to use this one for high availability and backup storage.
My idea is to get just the AMD Ryzen 5 5500 CPU and a cheap board for it with 2 m2 connectors maybe and split the hardware between 1 & 3. 64 Ram each and the main "big" disks in PVE 3. Maybe 2 NVMe extras for PVE 3 as well.

Would be Ceph a good idea or is this too much of a bottleneck? Especially PVE 3 will be only WIFI for now.

Thanks!

 

[gurubert]

Would be Ceph a good idea or is this too much of a bottleneck? Especially PVE 3 will be only WIFI for now.

Technically it may work but you would get no usable performance out of such a system.

 

[danman]

Technically it may work but you would get no usable performance out of such a system.

What exactly would be the issue?
Optiplex
Wifi
HDDs
...
Would it be better to have each node their own storage via zfs and is HA still possible for pve 3?

 

[UdoB]

Disclaimer: I only have a _test_-cluster with Ceph, I am *not* an actual Ceph-user.

Please read https://docs.ceph.com/en/quincy/start/hardware-recommendations/#minimum-hardware-recommendations

• you really want to have several OSDs per node - one failing OSD should not render the whole node to be "failed"
• you need some "MON"s
• you need some "MDS"s
• and usually it is recommended to have at least 10 GBit/s - if possible dual NICs for redundancy. But for a homelab this might be overkill...

Nevertheless: with 3 nodes * 3 OSD/node * (4GB+ per daemon) plus 3 Mons * (2-4GB+ per daemon) + 3 MDS * (2GB+ per daemon) you will consume more something like 3*16 _only_ for ceph. That Ram is gone = it is not available for VMs.

Please note also that 3 nodes is the absolute minimum; I would begin this journey only if 5 (or more) nodes are on the roadmap.

Ceph is really, really interesting and I want to have it too. But ZFS-replication - with all its drawbacks - is better suited for my personal usecase.

Best regards - just my 2€¢

 

[danman]

Ok. Thank you both!
I'll stick to zfs.