CEPH and HA Configuration Help

[kenny82]

Good afternoon,

I am going to install a Proxmox cluster in HA, and the virtual machines will be hosted in CEPH and NAS (NFS). The most important ones will be in CEPH, and the rest in the NAS.
My doubts are about latency when using the CEPH network for everything. We’ve based this network on the existing manual to set up a Mesh network with FRR. My network setup is as follows:

• Mesh network at 100G
• Link network at 10G with LACP to the switch where the NAS and Proxmox Backup are located
• Network 1, this is a service network for the virtual machines (OT)
• Network 2, another service network (IT)
• Network 3, another network for the VMs

I had planned to use the networks in this way:

• Mesh network 100G: public and private CEPH network, Corosync (link 0 - higher priority)
• 10G LACP network: Corosync (link 1 - lower priority), NAS, and Proxmox Backup
• Network 2: In addition to using it for the VMs, also use it for Proxmox management, NAS, and Proxmox Backup

Is this a good idea? Will I experience downtime or high latency?

Please provide suggestions for making the system as robust as possible.

Thank you very much.

 

[UdoB]

Is this a good idea?

Personally I am a plain ZFS user, but I have a Testcluster with Ceph (and a Homelab) and had run some tests, including injecting failures and observing what happens. From that very limited experience I need to say that I (!) would never run a "real" Ceph cluster with less than five stable nodes. (My Homelab currently has six...)

You seem to have some good hardware, but three nodes is the absolute minimum. As soon as anything fails you are degraded - and you stay degraded as there is no redundancy for Ceph to heal itself. (In the worst case that is, for example if you go with only one OSD per node and that one fails...)

Some details: https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/