I am going to install a Proxmox cluster in HA, and the virtual machines will be hosted in CEPH and NAS (NFS). The most important ones will be in CEPH, and the rest in the NAS.
My doubts are about latency when using the CEPH network for everything. We’ve based this network on the existing manual to set up a Mesh network with FRR. My network setup is as follows:
Mesh network at 100G
Link network at 10G with LACP to the switch where the NAS and Proxmox Backup are located
Network 1, this is a service network for the virtual machines (OT)
Network 2, another service network (IT)
Network 3, another network for the VMs
I had planned to use the networks in this way:
Mesh network 100G: public and private CEPH network, Corosync (link 0 - higher priority)
Personally I am a plain ZFS user, but I have a Testcluster with Ceph (and a Homelab) and had run some tests, including injecting failures and observing what happens. From that very limited experience I need to say that I (!) would never run a "real" Ceph cluster with less than five stable nodes. (My Homelab currently has six...)
You seem to have some good hardware, but three nodes is the absolute minimum. As soon as anything fails you are degraded - and you stay degraded as there is no redundancy for Ceph to heal itself. (In the worst case that is, for example if you go with only one OSD per node and that one fails...)
A minimum for a good and totally seperated network layout is at least 5 distinct networks:
ceph private
ceph public
pve cluster interconnect
pve cluster migration
public
this is of course a lot. You can - depending on your needs - skip some of them. Corosync can run everywhere if you ensure package priorisation and monitor it. This is more for advanced users. If you have two good switches, you can also use vlans for everything, e.g. 4x 100G LACP and to bandwidth limitations. Often it is more a question of how much would you want to spend. With 3 nodes, FRR is a good choice, yet if you want to scale horizontally, it does not scale, so you need to rebuild everything. If you already start with a two switch layout, it is much easier to scale horizontally.
