Ceph 3-node cluster with direct links

[PerG]

What is the recommendation for setting up Ceph on a 3-node cluster with direct cabling between hosts, for Ceph?

It seems one should run Ceph on Loopback interfaces, but it seems not supported in the GUI.

So this will use a bond of 2x25G interfaces directly between the 3 hosts for Ceph, as a single network for both Public and Cluster traffic. Other 1G and 10G interfaces (connected via Ethernet switches) will be used for Corosync, management and VM traffic, so this question is only for the Ceph traffic.

It seems one would want Ceph on the (lo) loopback interfaces, and then just use static routes to reach these, via the direct link, or via the other host. So two static routes on H1, that 1) would reach H2 via the /31 p2p link directly to H2, and 2) via the /31 link to H3 with lower metric. One could also use FRR/OSPF, etc., but regardless of how IP routing is setup, it seems it is not possible to choose the lo interfaces for Ceph in the new GUI.

Also, if one selects the /31 interfaces in the GUI for Ceph Public, then how is that replicated to the other hosts, since not all 3 hosts are connected to the same /31 p2p interfaces?

 

[shanreich]

this is possible by utilizing SDN, please check out our guide in the wiki [1]

[1] https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Using_SDN_Fabrics

 

[PerG]

Great, so using SDN/OpenFabric we do not configure any IP addresses in Linux on loopback or physical interfaces.

Instead, the subnet for the OpenFabric (with subnet mask) is configured in the "Create OpenFabric" dialog box, and then each node is given an IP address from that subnet, which is effectively used by FRR as "lo" interface. (I assume the NET address is derived automatically.) Can we still use bond interfaces (2x25G), which are first configured in networking and later used in OpenFabric? Or since the interfaces are unnumbered is it better to use two individual interfaces? I suspect that with bonds it is easier to control the load balancing hash configuration.

As an alternative, should it also be possible to configure lo interface and static routing in Linux? Or if doing so it is not possible to afterwards use the GUI to configure Ceph?

 

[shanreich]

Can we still use bond interfaces (2x25G), which are first configured in networking and later used in OpenFabric? Or since the interfaces are unnumbered is it better to use two individual interfaces? I suspect that with bonds it is easier to control the load balancing hash configuration.

I'd prefer using the interfaces separately - some users have ran into problems with using bonds + OpenFabric / OSPF. The hashing configuration can be done via the ECMP sysctls, since it is a kernel function - fib_multipath_hash_policy is probably what's interesting in that case.

As an alternative, should it also be possible to configure lo interface and static routing in Linux? Or if doing so it is not possible to afterwards use the GUI to configure Ceph?

It is possible, but you will need to manually create the Ceph cluster via pveceph. The other methods, including static routing, are also documented in the Wiki article.