Proxmox + Ceph, separating public and cluster networks

[Klug]

Hi all.

I currently have a 5 nodes cluster.
On each node, there's :
. 2x10 Gbps NIC in LACP for "data" (VM traffic using VLANs)
. 2x10 Gbps NIC in LACP for Ceph (both public and cluster networks on the same NIC)
. 2x10 Gbps NIC currently unused (unused because lack of cables when setting up the cluster)

Now I'd like to separate the public and cluster network for Ceph, using the unused NICs.
However, I don't really know how to do it.

Creating the new bound is easy.
But once it's crceated, I don't know how to switch the cluster (or public) trafic from the current subnet it uses (10.10.10.0/24) to the new one created for this use (10.10.11.0/24).
If I do this on a node, it won't be able to connect the other nodes (thus no more cluster or public trafic).
"Loosing" one node is OK, but how to do the same on the other nodes?

Do you have any suggestion about a way to do this (without stopping the whole cluster)?

 

[Uplink]

@Klug I'm a little confused. Your post makes it sounds like you've already created public and cluster networks for Ceph,

both public and cluster networks on the same NIC

Let's label your NIC bonds for clarity

bond1: . 2x10 Gbps NIC in LACP for "data" (VM traffic using VLANs)
bond2: . 2x10 Gbps NIC in LACP for Ceph (both public and cluster networks on the same NIC)
bond3: . 2x10 Gbps NIC currently unused (unused because lack of cables when setting up the cluster)

Assuming both networks are already created, and Ceph already uses both networks, you don't need to change the subnets the Ceph traffic goes on. The subnet are already used.

What I (think) you want to do is move one of the networks, say the public one, to bond3
In this case, it's simple networking. All you have to do is move the Linux VLAN / OVSInt port to bond 3, which is a simple change in the network config. As long as you've configured the switches appropriately, each node will only lose access to that network momentarily, as you apply the new network configuration. Using ifupdown2 for this is very handy.

 

[VictorSTS]

May we take a look at your /etc/ceph/ceph.conf ?

If it's already using a cluster network, you'll have somethig like cluster_network = 10.195.5.0/24 there.

 

[Klug]

Here's my ceph.conf :

[global]
         auth client required = cephx
         auth cluster required = cephx
         auth service required = cephx
         cluster network = 10.10.10.0/24
         fsid = d72132ab-739f-4782-be07-7d5980f8dabe
         mon allow pool delete = true
         osd journal size = 5120
         osd pool default min size = 2
         osd pool default size = 3
         public network = 10.10.10.0/24
         mon_host = 10.10.10.1 10.10.10.3 10.10.10.5

[client]
         keyring = /etc/pve/priv/$cluster.$name.keyring

[osd]
         keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mon.pve01]
         host = pve01
         mon addr = 10.10.10.1:6789

[mon.pve03]
         host = pve03
         mon addr = 10.10.10.3:6789

[mon.pve05]
         host = pve05
         mon addr = 10.10.10.5:6789

I might have not be clear enough in my first post: cluster and public are currently using the same subnet.
So it's more a "logical" networking issue (subnets) than "physical" (bounds).

I could be using a single bound, the upcoming problem is separating the current subnet in two different subnets (one for public, one for cluster).

Or maybe:
1. setup network switch so bound2 and bound3 are in the same VLAN
2. give an IP to each node, on bound3, in 10.10.11.0/20
2. change subnet of IPs given to the nodes on bound2 to /20 instead of /24 then restart Ceph (change in /etc/network/interface and /etc/ceph/ceph.conf)
3. separate cluster and public network on each node (10.10.10.0/20 and 10.10.11.0/20) then restart Cecph (change /etc/ceph/ceph.conf)
4. do 2 and 3 again to change the subnets to /24
5. change switch configuration so bound2 and bound3 are in different VLANs