Moving from 3-node full-mesh SDN to a 6-node using switches

[traveltrex]

Hi there,

we currently have a 3-node full-meshed SDN using OpenFabric for our ceph backend (used on separate interfaces and networks). Ceph backend uses 25gbit/s FC (we have 2x 2 Port Cards). We want to increase our cluster by 3 nodes but dissolving the SDN in the process, as the setup using a switch with this many nodes is much simpler and less cost-intensive.

What would be the best way to switch away from SDN without downtime?

Thanks in advance!

Patrick

 

[woma]

Hi Patrick,

Because new and old machines are same numbers you will have resources to temporarily run all guests only using new machines, so my safe and boring approach would be:

• build new 3 node cluster connected to switches
• setup up and test virtual network bridges and storage needed by your guests
• add both clusters to Proxmox Datacenter Manager
• PDM: cross cluster live migration of guests from old to new cluster
• shutdown old cluster machines
• rewire networking and connect old machines to switches
• perform fresh install of PVE on old machines
• join old machines to new cluster

If you don't like or have PDM, "qm remote-migrate" on CLI is possible.

Source and target clusters both keep stable configuration and network cabling while doing that transfer.
You can test new cluster before migrating guests to it and you do not touch old known working cluster either.

 

[traveltrex]

Sorry for late response, I was on a long vacation and just came back. Besides the mentioned approach - which i also appreciate - is there another way, maybe with accepted downtime of 1 host or all hosts (in a maintenance window)?

 

[kalium]

If a maintenance window is possible, what I did in a similar case was:

• Create a VLAN in the switch;
• In the switch, allow tagged traffic from the ports or bonds which the nodes are connected;
• Now on PVE, create the VLAN connection on each node;
• Stop all VMs and pause Ceph traffic with OSD flags (noout and pause);
• On SDN configuration, delete the OpenFabric mesh and create an OSPF network (broadcast) with the VLAN connection, with the IPs previously used in the OpenFabric mesh.

After applying the changes and checking if the nodes can reach each other, it should be safe to remove the Ceph flags and later start the VMs again.

 

[traveltrex]

If a maintenance window is possible, what I did in a similar case was:

• Create a VLAN in the switch;
• In the switch, allow tagged traffic from the ports or bonds which the nodes are connected;
• Now on PVE, create the VLAN connection on each node;
• Stop all VMs and pause Ceph traffic with OSD flags (noout and pause);
• On SDN configuration, delete the OpenFabric mesh and create an OSPF network (broadcast) with the VLAN connection, with the IPs previously used in the OpenFabric mesh.

After applying the changes and checking if the nodes can reach each other, it should be safe to remove the Ceph flags and later start the VMs again.

What is the difference from using OSPF instead of just static network configuration? Is it by any means better?