[SOLVED] Migrate CEPH public_network online

[FXKai]

Hi,

we are in the following productive situation

5 x PVE CEPH Storage with 2x1GE NIC and 1 x dual 10GE NIC (LACP) for the CEPH public network
5 x PVE Compute Nodes with 2x1GE NIC and 2 x dual 10GE NIC (LACP), one is for the CEPH public network
2 x MLAG/LACP connected 10GE Switches (Lenovo, low latency) running the 10GE (LACP) CEPH public network in a VLAN
1 x Switch (Ubiquity) with 1GE for CoroSync and Management

New to this setup is now

- 2 x MLAG/LACP connected 40GE Mellanox Switches (low latency)
- 10 x Mellanox 40GE Dual Port NIC, one for each node
- each NIC has an IP in a separate subnet, each node can reach all the other nodes (CEPH and Compute)

after this condensed buzzword bingo ...

What i would like to achieve in Step 1 is : to migrate the public_network from 10GE to 40GE with no downtime in 2 steps

- Migrate the CEPH Storage nodes from the 10GE VLAN onto the new 40GE network, Node by Node
- if all works well, migrate all the compute nodes from their 10GE NIC to the new 40GE NIC with no downtime (should be easy, since i can free up each Compute node and reboot)

Alternatively i would be happy run the public_network on both 10GE and 40GE Switches with priority to the 40GE Switch or create a cluster_network on the 10GE VLAN later if the cluster grows.

So my questions:
- is it possible to migrate the CEPH Nodes from the 10GE Switch to the 40GE, node by node and how?
- is it possible to have the public_network running on two networks with different ip subnets (10GE and 40GE) and set priorities, if so how?
- is there any caveats i should consider while migrating?

Thanks and best,

Kai

 

[czechsys]

Old and new subnets are same vlan or different? The main rule is, multiple public_network need to be able to communicate.
You need to reconfigure monitors, so it's important to read documentation (and in some cases, test such migration).

 

[FXKai]

Thanks, right now they are in seperate vlans. routed L3 communication should be sufficient no?

What i understand is, if mon1 will be migrated to the new subnet but can still talk (via TCP) to the other mons, i should be fine?

 

[FXKai]

We managed a successful migration by uplinking the 40GE Switches to the old 10GE Storage VLAN and then moving each host's IP from the 10GE NIC to the 40GE NIC