Cluster new connectivity newtwork between servers.

[José Fermín Francisco]

Good evening.
We currently have three Dell R740 servers running Proxmox 8.3.2.
We initially had network connectivity between the three servers with IP address 10.100.250.xxx/24, and to manage our Proxmox, we had IP address 172.50.1.xxx/24.
We have a cluster configured, using high availability, and we also use CEPH for disk management.

It turns out that both networks were routed through the same network (same Cisco 350 switches), and when there was network congestion, all the physical servers would reboot, causing all work to stop while we waited for all the physical servers to reboot, and then for the virtual machines to reboot (this situation occurs once or twice a week).
We were investigating this photo and the internet in general and we came to the conclusion that to avoid this terrible situation we had to interconnect the 3 servers using optical fibers. We configure these networks:

10.100.2.1/30 -> Server 1
10.100.2.2/30 -> Server 2
10.200.2.1/30 -> Server 1
10.200.2.2/30 -> Server 3
10.250.2.1/30 -> Server 2
10.250.2.2/30 -> Server 3

So, to finish solving the network congestion using the Cisco switches, I need to replace the communication networks of servers 10.100.250.1/24 (server 1), 10.100.250.2/24 (server 2), and 10.100.250.3/24 (server 3) with the newly created networks.

The administrative network 172.50.1.xxx/24 must remain the same in the configuration.

How can we do this?

 

[José Fermín Francisco]

You will need to follow these steps:

- Step 1: Verify New Fiber Network Connections
Before making any changes, you must confirm that the new fiber connections are working and can communicate with each other. From each server, you should be able to ping the other two servers using the new IP addresses.

- Step 2: Update Corosync Cluster Configuration
This is the most important step for Proxmox. You will edit the /etc/pve/corosync.conf file to tell the cluster to use the new fiber network for communication.

- Step 3: Update CEPH Configuration
Next, you need to tell CEPH to use the new networks for the internal cluster_network and public_network.

- Step 4: Final Verification and Cleanup
After restarting all services, verify that CEPH is working properly and all Proxmox nodes are communicating.

Good afternoon, jumperspy, and thank you for your quick response.
Currently, server 1 successfully pings the following IP addresses:
10.100.2.1
10.100.2.2
10.200.2.1
10.200.2.2

Currently, server 2 successfully pings the following IP addresses:
10.100.2.1
10.100.2.2
10.250.2.1
10.250.2.2

Currently, server 3 successfully pings the following IP addresses:
10.200.2.1
10.200.2.2
10.250.2.1
10.250.2.2
10.100.2.2

I don't know how to do this last step (Step 3: Update CEPH Configuration).