Change ceph network

[carlosmp]

Hoping someone else has done this first...going to upgrade the network that my ceph runs on and wondering if my thinking is correct. Based on http://docs.ceph.com/docs/master/rados/operations/add-or-rm-mons/ it seems that the way to accomplish this is to first get the new network hardware up and running on all systems, test network connectivity, create new monitors, wait for it to report/quorom, then remove the old monitor, repeat as needed...This doesn't seem all that bad...(must be missing something...)

TIA
Carlos.

 

[carlosmp]

Took the plunge last night. The above procedure does work, in terms of adding new mons, waiting for quorom, then removing the old. What made ours a bit more complex is that I was changing the NIC that we were using for vmbr and the ceph side. I temporarily used the 1Gb network that has the proxmox cluster service, and then moved it all back. Once you get the monitor working on all nodes, and get quorom, go into the storage.conf and edit the monitor IP addresses. One gotcha was having to recreate vmbr0 through the interface to get the new NICs to be usable by the guests.

 

[RobFantini]

Hello
we need to change our ceph network from a 10G IP network to infiniband.
so the network subnet will be different. the 8 port 10G network is too small for the amount of ceph nodes, and we've plenty of infiniband parts.

I like Charlies write up and the link provided for changing network.

With the new pve ceph web control this change should be easier then just using cli.

It look like all that is needed is

0- do backups and shutdown all vm's before hand.
1-add 3 new mons using new network,
2-delete the old mons.
3- then edit storage.conf to correct new address.

any comments on that approach?

 

[RobFantini]

.adding mons to a different network will not work using pve - there is not an option to assign a network.

 

[navigator]

Wow this is so bad, Proxmox should add this features for the next versions. Not being able to upgrade CEPH on a easy way, making it a pain in the ass to open a 12 Nodes setup to reconfigure it monitor IP addres is horrible.

 

[RobFantini]

This is an old thread, and PVE + Ceph are much easier to understand and manage. Check the documentation .

 

[navigator]

This is an old thread, and PVE + Ceph are much easier to understand and manage. Check the documentation .

I Already checked the entire documentation and im not able to find a way to migrate my entire Ceph Architecture to the new 10Gbps NICs.

Also those NIC are running on a different Network 10.0.0.0 and not on 192.168.0.0 like before.

Can you please share to me the way, or any link that could help on this migration?

 

[RobFantini]

Hello Marcos.
first your initial post of 'Wow ..' was not a good way to get help. it was insulting to me and probably others . certainly not a good way to ask for help. the post NOT POLITE or clear on what you are trying to ccomplish.


Question - what are you trying to do? ceph software upgrade or change IP address of ceph network?

 

[navigator]

Sorry for my words, but dealing with I.T infrastructure of Healthcare during Covid pandemic and having to deal with Ceph Network change had been very stresfull for me this last months in Brazil. Didn't wanted to offend anyone.

Im trying to change Ceph Network to work under a 10Gbps Switch (Under 10.0.0.1/24) Network before it was running under 192.168.0.1/24 (the same network as the PVE Cluster.

As i couldn't find a solution i started to re-instal proxmox on each node and recreate the CEPH network again, but i hope that if you have any solution for changing the Ceph Neetwork and could share i'll aprritiate.

Sorry again if i was unpolite but things here are pretty a war scenario