Change ceph network

carlosmp

Member
Jun 2, 2010
44
1
6
Hoping someone else has done this first...going to upgrade the network that my ceph runs on and wondering if my thinking is correct. Based on http://docs.ceph.com/docs/master/rados/operations/add-or-rm-mons/ it seems that the way to accomplish this is to first get the new network hardware up and running on all systems, test network connectivity, create new monitors, wait for it to report/quorom, then remove the old monitor, repeat as needed...This doesn't seem all that bad...(must be missing something...)

TIA
Carlos.
 

carlosmp

Member
Jun 2, 2010
44
1
6
Took the plunge last night. The above procedure does work, in terms of adding new mons, waiting for quorom, then removing the old. What made ours a bit more complex is that I was changing the NIC that we were using for vmbr and the ceph side. I temporarily used the 1Gb network that has the proxmox cluster service, and then moved it all back. Once you get the monitor working on all nodes, and get quorom, go into the storage.conf and edit the monitor IP addresses. One gotcha was having to recreate vmbr0 through the interface to get the new NICs to be usable by the guests.
 
  • Like
Reactions: RobFantini

RobFantini

Renowned Member
May 24, 2012
1,721
44
68
Boston,Mass
Hello
we need to change our ceph network from a 10G IP network to infiniband.
so the network subnet will be different. the 8 port 10G network is too small for the amount of ceph nodes, and we've plenty of infiniband parts.

I like Charlies write up and the link provided for changing network.

With the new pve ceph web control this change should be easier then just using cli.

It look like all that is needed is

0- do backups and shutdown all vm's before hand.
1-add 3 new mons using new network,
2-delete the old mons.
3- then edit storage.conf to correct new address.

any comments on that approach?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!