RE-IP Proxmox VE and Ceph

[complexplaster27]

Hey All,

So our company has requested that we need to change our LAN IP in our Datacenter which were are currently sitting on 192.168.x.x/24 subnet.
They want our router to be RE-IP'd to 10.192.x.x/24 and subsequently all the devices in the Datacenter also need to change.

Currently running 4 clusters with 3 of them using Ceph for storage.

I've had to Re-ip proxmox before so that shouldn't be too much of an issue but I've never had to re-ip the ceph cluster.
Is it as simple as modifying the ceph.conf file and changing

cluster_network, mon_host, public_network and public_addr

for each of the mons?

I do have a test cluster I can test on break at any time so happy to implement any solutions if you have any.

Thanks in advance!

 

[mfed]

Hi @complexplaster27

I responded to similar questions on the forum. I believe you will have to ensure there is a routing between your old and new subnets for the duration of the transition. You are correct, you just modify those parameters appropriately as you go (I believe there is also cluster_addr on OSD).
The thing that I learnt from my excercise is that public_network and cluster_network act as a kind of ACLs for OSDs, so you will need to add your new subnet to those and restart each OSD (and maybe MON, don't remember) to make sure that they know about the new subnet and will accept the connections from the new subnet.

 

[spirit]

Hey All,

So our company has requested that we need to change our LAN IP in our Datacenter which were are currently sitting on 192.168.x.x/24 subnet.
They want our router to be RE-IP'd to 10.192.x.x/24 and subsequently all the devices in the Datacenter also need to change.

Currently running 4 clusters with 3 of them using Ceph for storage.

I've had to Re-ip proxmox before so that shouldn't be too much of an issue but I've never had to re-ip the ceph cluster.
Is it as simple as modifying the ceph.conf file and changing

cluster_network, mon_host, public_network and public_addr

for each of the mons?

I do have a test cluster I can test on break at any time so happy to implement any solutions if you have any.

Thanks in advance!

AFAIK, the clean official way is:
you need to create new monitors and delete the olders after. (and both need to be able to communicate during the transitions).

But I think it's possible to dump, modify && reinject the monmap, but it's not easy:

https://heiterbiswolkig.blogs.nde.ag/2020/12/18/cephadm-changing-a-monitors-ip-address/

 

[complexplaster27]

@mfed Does having routing between old and new subnets just mean that I need to re-ip ceph first and then re-ip proxmox second? or is that not the correct order of operations?

@spirit I'll have a read into this thanks for the document.

 

[mfed]

@complexplaster27 No, it's not related to proxmox network. When you restart an OSD configured to use the new subnet you will need it to be able to communicate to the OSDs that are still on the old subnet. Same with the MONs, when you re-create the new one still need to communicate to the old ones...

 

[complexplaster27]

@mfed So would I just keep both old and new IPs in the config? And then once restarted delete the old IPs?

 

[mfed]

@complexplaster27 For each OSD you configure the address to use, something like below (I use different cluster and public subnets, but you can just use the same address for both). Then you restart that OSD (osd.8) to start using the new IP address. Check that everything works, the cluster still healthy and the OSD running on the new IP. Then repeat the process for every OSD. Make sure that cluster_network and public_network in the global configuration are configured with both new and old subnets.

[osd.8]
        cluster_addr = 192.168.10.11
        public_addr = 192.168.9.11

 

[complexplaster27]

@mfed Thanks for the clarification and your advice! I think I was just overthinking on how to tackle this, it does seem a lot simpler with the way you outlined it.

 

[_--James--_]

If you are not routing Ceph outside of your PVE cluster, or its a local metric to external clusters, you do not need to re-IP Ceph unless there are services on that subnet that will need to be accessed by the cluster. Then you just have deal with the Corosync network(s) instead.