RE-IP Proxmox VE and Ceph

C

complexplaster27

Member
Proxmox Subscriber
Jul 10, 2023
57
10
13
Hey All,

So our company has requested that we need to change our LAN IP in our Datacenter which were are currently sitting on 192.168.x.x/24 subnet.
They want our router to be RE-IP'd to 10.192.x.x/24 and subsequently all the devices in the Datacenter also need to change.

Currently running 4 clusters with 3 of them using Ceph for storage.

I've had to Re-ip proxmox before so that shouldn't be too much of an issue but I've never had to re-ip the ceph cluster.
Is it as simple as modifying the ceph.conf file and changing
Code: 
cluster_network, mon_host, public_network and public_addr
for each of the mons?

I do have a test cluster I can test on break at any time so happy to implement any solutions if you have any.

Thanks in advance!
 
Hi @complexplaster27

I responded to similar questions on the forum. I believe you will have to ensure there is a routing between your old and new subnets for the duration of the transition. You are correct, you just modify those parameters appropriately as you go (I believe there is also cluster_addr on OSD).
The thing that I learnt from my excercise is that public_network and cluster_network act as a kind of ACLs for OSDs, so you will need to add your new subnet to those and restart each OSD (and maybe MON, don't remember) to make sure that they know about the new subnet and will accept the connections from the new subnet.
 
complexplaster27 said:
Hey All,

So our company has requested that we need to change our LAN IP in our Datacenter which were are currently sitting on 192.168.x.x/24 subnet.
They want our router to be RE-IP'd to 10.192.x.x/24 and subsequently all the devices in the Datacenter also need to change.

Currently running 4 clusters with 3 of them using Ceph for storage.

I've had to Re-ip proxmox before so that shouldn't be too much of an issue but I've never had to re-ip the ceph cluster.
Is it as simple as modifying the ceph.conf file and changing
Code: 
cluster_network, mon_host, public_network and public_addr
for each of the mons?

I do have a test cluster I can test on break at any time so happy to implement any solutions if you have any.

Thanks in advance!
Click to expand...
AFAIK, the clean official way is:
you need to create new monitors and delete the olders after. (and both need to be able to communicate during the transitions).

But I think it's possible to dump, modify && reinject the monmap, but it's not easy:

https://heiterbiswolkig.blogs.nde.ag/2020/12/18/cephadm-changing-a-monitors-ip-address/
 
@complexplaster27 No, it's not related to proxmox network. When you restart an OSD configured to use the new subnet you will need it to be able to communicate to the OSDs that are still on the old subnet. Same with the MONs, when you re-create the new one still need to communicate to the old ones...
 
@complexplaster27 For each OSD you configure the address to use, something like below (I use different cluster and public subnets, but you can just use the same address for both). Then you restart that OSD (osd.8) to start using the new IP address. Check that everything works, the cluster still healthy and the OSD running on the new IP. Then repeat the process for every OSD. Make sure that cluster_network and public_network in the global configuration are configured with both new and old subnets.

Code: 
[osd.8]
        cluster_addr = 192.168.10.11
        public_addr = 192.168.9.11
 
  • Like
Reactions: complexplaster27
If you are not routing Ceph outside of your PVE cluster, or its a local metric to external clusters, you do not need to re-IP Ceph unless there are services on that subnet that will need to be accessed by the cluster. Then you just have deal with the Corosync network(s) instead.
 
You must log in or register to reply here.
Top Bottom