Change Ceph Network on Existing Cluster

[beav]

Hello, I have a question for the Ceph masterminds... I have setup a better network on my Proxmox cluster to be used for Ceph. My existing Ceph cluster is using the original network and I would like to change to the new.

Is there a proper method of changing the Ceph cluster config to use the new network interfaces that would be non-disruptive?

 

[gurubert]

Add the new network as additional network to the "public_network" configuration option of Ceph.
Remove one MON on a node and re-provision it with an IP from the new network. Do that for each MON.
Remove the old network from "public_network".
Restart all other Ceph components (OSDs, MDSs).

 

[beav]

Few questions as I do not want to misunderstand:

1. Do I need to shutdown any of my VMs or anything else to prevent possible corruption?
2. This has to be done in the CLI by editing '/etc/pve/ceph.conf', correct? I do not see a means to change anything in the UI.
3. It is only the Ceph cluster network that I am changing. So am I to change 'public_network' or 'cluster_network' in the config?
4. The new network is IPv6 so is this what your are suggesting it would look like in the config?
   cluster_network = 10.10.10.13/24 fc00::21/29
5. I only have one MON on each node. Should I be creating an additional one with the new NW per node?
6. Then after reworking the MONs and restarting the other components as you describe, do I go back and remove the old IPv4 address at that time.

Apologies if I am missing your guidance. I am new to Ceph.

 

[gurubert]

Ceph cannot run double-stacked. So it's either all IPv4 or all IPv6, but not mixed.

Changing just the cluster_network is way easier as you do not need to change the MONs' IPs.

If the cluster_network is in your ceph.conf just change it there and restart all OSDs.

 

[beav]

Ceph cannot run double-stacked. So it's either all IPv4 or all IPv6, but not mixed.

Changing just the cluster_network is way easier as you do not need to change the MONs' IPs.

If the cluster_network is in your ceph.conf just change it there and restart all OSDs.

Ok, that makes it easier - maybe. However, to be clear even cluster_network cannot be IPv6 if only public is IPv4? Therefore, I would have to make public IPv6 as well, right?

 

[gurubert]

I would think making the new cluster_network IPv4 would be easier.

Or are all of your Ceph clients already IPv6 enabled?

 

[beav]

I would think making the new cluster_network IPv4 would be easier.

Or are all of your Ceph clients already IPv6 enabled?

The intention is to use the Thunderbolt NW I have setup between the nodes for the Ceph Cluster NW which is configured for dual stack across the Proxmox nodes. I have the Proxmox cluster using it via IPv6 for migration and wanted the same for the Ceph NW.

I have had some issues getting the IPv4 stack consistent and was not too worried about it as I was planning to configure the Ceph cluster to use IPv6 on the TB NW. I can look into getting IPv4 stable on that network I suppose if it is not possible to use IPv6.

How do I confirm whether or not the Ceph Clients are setup for IPv6?

...and thanks very much for the guidance and patience here!

 

[beav]

I have successfully changed the cluster_network to the IPv4 stack I have configured on the Thunderbolt NW. Followed by a restart of each OSD and Proxmox host, all came back up and seems to be working well. Thanks for the assistance!