Change Ceph Network on Existing Cluster

B

beav

New Member
Aug 8, 2024
8
1
3
Hello, I have a question for the Ceph masterminds... I have setup a better network on my Proxmox cluster to be used for Ceph. My existing Ceph cluster is using the original network and I would like to change to the new.

Is there a proper method of changing the Ceph cluster config to use the new network interfaces that would be non-disruptive?
 
Few questions as I do not want to misunderstand:
  1. Do I need to shutdown any of my VMs or anything else to prevent possible corruption?
  2. This has to be done in the CLI by editing '/etc/pve/ceph.conf', correct? I do not see a means to change anything in the UI.
  3. It is only the Ceph cluster network that I am changing. So am I to change 'public_network' or 'cluster_network' in the config?
  4. The new network is IPv6 so is this what your are suggesting it would look like in the config?
    cluster_network = 10.10.10.13/24 fc00::21/29
  5. I only have one MON on each node. Should I be creating an additional one with the new NW per node?
  6. Then after reworking the MONs and restarting the other components as you describe, do I go back and remove the old IPv4 address at that time.
Apologies if I am missing your guidance. I am new to Ceph.
 
Ceph cannot run double-stacked. So it's either all IPv4 or all IPv6, but not mixed.

Changing just the cluster_network is way easier as you do not need to change the MONs' IPs.

If the cluster_network is in your ceph.conf just change it there and restart all OSDs.
 
gurubert said:
Ceph cannot run double-stacked. So it's either all IPv4 or all IPv6, but not mixed.

Changing just the cluster_network is way easier as you do not need to change the MONs' IPs.

If the cluster_network is in your ceph.conf just change it there and restart all OSDs.
Click to expand...
Ok, that makes it easier - maybe. However, to be clear even cluster_network cannot be IPv6 if only public is IPv4? Therefore, I would have to make public IPv6 as well, right?
 
Last edited:
gurubert said:
I would think making the new cluster_network IPv4 would be easier.

Or are all of your Ceph clients already IPv6 enabled?
Click to expand...
The intention is to use the Thunderbolt NW I have setup between the nodes for the Ceph Cluster NW which is configured for dual stack across the Proxmox nodes. I have the Proxmox cluster using it via IPv6 for migration and wanted the same for the Ceph NW.

I have had some issues getting the IPv4 stack consistent and was not too worried about it as I was planning to configure the Ceph cluster to use IPv6 on the TB NW. I can look into getting IPv4 stable on that network I suppose if it is not possible to use IPv6.

How do I confirm whether or not the Ceph Clients are setup for IPv6?

...and thanks very much for the guidance and patience here!
 
I have successfully changed the cluster_network to the IPv4 stack I have configured on the Thunderbolt NW. Followed by a restart of each OSD and Proxmox host, all came back up and seems to be working well. Thanks for the assistance!
 
  • Like
Reactions: gurubert
You must log in or register to reply here.
Top Bottom