ceph changing public network

D

daviditty

Member
Proxmox Subscriber
Jun 24, 2021
20
8
8
Netherlands
Hi Guys,

So, I'm in a situation where I want to change the ceph public network,
I've read up on it quite a bit, and before "testing" it in production ;-) I built a virtual test environment,
which amazed me, it was actually quite easy and I want to fact check it here...

The virtual cluster consisted of:
3x pve 7.3 host, 2 networks on 2 nic's, ceph octopus, 3 OSD's per node.
ceph cluster network 10.10.10.1/24
ceph public network 10.0.0.1/24
monitors are on the 10.0.0.1/24 this way initally.

I made 1 VM after clustering and put it on RBD.
Within the VM I ran a while loop to see if disk IO was possible during the process, as I want to do it in a live environment.
within ssh session 1: while true; do echo $RANDOM >> test; sleep 2; done
within ssh session 2: tail -f test

During the whole process writing was possible, and the SSH session stayed alive as I migrated the VM between the nodes.

Steps I did are:
- edit the ceph public network to 10.10.10.1/24 in /etc/pve/ceph.conf, save the file;
- destroy / recreate all monitors one by one via the GUI, they will come up with a new ip address in the new network, check the ceph and VM health in between;
- destroy / recreate all managers one by one via the GUI, they will come up with a new ip address in the new network, check the ceph and VM health in between;
- migrate VM to other node;
- restart one node;
- move VM back;
- reboot other nodes;
- done.

I was so amazed, I also seperated the ceph public network again after merging it to see if I could, and it works perfectly.

In between I also checked the RBD watchers with rbd ls, find a VM disk, and check it with rbd status vm-100-disk-0 for example,
you will see after live migrating that the listener IP changes to the new ip address, too.

With netstat -tulpn | grep ceph you also see that the old ip's will be gone after rebooting the nodes.


Am I missing something, or is it really this easy? It looks like it is...


Kind regards,
David
 
  • Like
Reactions: inta, pepex7, BruceX and 1 other person
daviditty said:
Hi Guys,

So, I'm in a situation where I want to change the ceph public network,
I've read up on it quite a bit, and before "testing" it in production ;-) I built a virtual test environment,
which amazed me, it was actually quite easy and I want to fact check it here...

The virtual cluster consisted of:
3x pve 7.3 host, 2 networks on 2 nic's, ceph octopus, 3 OSD's per node.
ceph cluster network 10.10.10.1/24
ceph public network 10.0.0.1/24
monitors are on the 10.0.0.1/24 this way initally.

I made 1 VM after clustering and put it on RBD.
Within the VM I ran a while loop to see if disk IO was possible during the process, as I want to do it in a live environment.
within ssh session 1: while true; do echo $RANDOM >> test; sleep 2; done
within ssh session 2: tail -f test

During the whole process writing was possible, and the SSH session stayed alive as I migrated the VM between the nodes.

Steps I did are:
- edit the ceph public network to 10.10.10.1/24 in /etc/pve/ceph.conf, save the file;
- destroy / recreate all monitors one by one via the GUI, they will come up with a new ip address in the new network, check the ceph and VM health in between;
- destroy / recreate all managers one by one via the GUI, they will come up with a new ip address in the new network, check the ceph and VM health in between;
- migrate VM to other node;
- restart one node;
- move VM back;
- reboot other nodes;
- done.

I was so amazed, I also seperated the ceph public network again after merging it to see if I could, and it works perfectly.

In between I also checked the RBD watchers with rbd ls, find a VM disk, and check it with rbd status vm-100-disk-0 for example,
you will see after live migrating that the listener IP changes to the new ip address, too.

With netstat -tulpn | grep ceph you also see that the old ip's will be gone after rebooting the nodes.


Am I missing something, or is it really this easy? It looks like it is...


Kind regards,
David
Click to expand...
Hi there,

I was looking for this for a long, long, time and got no clear answers. You solved my issue with this one post (in my case, other way around - taking out cluster network and changing public).

EDIT: I did almost as you did, but without restarting the hosts, with the following steps:
  1. Change network configuration in "ceph.conf" (in my case, dropped the cluster network and kept just the public, moving to the new subnet);
  2. Destroy and recreate monitors (one by one);
  3. Destroy and recreate managers (one by one, leaving the active one for last);
  4. Destroy and recreate metadata servers (one by one, leaving the active one for last;
  5. Restart OSDs (one by one - or more, depending how many OSDs you have in the cluster - so you avoid restarting the hosts);
Thank you very much, best regards!
 
Last edited:
  • Like
Reactions: inta, BruceX and daviditty
cetjunior said:
Hi there,

I was looking for this for a long, long, time and got no clear answers. You solved my issue with this one post (in my case, other way around - taking out cluster network and changing public).

EDIT: I did almost as you did, but without restarting the hosts, with the following steps:
  1. Change network configuration in "ceph.conf" (in my case, dropped the cluster network and kept just the public, moving to the new subnet);
  2. Destroy and recreate monitors (one by one);
  3. Destroy and recreate managers (one by one, leaving the active one for last);
  4. Destroy and recreate metadata servers (one by one, leaving the active one for last;
  5. Restart OSDs (one by one - or more, depending how many OSDs you have in the cluster - so you avoid restarting the hosts);
Thank you very much, best regards!
Click to expand...

Cool! Glad that you sorted it out, I still have to change it in production as we're sorting our network out first :)
 
  • Like
Reactions: pepex7
cetjunior said:
Hi there,

I was looking for this for a long, long, time and got no clear answers. You solved my issue with this one post (in my case, other way around - taking out cluster network and changing public).

EDIT: I did almost as you did, but without restarting the hosts, with the following steps:
  1. Change network configuration in "ceph.conf" (in my case, dropped the cluster network and kept just the public, moving to the new subnet);
  2. Destroy and recreate monitors (one by one);
  3. Destroy and recreate managers (one by one, leaving the active one for last);
  4. Destroy and recreate metadata servers (one by one, leaving the active one for last;
  5. Restart OSDs (one by one - or more, depending how many OSDs you have in the cluster - so you avoid restarting the hosts);
Thank you very much, best regards!
Click to expand...
I upgraded my existing 3 node cluster with dual 10Gbe NICs and configured them as a Full Mesh Network. I needed to get my existing ceph to move over to the new 10Gbe mesh network. This worked great, especially restarting the OSDs rather than rebooting the nodes. My initial attempts to make all the needed changes in ceph.conf hadn't worked out well. It was great to see my test LXC almost triple its disk write speeds over the new network without any reboots or LXC reconfiguration. Thanks for adding these experiences (neither chatGPT nor Bard found these insights!)
 
Last edited:
  • Like
Reactions: pepex7, cetjunior and daviditty
I just did this in our production environment too, and it worked perfectly with your addition @cetjunior :)

I want to add to this that if you do this in the future, you only have to adjust the public_network or cluster_network under "global" in /etc/pve/ceph.conf, don't worry about all the mon_host, ceph handles these themselves once you follow all the steps!
 
  • Like
Reactions: cetjunior
Hello Together

Just to clarify it in the beginning of my post, I am defently no ceph expert, so take my advice with a grain of salt.

I had problems (Downtimes of Managemnt and dashboard) using this solution:
cetjunior said:
Hi there,

I was looking for this for a long, long, time and got no clear answers. You solved my issue with this one post (in my case, other way around - taking out cluster network and changing public).

EDIT: I did almost as you did, but without restarting the hosts, with the following steps:
  1. Change network configuration in "ceph.conf" (in my case, dropped the cluster network and kept just the public, moving to the new subnet);
  2. Destroy and recreate monitors (one by one);
  3. Destroy and recreate managers (one by one, leaving the active one for last);
  4. Destroy and recreate metadata servers (one by one, leaving the active one for last;
  5. Restart OSDs (one by one - or more, depending how many OSDs you have in the cluster - so you avoid restarting the hosts);
Thank you very much, best regards!
Click to expand...

The way I accomplished it and how I would do it again is by adding a second public network first.

  1. Add network config and test connection between each node (I used iperf3)
  2. Add an additional Public Network (New Network) to /etc/pve/ceph.conf
    Code: 
    # Example with variables    public_network = <host ip/cidr of new Network>, <host ip/cidr of old Network>

# Example with real IPs
    public_network = 172.22.21.201/24, 172.22.3.201/24
    Add new network first to see new IP in pve Web-UI
  3. Destroy and recreate monitors (one by one)
  4. Destroy and recreate managers (one by one, leaving the active one for last)
  5. Destroy and recreate metadata servers (one by one, leaving the active one for last
  6. Remove old Public Network from /etc/pve/ceph.conf
  7. Destroy and recreate monitors (one by one)
  8. Destroy and recreate managers (one by one, leaving the active one for last)
  9. Destroy and recreate metadata servers (one by one, leaving the active one for last
  10. Set noout flag (Either in UI or with shell "ceph osd set noout"
  11. Restart OSDs (one by one - or more, depending how many OSDs you have in the cluster - so you avoid restarting the hosts)
  12. Unset noout flag (Either in UI or with shell "ceph osd unset noout"
  13. Remove old network config from all host (if needed)
^Maby this could be some more optimized somehow, but like this I did not get any downtime
Hope this helps anyone
 
  • Like
Reactions: avluis86, UdoB and daviditty
Waaaat said:
Hello Together

Just to clarify it in the beginning of my post, I am defently no ceph expert, so take my advice with a grain of salt.

I had problems (Downtimes of Managemnt and dashboard) using this solution:


The way I accomplished it and how I would do it again is by adding a second public network first.

  1. Add network config and test connection between each node (I used iperf3)
  2. Add an additional Public Network (New Network) to /etc/pve/ceph.conf
    Code: 
    # Example with variables    public_network = <host ip/cidr of new Network>, <host ip/cidr of old Network>

# Example with real IPs
    public_network = 172.22.21.201/24, 172.22.3.201/24
    Add new network first to see new IP in pve Web-UI
  3. Destroy and recreate monitors (one by one)
  4. Destroy and recreate managers (one by one, leaving the active one for last)
  5. Destroy and recreate metadata servers (one by one, leaving the active one for last
  6. Remove old Public Network from /etc/pve/ceph.conf
  7. Destroy and recreate monitors (one by one)
  8. Destroy and recreate managers (one by one, leaving the active one for last)
  9. Destroy and recreate metadata servers (one by one, leaving the active one for last
  10. Set noout flag (Either in UI or with shell "ceph osd set noout"
  11. Restart OSDs (one by one - or more, depending how many OSDs you have in the cluster - so you avoid restarting the hosts)
  12. Unset noout flag (Either in UI or with shell "ceph osd unset noout"
  13. Remove old network config from all host (if needed)
^Maby this could be some more optimized somehow, but like this I did not get any downtime
Hope this helps anyone
Click to expand...
May I ask what kind of downtime you experienced? My guess would be with the Ceph MDS? It is the only thing different from the earlier posts, we didn't use them, only RBD.


Also, nice addition I think, didn't know it was possible to add a "double" ring.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back