Changing Ceph IP Range For Data Sync

[infinityM]

Hey Guys,

I made a mistake it seems when setting up ceph...
I used the same IP range for ceph as for our live IP ranges... From what I have heard and ready this is a big no-no... So i'd love to fix it...

How can I safely change the ip range form example: 1.1.1.14/28 to 192.168.155.0/24 for ceph's data sync.

I want to be sure as ceph grows it does not slow down network traffic for end users accessing the servers...

I look forward to your response .
Thanks guys

 

[Lephisto]

Hi,

A solution might be:

Temporarily change cluster_network in ceph.conf to the same range as public_network, so you don't need to shutdown everything and can do it on a live cluster (it's fine for ceph do have public/cluster network on the same l2/3 partition). After that, restart all osd's (wait every time until HEALTH_OK is kicking in), observe the network cards that used to be your cluster_network and double check there's no traffic left. Next reconfigure the IP's for those dedicated Network Interfaces to your new desired ceph cluster_network ip range and , change the ceph.conf again to that range, and restartd all osd's again.

You could probably skip the step migrating the cluster_network traffic to your public_network if you don't need to do everything live.

 

[infinityM]

That's the thing. I have 2 network ports, each running their own IP, it's only ceph that's using the live IP range instead of the local IP range... So I need to swap that to the internal IP range

 

[Alwin]

Form Ceph's perspective you can use multiple public/cluster networks, they just need to be routed. Then you can create new Ceph services on the added subnet.

ATM our stack checks the use of multiple networks and forbids to add a new Ceph service. But I am working on a patch for this.
https://bugzilla.proxmox.com/show_bug.cgi?id=2422

 

[infinityM]

Form Ceph's perspective you can use multiple public/cluster networks, they just need to be routed. Then you can create new Ceph services on the added subnet.

ATM our stack checks the use of multiple networks and forbids to add a new Ceph service. But I am working on a patch for this.
https://bugzilla.proxmox.com/show_bug.cgi?id=2422

Ahm... Thanks I think
But I'm still kinda stuck though LOL

 

[Alwin]

But I'm still kinda stuck though LOL

If you explain a bit, maybe we can get you un-stuck?

 

[infinityM]

If you explain a bit, maybe we can get you un-stuck?

I love the way you said that

Ok so I have 2 ethernet ports plugged in, 2 network bridges setup.
1 has IP range 1.1.1.14/28 each server having one of the IP's on the first network port, And a second range 192.168.155.0/24 with each server again having an ip in this range on port/bridge 2

Right now ceph is syncing on 1.1.1.14/28 when syncing data, i'd like it to sync on 192.168.155.0/24 to prevent the connection to outside being slowed down by ceph's syncing...

I hope that's enough info to get me un stuck?
I don't know where to get started in changing it safely?

 

[Alwin]

Right now ceph is syncing on 1.1.1.14/28 when syncing data, i'd like it to sync on 192.168.155.0/24 to prevent the connection to outside being slowed down by ceph's syncing...

That brings me basically back to this:

Form Ceph's perspective you can use multiple public/cluster networks, they just need to be routed. Then you can create new Ceph services on the added subnet.

Both network will need to be routed, so all Ceph services can reach both networks. Once that is done the networks in public_network can be changed to include the new range. Then any new service can be setup with the new IP range. So, removing a Ceph MON and then re-creating it on the network.

Yes, it has to be "complicated" since the MONs save the IP address of their peer MONs in the database. This way they know, who belongs to whom.

 

[kyriazis]

Sorry, late response, but I am wondering what you mean by both networks needs to be routed.

Does that mean:
a). If (let's say) the public network is on subnet A, it can also reach subnet B via subnet A's gateway (ie. reachable)
or
b). overlay networks, ie. have machines with IP addresses belonging on both subnet A and/or subnet B (ie. one machine on subnet A and one machine on subnet B) over the same physical public network switch. (ie. both networks routed through the same switch)

(a) is obviously the simple scenario.

Thanks!