Ceph cluster private network help

[dwheeler]

Hello I am new to proxmox and need help figuring out how to change my ceph traffic to use a different network.
Bond1 and Bond2 are what I want my ceph traffic to use as each of those bonds is 4x25gb.
Bond 1 and 2 share a switch that is seperated from everything else
I tried just changing the cluster_network but that didn't work.

 

[Falk R.]

Hello I am new to proxmox and need help figuring out how to change my ceph traffic to use a different network.
Bond1 and Bond2 are what I want my ceph traffic to use as each of those bonds is 4x25gb.
Bond 1 and 2 share a switch that is seperated from everything else
I tried just changing the cluster_network but that didn't work.

View attachment 72470

View attachment 72471

You have mixed the Ceph configuration and Cluster/Corosync configuration.

Your 10.166.x.x network has no place in the Ceph configuration, this is intended for the host and the VMs.

Ceph Cluster is the backend traffic (OSD to OSD replication traffic).
Ceph Client Network is the network where the clients (storage consumers) access the OSD and monitor.

Currently your storage traffic and replication traffic runs over the VM network.

 

[dwheeler]

You have mixed the Ceph configuration and Cluster/Corosync configuration.

Your 10.166.x.x network has no place in the Ceph configuration, this is intended for the host and the VMs.

Ceph Cluster is the backend traffic (OSD to OSD replication traffic).
Ceph Client Network is the network where the clients (storage consumers) access the OSD and monitor.

Currently your storage traffic and replication traffic runs over the VM network.

Ok so I should swap out both the public_network and cluster_network to use the 10.1.1.x Ip address instead?

 

[Falk R.]

Two IPs from one subnet from one host is never a good idea.
Either you change the subnet on a bond and use one subnet for CLient and one for cluster traffic or you build a bond across all 8 interfaces and run all traffic over it.

I hope you have configured LACP Layer3+4. Otherwise the balancing will not be good.

Of course you also have to rebuild the monitors and I have no idea if there is a way to make all these changes on the fly.
I would completely redo the Ceph configuration, which is probably the quickest way.

 

[dwheeler]

Two IPs from one subnet from one host is never a good idea.
Either you change the subnet on a bond and use one subnet for CLient and one for cluster traffic or you build a bond across all 8 interfaces and run all traffic over it.

I hope you have configured LACP Layer3+4. Otherwise the balancing will not be good.

Of course you also have to rebuild the monitors and I have no idea if there is a way to make all these changes on the fly.
I would completely redo the Ceph configuration, which is probably the quickest way.

I have reset Ceph on all 9 of my hosts and fixed the double Ips. However when i setup and configure ceph to run on 10.1.1.11 after i create 1 monitor i can no longer create another, i just get a timeout. All the hosts can ping each other on the 10.1.1.x network

There is something that I am either missing or misunderstanding. Thanks for the help!

 

[dwheeler]

I have reset Ceph on all 9 of my hosts and fixed the double Ips. However when i setup and configure ceph to run on 10.1.1.11 after i create 1 monitor i can no longer create another, i just get a timeout. All the hosts can ping each other on the 10.1.1.x network

There is something that I am either missing or misunderstanding. Thanks for the help!

View attachment 72473
View attachment 72474

View attachment 72475
View attachment 72476

Does it just not like all the bonded ports?

 

[Falk R.]

I have reset Ceph on all 9 of my hosts and fixed the double Ips. However when i setup and configure ceph to run on 10.1.1.11 after i create 1 monitor i can no longer create another, i just get a timeout. All the hosts can ping each other on the 10.1.1.x network

There is something that I am either missing or misunderstanding. Thanks for the help!


View attachment 72475

You must use the 10.1.1.x IP and not the DNS Name. The DNS Name Pointers to the wrong Network.

 

[dwheeler]

You must use the 10.1.1.x IP and not the DNS Name. The DNS Name Pointers to the wrong Network.

I guess I'm not seeing how to use only an IP and not a host name at all?
Also thank you really I appreciate the help.

 

[Falk R.]

Can you ping the other Interfaces and have you check the MTU?

 

[dwheeler]

Can you ping the other Interfaces and have you check the MTU?

All the interfaces could ping each other, i believe the isolated switch they use for that network is set up properly.

I thought i had configured everything properly to use MTU 9000 for those bonds but after switching them back to default (1500) it seems the connection is a lot more stable and i can actually connect all the hosts as monitors.

The gui seems to be a bit slow/laggy when looking at ceph.

But I believe this is now properly set up where Ceph is on its own network?

 

[Falk R.]

The configuration looks good.

I suspect the switch is not configured correctly, although I am also interested in what kind of switch you are using if you have 72 x 25GBit ports. I suspect that it is a stack.
You can adjust the MTU afterwards, but you should also look at the LACP configuration on the switch. If the switch supports Layer 3+4, you should also set this on your bond.

Background: You only have one IP or MAC address and the load balancing is not that great. As each OSD communicates via its own port, load balancing works much more efficiently with LACP Layer 3+4.

 

[dwheeler]

The configuration looks good.

I suspect the switch is not configured correctly, although I am also interested in what kind of switch you are using if you have 72 x 25GBit ports. I suspect that it is a stack.
You can adjust the MTU afterwards, but you should also look at the LACP configuration on the switch. If the switch supports Layer 3+4, you should also set this on your bond.

Background: You only have one IP or MAC address and the load balancing is not that great. As each OSD communicates via its own port, load balancing works much more efficiently with LACP Layer 3+4.

thanks very helpful! Now that its working ill try and troubleshoot the stability issues with the tips you gave.
The migration speed is 5-10x faster now that its on the correct network.

the switch is a
Dell Networking S5048F-ON:

• 48 x 25 Gigabit SFP28 + 6 x 100 Gigabit QSFP28 (The 6x 100 G is broken out with breakouts to 25G)

 

[Falk R.]

I don't think this is a good configuration.

You have good redundancy with 9 servers, but a single point of failure, your switch.
I would definitely make it redundant. The DELL switches can do VLT (something like MLAG) and with that you could also build a LACP over two switches.

With 6x 100 GBit you also have more than enough bandwidth for the peer link.