[SOLVED] Ceph sending data through wrong interface

[Nikuuuuu]

Hello

I have a 3 node cluster running with ceph. Each server has the internal 1gbe nic for management and a 10gbe nic for the ceph/vm network.
It was all generated through the gui.
For some reason all my ceph traffic is going through my 1gbe nics instead of my 10gbe nics so migrations are limited to 125MB/s.
Is there a way to change this? I don't see any settings in the gui and I don't remember getting the option to select the interface, only set the network (which I set to my 10gbe network).
I don't mind going through the cli if I need to.

My management network is 192.168.100.0/24 (vmbr0) [1gbe]
My ceph/vm network is 10.0.0.0/24 (vmbr1) [10gbe]

My ceph global config:

[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = 10.0.0.250/24
fsid = 162d06c7-25e0-4f6a-a978-7e0cd1abadd6
mon_allow_pool_delete = true
mon_host = 10.0.0.250 10.0.0.248 10.0.0.249
ms_bind_ipv4 = true
ms_bind_ipv6 = false
osd_pool_default_min_size = 2
osd_pool_default_size = 3
public_network = 10.0.0.250/24

Thanks

 

[Nikuuuuu]

I just noticed that my cluster_network is set to the interface ip 10.0.0.250/24 instead of the actual network 10.0.0.0/24
Could this be the issue?

 

[Nikuuuuu]

I think I realized that my problem is not ceph.
Ceph is working correctly and is actually going through my 10gbe nics and I was not looking at the problem correctly.
The problem I am having is that when doing a live migration, it takes the state of the vm from memory and migrates that over the 1gbe network.

 

[Nikuuuuu]

Changing the Migration Settings network to the correct interface fixed the performance issues
It can be found in Datacenter -> Options -> Migration Settings -> Network