[SOLVED] Mesh network for 3 ceph servers and 4 nodes

[Rudy]

Hello,

I have a 3 proxmox nodes cluster with ceph using an InfiniBand (IB) mesh network. I used the method 2 from this tuto to configure it since broadcast is not supported by IB.
Today, I decided to add a 4th node but I don't manage to correctly configure my network to access the ceph network from the 4th node using the communication network. Any suggestion would be great (my cluster communication is 192.168.1.x using a vlan bond, a vlan can be configure, I was thinking to bridge a vlan but in this case, I could have loop problem).

Thank you in advance for your help,
Best regards,
Rudy



First interfaces for node 1

auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

auto ib0
iface ib0 inet static
        address  10.0.1.11
        netmask  255.255.255.0
        pre-up modprobe ib_ipoib
        pre-up echo connected > /sys/class/net/ib0/mode
        mtu 65520
        up ip route add 10.0.1.12/32 dev ib0
        down ip route del 10.0.1.12/32

auto ib1
iface ib1 inet static
        address  10.0.1.11
        netmask  255.255.255.0
        pre-up modprobe ib_ipoib
        pre-up echo connected > /sys/class/net/ib1/mode
        mtu 65520
        up ip route add 10.0.1.13/32 dev ib1
        down ip route del 10.0.1.13/32

auto bond0
iface bond0 inet manual
        slaves eth0 eth1
        bond_miimon 100
        bond_mode 802.3ad
        bond_xmit_hash_policy layer2+3

auto vmbr0
iface vmbr0 inet static
        address  192.168.1.11
        netmask  255.255.255.0
        gateway  192.168.1.250
        bridge_ports bond0.1
        bridge_stp off
        bridge_fd 0

 

[Rudy]

Hi,

By adding a route in the new node to the proxmox ceph network, it works now; the node 4 can access the ceph storage by using the vmbr0/lan network (1Gbit/s)

In my case :
ip route add 10.0.1.11 via 192.168.1.11
ip route add 10.0.1.12 via 192.168.1.12
ip route add 10.0.1.13 via 192.168.1.13

Kind regards,
Rudy