[SOLVED] Mesh network for 3 ceph servers and 4 nodes

Rudy

Renowned Member
Feb 25, 2012
19
0
66
Hello,

I have a 3 proxmox nodes cluster with ceph using an InfiniBand (IB) mesh network. I used the method 2 from this tuto to configure it since broadcast is not supported by IB.
Today, I decided to add a 4th node but I don't manage to correctly configure my network to access the ceph network from the 4th node using the communication network. Any suggestion would be great (my cluster communication is 192.168.1.x using a vlan bond, a vlan can be configure, I was thinking to bridge a vlan but in this case, I could have loop problem).

Thank you in advance for your help,
Best regards,
Rudy



First interfaces for node 1
Code:
auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

auto ib0
iface ib0 inet static
        address  10.0.1.11
        netmask  255.255.255.0
        pre-up modprobe ib_ipoib
        pre-up echo connected > /sys/class/net/ib0/mode
        mtu 65520
        up ip route add 10.0.1.12/32 dev ib0
        down ip route del 10.0.1.12/32

auto ib1
iface ib1 inet static
        address  10.0.1.11
        netmask  255.255.255.0
        pre-up modprobe ib_ipoib
        pre-up echo connected > /sys/class/net/ib1/mode
        mtu 65520
        up ip route add 10.0.1.13/32 dev ib1
        down ip route del 10.0.1.13/32

auto bond0
iface bond0 inet manual
        slaves eth0 eth1
        bond_miimon 100
        bond_mode 802.3ad
        bond_xmit_hash_policy layer2+3

auto vmbr0
iface vmbr0 inet static
        address  192.168.1.11
        netmask  255.255.255.0
        gateway  192.168.1.250
        bridge_ports bond0.1
        bridge_stp off
        bridge_fd 0
 
Hi,

By adding a route in the new node to the proxmox ceph network, it works now; the node 4 can access the ceph storage by using the vmbr0/lan network (1Gbit/s)

In my case :
ip route add 10.0.1.11 via 192.168.1.11
ip route add 10.0.1.12 via 192.168.1.12
ip route add 10.0.1.13 via 192.168.1.13

Kind regards,
Rudy
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!