prxoxmox ve 3.2 drbd cluster plus openvswitch

[ladmini]

prxoxmox ve 3.2 drbd cluster plus openvswitch Here is what i have:
a fully working 2 NIC cluster, basically following the wiki : http://pve.proxmox.com/wiki/DRBD

What i like to have: to add open vswitch support.

Tried to add open vswitch via the GUI for private VM connectivity. Only one VM should act as FW/Router with Internet access for the whole cluster, using "vmbr0".
But this failed. afterwards the cluster was not usable anymore.

Setup before creating OVS Bridge:

/etc/network/interfaces
# primary interface
auto eth0
iface eth0 inet static
address 148.XXX.XXX.XXX
netmask 255.255.255.224
gateway 148.XXX.XXX.XXX
broadcast 148.XXX.XXX.XXX
up route add -net 148.XXX.XXX.XXX netmask 255.255.255.224 gw 148.XXX.XXX.XXX eth0

# bridge for routed communication (Hetzner)
# external connection router/firewall-vm, classic Linux Bridge
auto vmbr0
iface vmbr0 inet static
address 148.XXX.XXX.XXX
netmask 255.255.255.248
bridge_ports none
bridge_stp off
bridge_fd 0

# internal connection drbd / cluster
auto eth1
iface eth1 inet static
address 172.24.10.1
netmask 255.255.255.0
Network after adding OVS Bridge with private IP via GUI and a reboot (eth0 and vmbr0 were untouched):

/etc/network/interfaces

...
allow-vmbr1 eth1
iface eth1 inet static
address 172.24.10.1
netmask 255.255.255.0
ovs_type OVSPort
ovs_bridge vmbr1

auto vmbr1
iface vmbr1 inet static
address 192.168.20.1
netmask 255.255.255.0
ovs_type OVSBridge
ovs_ports eth1

Now, node 1 can not ping node 2 anymore, the cluster (connection via 172.24.10.1 and 172.24.10.2) is down.
However, pinging the other OVS Bridge IP (192.168.20.2) is possible.
AFAIK, something is wrong here:
after this is delared "iface eth1 inet static" you can not use "ovs_ports eth1" in "vmbr1" anymore.
Maybe it would work to change "iface eth1 inet static" to "iface int1 inet static" and add "ovs_ports eth1 int1" to "vmbr1".

Is there an howto for adding open vswitch support to a 2-Node drbd cluster? That would be much appreciated.

 

[Mr.Holmes]

Hello ladmini

If you have connected a physical NIC to an ovs-bridge assigning an IP address to that NIC does not work properly any more (don´t know why, but that´s it). You have to use an internal OVS port instead (or assign the IP address to the bridge).

In the current case I would change the relevant parts of /etc/network/interfaces to

# remove the old eth1 entry
#auto eth1                   
        #iface eth1 inet static
        #address  172.24.10.1
        #netmask  255.255.255.0

#create an internal port instead - let´s call it "inport1"
allow-vmbr1 inport1
iface inport1 inet static
        address 172.24.10.1
        netmask 255.255.255.0
        ovs_type OVSIntPort
        ovs_bridge vmbr1


  
 allow-vmbr1 eth1
#define eth1 as "manual" and remove addresses
iface eth1 inet manual
        #address  172.24.10.1
        #netmask  255.255.255.0
        ovs_type OVSPort
        ovs_bridge vmbr1

 auto vmbr1
iface vmbr1 inet static
        address  192.168.20.1
        netmask  255.255.255.0
        ovs_type OVSBridge
#add internal port to bridge
        ovs_ports eth1 inport1

The changes can be also made by GUI (safer, in order to avoid writing mistakes)! Even GUI asks for reboot - it´s enough to copy the content of /etc/network/interfaces.new to /etc/network/interfaces and restart networking.

Kind regards

Mr.Holmes

 

[ladmini]

Thanks for the help!
That is what i thought, it is not possible to declare an IP with an Interface, and then an ovs bridge with that interface. Instead, the IP has to become an IntPort as part of the ovs bridge.
I have followed your suggestions:

allow-vmbr1 inport1
iface inport1 inet static
address 172.24.10.1
netmask 255.255.255.0
ovs_type OVSIntPort
# ovs_type OVSPort
ovs_bridge vmbr1

allow-vmbr1 eth1
iface eth1 inet manual
ovs_type OVSPort
ovs_bridge vmbr1

auto vmbr1
iface vmbr1 inet static
address 192.168.222.1
netmask 255.255.255.0
ovs_type OVSBridge
ovs_ports eth1 inport1

And the same on the other node with IP 172.24.10.2 and 192.168.222.2.

From a network point of view everything looks fine, i can ping everything from anywhere.
However, this seems to be messing with the cluster somehow!
Login via web gui is extremly slow, if even possible. If successfull, one node (left panel) is always turned red.
pvecm nodes shows 2 nodes, but summary even for the green node shows nothing.
There is something strange going on.

 

[Mr.Holmes]

Hello ladmini,

Login via web gui is extremly slow, if even possible. If successfull, one node (left panel) is always turned red.
pvecm nodes shows 2 nodes, but summary even for the green node shows nothing.
There is something strange going on.

Unfortunately I can confirm your observation.

I made an experiment as follows:

In an existing wee-working cluster I "moved" (note: it´s not a physical move, just a logical one) the cliúster-node-IP from eth0 to an internal OVS port which was member of the same OVS bridge as eth0. All the effects you described I hat too. Afterwards I "moved" (i.e. deleted the OVS bridge) ith IP address back to eth0 - everything works fine again now.

Conclusion: somewhere (I searched but didn´t find anything) there is a relation to the NIC (identified rather by name than by MAC address) which manages the (multicast-) connection within the cluster.

I must admit at the moment I have no idea how to keep both the node in the cluster and the OVS bridge .....

But maybe somebody else has?

Kind regards

Mr.Holmes

 

[ladmini]

I must admit at the moment I have no idea how to keep both the node in the cluster and the OVS bridge .....

Me neither. However I found a "solution", I moved the cluster IP to the beginning of the ovs bridge.
And i had to define "eth1" as static, like this:
allow-vmbr1 eth1
iface eth1 inet static
address 192.168.20.1
netmask 255.255.255.0
ovs_type OVSPort
ovs_bridge vmbr1

auto vmbr1
iface vmbr1 inet static
address 172.24.10.1
netmask 255.255.255.0
ovs_type OVSBridge
ovs_ports eth1

This configuration (mis-configuration) seems to work, but now my VMs have to be on the 172.... subnet, but that is okay. Note also that pinging the second node over 192.168.20.2 does not work. Still, it is weird.