OpenVswitch bridge for Cluster VMs across private network

T

tuathan

Member
May 23, 2020
52
6
8
I have set up a cluster with VMs on different host nodes: h1, h2, h3, h4

I have used a OpenVswitch bridge (vmbr2) defined on h1:

auto vmbr2
iface vmbr2 inet manual
ovs_type OVSBridge
post-up ovs-vsctl add-port vmbr2 gre1 -- set interface gre1 type=gre options:remote_ip=''ip h2''
post-up ovs-vsctl add-port vmbr2 gre2 -- set interface gre2 type=gre options:remote_ip=''ip h3''
post-up ovs-vsctl add-port vmbr2 gre3 -- set interface gre3 type=gre options:remote_ip=''ip h4''


On h2 ,h3 and h4 I have also defined an ovsbridge (vmbr2). E.g. On h2 like this:

auto vmbr2
iface vmbr2 inet manual
ovs_type OVSBridge
post-up ovs-vsctl add-port vmbr2 gre1 -- set interface gre1 type=gre options:remote_ip=''ip h1''

I then use the vmbr2 device for VMs on private network (192.168.2.X) and have a connection between VMs on different hosts (i.e. can ping/SSH/scp between VMs on h1 and others and h2 and h3 and h3 and h4) unless h1 goes down.

Do I need to configure ovsbridge type devices on h2, h3, h4 and configure as I have? Have I done this correctly?

How do I make my setup redundant so if h1 is down VMs on private network on h2, h3 and h4 can still communicate?

Do I need separate OVS bridge devices and configuration between each pair of hosts: i.e. h1 and h2, h2 and h3, h3 and h4 etc.
 
Last edited:
you shouldn't use gre tunnel for this. It could works 3 host max + spanning tree to avoid loop, but I'll never be super stable.
and I don't think it'll works with more than 3 hosts.
If you want to test, you really need to do a full mesh between all yours nodes. (on each node, create 1 gre tunnel to each other nodes), and enabled spanning tree option "ovs_extra set Bridge vmbr2 rstp_enable=true"



for your kind of setup, you should use vxlan, when 1 tunnel can be connected to multiple hosts.


They are a beta test for the new sdn feature coming soon in proxmox, with vxlan support added
https://forum.proxmox.com/threads/proxmox-6-2-sdn-beta-test.69655/page-10#post-341003
 
spirit said:
you shouldn't use gre tunnel for this. It could works 3 host max + spanning tree to avoid loop, but I'll never be super stable.
and I don't think it'll works with more than 3 hosts.
If you want to test, you really need to do a full mesh between all yours nodes. (on each node, create 1 gre tunnel to each other nodes), and enabled spanning tree option "ovs_extra set Bridge vmbr2 rstp_enable=true"



for your kind of setup, you should use vxlan, when 1 tunnel can be connected to multiple hosts.


They are a beta test for the new sdn feature coming soon in proxmox, with vxlan support added
https://forum.proxmox.com/threads/proxmox-6-2-sdn-beta-test.69655/page-10#post-341003
Click to expand...

Thanks. That future SDN feature looks like it will be very useful!

In terms of GRE tunnels I came across the need for STP for what I wanted to do and in the Proxmox documentation that to enable RSTP using:

up ovs-vsctl set bridge ${IFACE} rstp_enable=true

But here:

https://serverfault.com/questions/8...h-without-loops-and-forwards-with-openvswitch

it was done like this:

up ovs-vsctl set bridge ${IFACE} stp_enable=true

any ideas what the difference would be?

Is there any relatively simple documentation to follow on vxlan setup on hosts for cross-cluster VM communications?
 
it was done like this:

up ovs-vsctl set bridge ${IFACE} stp_enable=true

any ideas what the difference would be?
Click to expand...

I think it's spanning tree vs rapid spanning tree. (rstp is more modern with faster convergence).
But keep in mind that it's still spanning tree, so when a node will go down, you'll have a small lag on all your gre network.


Is there any relatively simple documentation to follow on vxlan setup on hosts for cross-cluster VM communications?
Click to expand...

If you don't want to do it with new sdn features, it's still possible to do it manually in /etc/network/interfaces
(you need to install ifupdown2 package)

I have wrote some doc here:
https://git.proxmox.com/?p=pve-docs.git;a=blob_plain;f=vxlan-and-evpn.adoc;hb=HEAD
->VXLAN layer2 with vlan unware linux bridges->unicast_mode

for your setup it'll be something like:

for node1
--------------
Code: 
auto vmbr2
iface vmbr2 inet manual
        bridge_ports vxlan2
        bridge_stp off
        bridge_fd 0


auto vxlan2
iface vxlan2 inet manual
        vxlan-id 2
        vxlan_remoteip X.X.X.X (ip h2)
        vxlan_remoteip X.X.X.X (ip h3)
        vxlan_remoteip X.X.X.X (ip h4)

do the same on others nodes, with correct vxlan_remoteip.


(you can also use vlan aware bridge fro vmbr2, if you want to use vlans on top of vxlan).

if you need multiple vxlan, you need to define multiple vmbr2/vxlan couple currently.
 
Last edited:
  • Like
Reactions: tuathan
spirit said:
I think it's spanning tree vs rapid spanning tree. (rstp is more modern with faster convergence).
But keep in mind that it's still spanning tree, so when a node will go down, you'll have a small lag on all your gre network.




If you don't want to do it with new sdn features, it's still possible to do it manually in /etc/network/interfaces
(you need to install ifupdown2 package)

I have wrote some doc here:
https://git.proxmox.com/?p=pve-docs.git;a=blob_plain;f=vxlan-and-evpn.adoc;hb=HEAD
->VXLAN layer2 with vlan unware linux bridges->unicast_mode

for your setup it'll be something like:

for node1
--------------
Code: 
auto vmbr2
iface vmbr2 inet manual
        bridge_ports vxlan2
        bridge_stp off
        bridge_fd 0


auto vxlan2
iface vxlan2 inet manual
        vxlan-id 2
        vxlan_remoteip X.X.X.X (ip h2)
        vxlan_remoteip X.X.X.X (ip h3)
        vxlan_remoteip X.X.X.X (ip h4)

do the same on others nodes, with correct vxlan_remoteip.


(you can also use vlan aware bridge fro vmbr2, if you want to use vlans on top of vxlan).

if you need multiple vxlan, you need to define multiple vmbr2/vxlan couple currently.
Click to expand...


Thanks a lot! This is super helpful.

I've started off by removing the ovswitch bridges from my nodes and tried to install ifupdown2 but getting this error?!

# apt install ifupdown2
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
python-argcomplete
Suggested packages:
python-gvgen
The following packages will be REMOVED:
ifenslave ifupdown libproxmox-acme-perl libpve-access-control libpve-cluster-api-perl libpve-cluster-perl libpve-common-perl libpve-guest-common-perl
libpve-http-server-perl libpve-storage-perl librados2-perl proxmox-ve pve-cluster pve-container pve-firewall pve-ha-manager pve-manager pve-xtermjs qemu-server
The following NEW packages will be installed:
ifupdown2 python-argcomplete
0 upgraded, 2 newly installed, 19 to remove and 73 not upgraded.
Need to get 0 B/225 kB of archives.
After this operation, 14.3 MB disk space will be freed.
Do you want to continue? [Y/n] y
W: (pve-apt-hook) !! WARNING !!
W: (pve-apt-hook) You are attempting to remove the meta-package 'proxmox-ve'!
W: (pve-apt-hook)
W: (pve-apt-hook) If you really want to permanently remove 'proxmox-ve' from your system, run the following command
W: (pve-apt-hook) touch '/please-remove-proxmox-ve'
W: (pve-apt-hook) run apt purge proxmox-ve to remove the meta-package
W: (pve-apt-hook) and repeat your apt invocation.
W: (pve-apt-hook)
W: (pve-apt-hook) If you are unsure why 'proxmox-ve' would be removed, please verify
W: (pve-apt-hook) - your APT repository settings
W: (pve-apt-hook) - that you are using 'apt full-upgrade' to upgrade your system
E: Sub-process /usr/share/proxmox-ve/pve-apt-hook returned an error code (1)
E: Failure running script /usr/share/proxmox-ve/pve-apt-hook

Think I might need to update the repos: https://pve.proxmox.com/wiki/Package_Repositories


UPDATE: editing apt to point to no-subscription repos did the trick
 
Last edited:
spirit said:
I think it's spanning tree vs rapid spanning tree. (rstp is more modern with faster convergence).
But keep in mind that it's still spanning tree, so when a node will go down, you'll have a small lag on all your gre network.




If you don't want to do it with new sdn features, it's still possible to do it manually in /etc/network/interfaces
(you need to install ifupdown2 package)

I have wrote some doc here:
https://git.proxmox.com/?p=pve-docs.git;a=blob_plain;f=vxlan-and-evpn.adoc;hb=HEAD
->VXLAN layer2 with vlan unware linux bridges->unicast_mode

for your setup it'll be something like:

for node1
--------------
Code: 
auto vmbr2
iface vmbr2 inet manual
        bridge_ports vxlan2
        bridge_stp off
        bridge_fd 0


auto vxlan2
iface vxlan2 inet manual
        vxlan-id 2
        vxlan_remoteip X.X.X.X (ip h2)
        vxlan_remoteip X.X.X.X (ip h3)
        vxlan_remoteip X.X.X.X (ip h4)

do the same on others nodes, with correct vxlan_remoteip.


(you can also use vlan aware bridge fro vmbr2, if you want to use vlans on top of vxlan).

if you need multiple vxlan, you need to define multiple vmbr2/vxlan couple currently.
Click to expand...

This seems to work beautifully!
spirit said:
I think it's spanning tree vs rapid spanning tree. (rstp is more modern with faster convergence).
But keep in mind that it's still spanning tree, so when a node will go down, you'll have a small lag on all your gre network.




If you don't want to do it with new sdn features, it's still possible to do it manually in /etc/network/interfaces
(you need to install ifupdown2 package)

I have wrote some doc here:
https://git.proxmox.com/?p=pve-docs.git;a=blob_plain;f=vxlan-and-evpn.adoc;hb=HEAD
->VXLAN layer2 with vlan unware linux bridges->unicast_mode

for your setup it'll be something like:

for node1
--------------
Code: 
auto vmbr2
iface vmbr2 inet manual
        bridge_ports vxlan2
        bridge_stp off
        bridge_fd 0


auto vxlan2
iface vxlan2 inet manual
        vxlan-id 2
        vxlan_remoteip X.X.X.X (ip h2)
        vxlan_remoteip X.X.X.X (ip h3)
        vxlan_remoteip X.X.X.X (ip h4)

do the same on others nodes, with correct vxlan_remoteip.


(you can also use vlan aware bridge fro vmbr2, if you want to use vlans on top of vxlan).

if you need multiple vxlan, you need to define multiple vmbr2/vxlan couple currently.
Click to expand...

Just to feedback vxlans work beautifully! Definitely better than openvswitch (ovs) bridge with GRE tunnels!

For my cluster I have configured vmbr2 in /etc/network/interfaces as you mentioned on each host:

Code: 
auto vmbr2
iface vmbr2 inet manual
        bridge_ports vxlan2
        bridge_stp off
        bridge_fd 0

auto vxlan2
iface vxlan2 inet manual
        vxlan-id 2
        vxlan_remoteip X.X.X.X (ip h2)
        vxlan_remoteip X.X.X.X (ip h3)
        vxlan_remoteip X.X.X.X (ip h4)

and then rebooted the pve host node and for VMS on each host used the bridge vmbr2 (Hardware>Network with virtio and no firewall, default other settings) and within the VM OS assigned static IPs in range 192.168.2.X, mask 255.255.255.0 (no gateway).

ping/ssh/scp works without issue between VMs on different nodes! (definately more stable and faster than ovs/gre)

One thing I wonder about (and could test I suppose) would does this handle migration on a VM to another pve host node?

Thanks again.
 
tuathan said:
One thing I wonder about (and could test I suppose) would does this handle migration on a VM to another pve host node?

Thanks again.
Click to expand...
yes, sure, you can of course do live migration between nodes.

about vxlan, just be carefull that the tunnel encapsulation take 56 bytes, so you need to increase mtu on physical interfaces on your hosts or decrease mtu inside the vms, or maybe you'll have problem with ssh/https or other traffic where you can't have fragmented packets.

(for gre tunnel it's 24bytes)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom