Migrating to Open vSwitch

[MimCom]

Now that I have the openvswitch-switch package installed on v3.2, I have GUI options for OVS showing. Given the current lack of documentation, I'm going to see what I can learn with a bit of brute force experimentation. Host is currently using vmbr0 (bridged to eth0) with its primary IP. I added a new vmbr1 (also bridged to eth0) with a new IP in a different subnet and ran into some issues which drove the move to try OVS.

When I try to create a new OVS bridge, I put eth0 in the 'Bridge Ports' field and receive an error saying

port 'eth0' is already used on interface 'vmbr2'  (500)

If I change the vmbr number, the error follows with. Why can I create additional linux bridges to the same physical NIC but not OVS bridges?

 

[MimCom]

More info: I get a similar error:

port 'eth0' is already used on interface 'vmbr1'  (500)

if I enter nothing in the 'Bridge Ports' field, or if I enter an existing vmbr there.

 

[MimCom]

Does anyone have OVS working alongside Linux bridging on the same NIC?

Does OVS require exclusive access?

 

[Mr.Holmes]

Hello mhotel Normally one interface cannot be in 2 bridges at the same time. If you use linuxbridge and openvswitch parallel system (CLI) does not refuse to put a NIC in both, but probably only because they don´t know from each other. I don´t know the reason why you intend, maybe in order to connect with a CT? Indeed, I think here it´s not supported by PROXMOX web interface. But I did it manually and it works. Assuming your CT interface is veth500.0 and your ovs bridge is vmbr4 and vlan 14 is used: ovs-vsctl add-port vmbr4 veth500.0 tag=14 without vlan it´s simply without "tag=14" Unfortunately the command has only effect as long as a CT runs and must be repeated after a restart. Some additional observations regarding ovs: - in some cases after a server reboot manual /etc/init.d/networking restart is necessary - bridging (to VMS or internal ports) works also without physical NIC - even I saw it once now I cannot reproduce to reach a directly specified ip address in CT from a vlan Kind regards Mr.Holmes

 

[MimCom]

The intent is to move the entire host to OVS, assuming that will enable the CTs to talk to the KVM guests as I've been trying to configure for the past couple of weeks. If I delete the original vmbr, I lose the host completely other than console access. Not sure I want to make that leap, just setup some new hosts on OVS and get them talking both to each other and to the new router/upstream. Once that works, then start migrating the production servers over to OVS.

 

[mir]

There must be something wrong in your setup if CT's cannon speak with KVM's!

git.datanom.net is a CT and mail.datanom.net is a KVM.

$ ping -c1 mail.datanom.net
PING mail.datanom.net (172.16.1.2) 56(84) bytes of data.
64 bytes from yggdrasil.datanom.net (172.16.1.2): icmp_req=1 ttl=64 time=1.94 ms


--- mail.datanom.net ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.943/1.943/1.943/0.000 ms

$ ping -c1 git
PING git.datanom.net (172.16.1.11) 56(84) bytes of data.
64 bytes from git.datanom.net (172.16.1.11): icmp_req=1 ttl=64 time=0.405 ms


--- git.datanom.net ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.405/0.405/0.405/0.000 ms

 

[MimCom]

The issue came up when we added a second subnet. Existing hosts are on 192.168.44.x and the new subnet (the first of several planned) is 172.16.88.x, on which there is also a new router served by a new upstream provider. KVM guests can ping the router and some workstations with addresses in the new subnet, but CTs can only connect to each other inside the host -- they can not ping the KVMs or anything outside the box. I've created them with venet addresses and veth interfaces, with and without IPs in the original subnet and the behavior is the same. It's like there's a firewall between the host and the NIC blocking traffic from 172.16.88.x.

 

[mir]

Does your firewall practice local network isolation? Normally you would see an option allowing communication between local nets.
It could also be a firewall rule which considers traffic with src address and dst address both with local addresses as a spoofing attempt.

 

[MimCom]

Not at this time. 192.168.44.x subnet is fed by a pfSense box which knows nothing about 172.16.anything. The new upstream is connected to a Ubiquiti EdgeRouter Lite which currently has a minimal ruleset (NAT, block all inbound connections.) One of the KVM hosts is Untangle, which I plan to setup using multiple VLANs. OVS should facilitate this once I have it working, but for now I'm not using VLANs at all -- just trying to establish connectivity first.

 

[mir]

I am not referring to VLAN's, just simple routing.

 

[MimCom]

Understood -- just wanted to be clear that we were not using VLANs at this time, and that there is currently no firewall between the internal subnets.



Update: Trying to create a new Linux bridge with a port on eth0 gives me the same kind of error:

port 'eth0' is already used on interface 'vmbr2'  (500)

The existing vmbr1 bridge (172.16.88.20) was created while the host was running v2.3.

 

[Mr.Holmes]

Trying to create a new Linux bridge with a port on eth0 gives me the same kind of error:

port 'eth0' is already used on interface 'vmbr2'  (500)

The existing vmbr1 bridge (172.16.88.20) was created while the host was running v2.3.

The message is a (minor) bug - correct would be 'eth0' is already used on interface 'vmbr0' or 'vmbr1'. When I tested I got sometimes the same kind of bug. But serious is that eth0 is shown in 2 bridges at the same time! Maybe it´s just incorrectly shown by web interface or - more severe - something destroyed in linux. I made a short test directly in linux to put eth0 into a second bridge:

root@ebux:~# brctl show
bridge name bridge id STP enabled interfaces
vmbr0 8000.14dae9751f6b no eth0
vmbr1 8000.00e052c0f669 no eth1
root@ebux:~# brctl addbr vtest
root@ebux:~# brctl addif vtest eth0
device eth0 is already a member of a bridge; can't enslave it to bridge vtest.

here it was correctly refused.

Check with cat /etc/network/interfaces and brctl show what´s going on.

Don´t know why, but in my earlier post formatting disappeared, so I repeat how to enable access into CT via OVS:

It´s not supported by PROXMOX web interface. But I did it manually and it works.
Assuming your CT interface is veth500.0 and your ovs bridge is vmbr4 and vlan 14 is used:

ovs-vsctl add-port vmbr4 veth500.0 tag=14

without vlan it´s simply without "tag=14"

Unfortunately the command has only effect as long as a CT runs and must be repeated after a restart.

Don´t worry about loosing access to console in CT, you can always use vzctl enter <CT-id>

 

[MimCom]

The message is a (minor) bug - correct would be 'eth0' is already used on interface 'vmbr0' or 'vmbr1'. When I tested I got sometimes the same kind of bug. But serious is that eth0 is shown in 2 bridges at the same time! Maybe it´s just incorrectly shown by web interface or - more severe - something destroyed in linux. I made a short test directly in linux to put eth0 into a second bridge:

root@ebux:~# brctl show
bridge name bridge id STP enabled interfaces
vmbr0 8000.14dae9751f6b no eth0
vmbr1 8000.00e052c0f669 no eth1
root@ebux:~# brctl addbr vtest
root@ebux:~# brctl addif vtest eth0
device eth0 is already a member of a bridge; can't enslave it to bridge vtest.

here it was correctly refused.

Thanks, that does actually make more sense.

As it turns out, vmbr1 does not actually have eth0 in it, which would explain the lack of connectivity across those addresses:

root@pve1:~# brctl show
bridge name    bridge id        STP enabled    interfaces
vmbr0        8000.002354458b04    no        eth0
                            veth103.0
                            veth105.0
vmbr0v33        8000.002354458b04    no        eth0.33
vmbr0v88        8000.002354458b04    no        eth0.88
vmbr0v99        8000.002354458b04    no        eth0.99
vmbr1        8000.000000000000    no        
root@pve1:~#

Don´t know why, but in my earlier post formatting disappeared

I do recall reading that, but the restart issue is a blocker.

All I am really looking for is a way to build a few private links between Untangle, some internal servers, and two different WLANs, eventually using VLANs. If I were to remove the existing vmbr0 for the whole host and replace that with an OVS bridge, would that give me more flexibility in addressing?

 

[Mr.Holmes]

If I were to remove the existing vmbr0 for the whole host and replace that with an OVS bridge, would that give me more flexibility in addressing?

Let´s rather say "configuring virtual switches" - Sure, that´s the reason why using OVS - advantages compared with standard linux bridging.

 

[MimCom]

Right, makes perfect sense. So how do I migrate an existing host from Linux bridging to OVS? Is it as simple as removing the Linux bridge and adding the OVS bridge to the node? My concern is that I lose connectivity with the host during the process and run the potential risk of losing all net connectivity if something goes awry.

 

[Mr.Holmes]

In principle: YES, if possible connect keyboard & vga screen directly to server - you can never exclude making a mistake. If not: be EXTREMELY careful, you cannot afford any mistake then!

Method:
- remove all linux vmbr´s
- configure OVS vmbr
- assign (the basic) address to it
- verify again
- reboot (changes in network will be made after reboot only)
- then add step by step all the vlan stuff

 

[MimCom]

About what I figured. The VLANs never worked right anyway, so I'll just delete them before I start.
Edit:

root@pve1:~# brctl show
bridge name    bridge id        STP enabled    interfaces
vmbr0        8000.002354458b04    no        eth0

I'm hoping once I have the OVS bridge on eth0 that I'll be able to create multiple OVS IntPorts in different subnets, then bridge guests to those.

thanks~

 

[MimCom]

Straight replacement won't fly using the GUI. I removed the existing vmbr0, but when trying to add a new one I get a conflict with the (supposedly removed) config:


So I created a new vmbr1 with no IP address, then an IntPort with the existing host IP and gateway settings:


Here are the changes:

--- /etc/network/interfaces	2014-04-17 10:30:17.000000000 -0600
+++ /etc/network/interfaces.new	2014-04-17 11:27:20.000000000 -0600
@@ -1,15 +1,27 @@
 # network interface settings
+allow-vmbr1 port0
+iface port0 inet static
+	address  192.168.44.20
+	netmask  255.255.255.0
+	gateway  192.168.44.1
+	ovs_type OVSIntPort
+	ovs_bridge vmbr1
+
 auto lo
 iface lo inet loopback

Is this the right heirarchy? Is there any standard naming convention for an IntPort? I used port0 for this.

 

[MimCom]

Decided to move the existing Linux bridge to vmbr1 in order to end up with an OVS bridge named vmbr0. Turns out that removing an active bridge really just deactivates it:


Removing again and restarting does not change this:

--- /etc/network/interfaces    2014-04-17 11:54:24.000000000 -0600
+++ /etc/network/interfaces.new    2014-04-17 11:57:33.000000000 -0600
@@ -4,11 +4,6 @@
 
 iface eth0 inet manual
 
-iface vmbr0 inet manual
-    bridge_ports none
-    bridge_stp off
-    bridge_fd 0
-
 auto vmbr1
 iface vmbr1 inet static
     address  192.168.44.20

Yet:

# cat /etc/network/interfaces 
# network interface settings
auto lo
iface lo inet loopback

iface eth0 inet manual

auto vmbr1
iface vmbr1 inet static
    address  192.168.44.20
    netmask  255.255.255.0
    gateway  192.168.44.1
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0

Does Proxmox have some kind of hard-coded reference to vmbr0?

 

[Mr.Holmes]

Obviously there is a mismatch between gui and real configuration. What count is content of /etc/network/interfaces, probably vmbr does not exist there any more - in that case I would ignore the problem , you can use vmbr1 instead.

How to call a port is completely free - why not port0, port1 ...


Yes, I think it is the right hierarchy.

For better understanding I made a "global example" where I combined a lot of configuration variants (regardless if a real customer will ever need such a configuration), I´ve tested by trying pings between the endpoints. Here the description:

Global example for OVS use in PROXMOX:


Scenario:


LAN physically connected with eth2
- transports 3 vlan
tag=11 192.168.11.0/24
tag=12 192.168.12.0/24
tag=14 192.168.14.0/24
- and as non-vlan
notag 192.168.10.0/24
implemented as vmbr2

local virtual network (is NOT a vlan, just a LAN without wire)
192.168.13.0/24
implemented as vmbr3

6 virtual machines
ct 10
ct 11
ct 12
vm 10
vm 11
vm 12

each machine is connected with
192.168.xx.0/24 (xx = machine number)
as well as
192.168.13.0/24

Config files


If not mentioned otherwise all can be configured by PROXMOX web gui, after configuring all and all machines up they have to look as follows:

---- /etc/network/interfaces -----

allow-vmbr2 ipo14 <==== this is the virtual port for PROXMOST host
iface ipo14 inet static
address 192.168.14.1
netmask 255.255.255.0
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=14


allow-vmbr2 eth2
iface eth2 inet manual
ovs_type OVSPort
ovs_bridge vmbr2


auto vmbr2
iface vmbr2 inet manual
ovs_type OVSBridge
ovs_ports eth2 ipo14

auto vmbr3 <===== a bridge without physical connection
iface vmbr3 inet static
address 192.168.13.1
netmask 255.255.255.0
ovs_type OVSBridge


VM config files /etc/pve/qemu-server/xxx.conf:

---- vm 10 -----
name: LUC-10
net0: virtio=0AC:F5:197:A0,bridge=vmbr3 <=== for PROXMOX internal network 192.168.13.0/24
net1: virtio=72:E2:52:57:C21,bridge=vmbr2 <=== no vlan tag for network 192.168.10.0/24

---- vm 11 -----
name: LUC-11
net0: virtio=BE:23:5F:9B:F0:1D,bridge=vmbr3
net1: virtio=DE:10:31:72:90:17,bridge=vmbr2,tag=11 <=== tag 11 for network 192.168.11.0/24

----- vm 12 ----
net0: virtio=E2:B9:62:1A:AD:AD,bridge=vmbr3
net1: virtio=42:F2:90:73:BA:94,bridge=vmbr2,tag=12 <=== tag 12 for network 192.168.12.0/24


CT config files /etc/pve/openvz/xxx.conf:

--- ct 10 ----
NETIF="ifname=eth0,bridge=vmbr2,mac=A2:AE:5B:78:B4:90,host_ifname=veth301.0,host_mac=DA:96:1A:65:B0:4D;
ifname=eth1,bridge=vmbr3,mac=CE:83:20:87:5E:2A,host_ifname=veth301.1,host_mac=CE:06:B8:49:C0:67"

---- ct 11 ---
NETIF="ifname=eth0,bridge=vmbr2,mac=FE:39:16:E6:F3:3A,host_ifname=veth302.0,host_mac=4E:5C:88:32:40:AD;
ifname=eth1,bridge=vmbr3,mac=92:BB:3F:FB:35:F0,host_ifname=veth302.1,host_mac=E2:58:3C:E1:35:BA"

---- ct 12 ----
NETIF="ifname=eth0,bridge=vmbr2,mac=82:79:1D:B8:5B:18,host_ifname=veth303.0,host_mac=7A:84:EA:EC:94:30;
ifname=rth1,bridge=vmbr3,mac=26:C9:2C:51:45:60,host_ifname=veth303.1,host_mac=D6:F5:13:31:6A:F7"

----- ovs-vsctl show -------------
Bridge "vmbr3"
Port "veth303.1"
Interface "veth303.1" <====
Port "tap103i0"
Interface "tap103i0"
Port "veth301.1" <==== configuring OVS bridges to CTs by gui does not work - use "ovs-vsctl add-port vmbr3 veth301.1 [tag=xx]" etc. instead
Interface "veth301.1"
Port "vmbr3"
Interface "vmbr3"
type: internal
Port "veth302.1"
Interface "veth302.1" <====
Port "tap102i0"
Interface "tap102i0"
Port "tap101i0"
Interface "tap101i0"
Bridge "vmbr2"
Port "tap102i1"
tag: 11
Interface "tap102i1"
Port "veth301.0" <====
Interface "veth301.0"
Port "veth303.0" <====
tag: 12
Interface "veth303.0"
Port "tap103i1"
tag: 12
Interface "tap103i1"
Port "ipo14"
tag: 14
Interface "ipo14"
type: internal
Port "eth2"
Interface "eth2"
Port "veth302.0"
tag: 11
Interface "veth302.0" <====
Port "tap101i1"
Interface "tap101i1"
Port "vmbr2"
Interface "vmbr2"
type: internal