Openvswitch issues

I

igurvich

New Member
Sep 18, 2017
12
0
1
31
So I'm trying to configure vswitch on my 4-node cluster. I've managed to make a bridge on node 1 but when I use the same configuration for the other nodes they a) break after reboot and seems like they won't go up and b) show an error when I dun /etc/init.d/networking restart. this is despite that when the interfaces were up they could ping each other and had internet access. Also when they could ping the cluster broke down and I couldn't access it through the web portal.

Below are the /etc/network/interface files for node 1 (working) and node 2 (the rest are virtually identical except different ip addresses)

prox1.PNG prox2.PNG
 
Hi,

I think you have installed the wrong openvswitch packages.
What is the output of

Code: 
dpkg -l openvswitch-*
 
This is it. I run apt-get install openvswitch-switch
 

Attachments

  • output.PNG
    output.PNG
    9.5 KB · Views: 127
This are the debian openvswitch package and not compatible with PVE.
Set up your repository and install the correct packages.
 
wolfgang said:
This are the debian openvswitch package and not compatible with PVE.
Set up your repository and install the correct packages.
Click to expand...
How can this be? I'm using the newest version on proxmox and haven't touched the repository. How did I downloaded the wrong version?
 
OK so after a couple of tweaks I am able to bring it up on each machine individually but it takes a couple of resets for the init.d/networking but it brings up the ports and interfaces. The issue is that it breaks down the cluster and I can't access it via the web portal (can't start cmap service).
 
You have to use the 2.7.0 OVS version which is in the proxmox repo and compiled for debian 9.
 
How? I've put the no-subscription repository in the source list but when I try update openvswitch it tells me its already at the newest version
 
You removed the enterprise repo right? Try to remove the installed ovs package and reinstall it
 
Just to document this for others.

Installed Proxmox 5, upgraded to 5.1 and were waiting for customer to sign paper work for us to activate subscription. We subsequently added the 'non-subscription' repository to keep the system up to date. Installing openvswitch and configuring everything results in networking not starting at bootup. Subscribing the box and then simply running 'apt-get update; apt-get -y dist-upgrade' results in various packages being upgraded and openvswitch thereafter working perfectly.

Herewith the content of our /etc/network/interfaces configuration file:
Code: 
auto lo
iface lo inet loopback

allow-vmbr0 bond0
iface bond0 inet manual
        ovs_bridge vmbr0
        ovs_type OVSBond
        ovs_bonds eth0 eth1
        pre-up ( ifconfig eth0 mtu 9000 && ifconfig eth1 mtu 9000 )
        ovs_options bond_mode=active-backup tag=1 vlan_mode=native-untagged
        mtu 9000

auto vmbr0
allow-ovs vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports bond0 vlan1
        mtu 9000

allow-vmbr0 vlan1
iface vlan1 inet static
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=1
        ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
        address 192.168.1.26
        netmask 255.255.255.0
        gateway 192.168.1.1
        mtu 1500

auto eth2
iface eth2 inet manual

auto eth3
iface eth3 inet manual

PS: Proxmox without subscription also doesn't include net-tools package (provides 'ifconfig'). Replacing 'ifconfig ethX mtu 9000' with 'ip link set ethX mtu 9000', or even doing away with increased MTU altogether, made no difference...

PS: Not sure if this is a bug or not but adding more than two interfaces to an active-backup bond interface, at least when using the linux bridge, changes the mode to round-robin!
 
there are never any newer or extra packages in pve-enterprise (it's the other way round, packages are moved to enterprise after more extensive testing using no-subscription). net-tools is not part of any proxmox repository, it's part of stock Debian (but it is not needed anymore, it has been replaced by iproute2 containing ip, ss, ...).

maybe your repository configuration was somehow messed up when you were testing without a subscription?
 
We added and removed the following before and after subscribing the box:
Code: 
nano /etc/apt/sources.list
  # PVE pve-no-subscription repository provided by proxmox.com, NOT recommended for production use
  deb http://download.proxmox.com/debian stretch pve-no-subscription

Either way, we now have a simple policy not to release Proxmox installations to customers, for production workloads, until the system has been entitled to the subscription repository and updated.
 
I'm running into the same issues with openvswitch (with the pve-no-subscription repository installed).

A simple bridge like:

Code: 
auto vmbr0
allow-ovs vmbr0
iface vmbr0 inet manual
  ovs_type OVSBridge

Would not work with openvswitch 2.7.0-2 from the no-subscription repository.

This is a fresh install.

Also what David mentioned, about it needing ifconfig - if you look at the code of /usr/share/openvswitch/scripts/ifupdown.sh, it internally uses ifconfig to bring interfaces up/down, regardless if ifconfig is deprecated, and your openvswitch-switch package depends on it.

I am not sure if I want to risk the subscription, considering that the no-sub version doesn't work.
 
Znuff said:
I'm running into the same issues with openvswitch (with the pve-no-subscription repository installed).

A simple bridge like:

Code: 
auto vmbr0
allow-ovs vmbr0
iface vmbr0 inet manual
  ovs_type OVSBridge

Would not work with openvswitch 2.7.0-2 from the no-subscription repository.

This is a fresh install.

Also what David mentioned, about it needing ifconfig - if you look at the code of /usr/share/openvswitch/scripts/ifupdown.sh, it internally uses ifconfig to bring interfaces up/down, regardless if ifconfig is deprecated, and your openvswitch-switch package depends on it.

I am not sure if I want to risk the subscription, considering that the no-sub version doesn't work.
Click to expand...

if you don't provide more info than 'does not work' it will be hard to fix your problem.. please provide 'pveversion -v' output, as well as a clear description of the actual problem, including any error messages or expected behaviour.
 
Original problem we experienced was that OVS wouldn't start, the network initialisation script would eventually time out after 5 minutes. The system didn't have out of band management so it was difficult to troubleshoot. We did bring the system in to our offices though... No errors were logged anywhere, but I'm pretty sure that there were ovs or openswitch files in /etc/init.d

It may very well have been due to Debian ovswitch package having been installed, as we may have run 'apt-get install xxx' before adding 'no-subscription' repo...


I just tested this by installing PVE 5.1 as guest and everything works perfectly...

Herewith my notes:
Installed PVE 5.1 as virtual guest (nested virtualisation currently disabled), then logged in after reboot and did:
Code: 
apt-get update; apt-get -y dist-upgrade; apt-get autoremove; apt-get autoclean;
echo -e "\n# PVE pve-no-subscription repository provided by proxmox.com, NOT recommended for production use\ndeb http://download.proxmox.com/debian stretch pve-no-subscription" >> /etc/apt/sources.list
apt-get update; apt-get -y dist-upgrade; apt-get autoremove; apt-get autoclean;

Code: 
root@test-pve:~# pveversion -v
proxmox-ve: 5.1-28 (running kernel: 4.13.4-1-pve)
pve-manager: 5.1-36 (running version: 5.1-36/131401db)
pve-kernel-4.13.4-1-pve: 4.13.4-26
pve-kernel-4.13.8-2-pve: 4.13.8-28
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-15
qemu-server: 5.0-17
pve-firmware: 2.0-3
libpve-common-perl: 5.0-20
libpve-guest-common-perl: 2.0-13
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-16
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.1-2
pve-container: 2.0-17
pve-firewall: 3.0-3
pve-ha-manager: 2.0-3
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.0-2
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.3-pve1~bpo9

/etc/network/interfaces before:
Code: 
auto lo
iface lo inet loopback

iface ens18 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.52
        netmask 255.255.255.0
        gateway 192.168.1.1
        bridge_ports ens18
        bridge_stp off
        bridge_fd 0

Installed OVS:
Code: 
apt-get install openvswitch-switch

/etc/network/interfaces afterwards:
Code: 
auto lo
iface lo inet loopback

allow-vmbr0 ens18
iface ens18 inet manual
        ovs_bridge vmbr0
        ovs_type OVSPort
        ovs_options tag=1 vlan_mode=native-untagged
        mtu 9000

auto vmbr0
allow-ovs vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports ens18 vlan1
        mtu 9000

allow-vmbr0 vlan1
iface vlan1 inet static
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=1
        ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
        address 192.168.1.52
        netmask 255.255.255.0
        gateway 192.168.1.1
        mtu 1500


Everything assembles as expected. Herewith confirmation from OVS:
Code: 
root@test-pve:~# ovs-vsctl show
28766dfd-e40d-4843-9eab-8a246fbd62a7
    Bridge "vmbr0"
        Port "ens18"
            tag: 1
            Interface "ens18"
        Port "vlan1"
            tag: 1
            Interface "vlan1"
                type: internal
        Port "vmbr0"
            Interface "vmbr0"
                type: internal
    ovs_version: "2.7.0"
 
Last edited:
David Herselman said:
allow-vmbr0 ens18 iface ens18 inet manual ovs_bridge vmbr0 ovs_type OVSPort ovs_options tag=1 vlan_mode=native-untagged mtu 9000
Click to expand...

You need a minimum of 2 ports to set up a bond.

Code: 
ovs-vsctl add-bond vmbr0 bond0 ens18
will tell you that it needs a minimum of 4 arguments (that is, 2 physical ports).
 
Hi Znuff,

Correct. I discovered my error when I posted the reply and subsequently edited it.

As reference, for others that may find it useful, herewith a working interfaces configuration file when running OVS with bond interfaces:
Code: 
auto lo
iface lo inet loopback

allow-vmbr0 bond0
iface bond0 inet manual
        ovs_bridge vmbr0
        ovs_type OVSBond
        ovs_bonds eth0 eth1
        pre-up ( ifconfig eth0 mtu 9000 && ifconfig eth1 mtu 9000 )
        ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast tag=1 vlan_mode=native-untagged
        mtu 9000

auto vmbr0
allow-ovs vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports bond0 vlan1
        mtu 9000

allow-vmbr0 vlan1
iface vlan1 inet static
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=1
        ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
        address 192.168.1.26
        netmask 255.255.255.0
        gateway 192.168.1.1
        mtu 1500

auto eth2
iface eth2 inet manual

auto eth3
iface eth3 inet manual


Simply replace ovs_options above with the following for active/backup:
Code: 
ovs_options bond_mode=active-backup tag=1 vlan_mode=native-untagged
 
@David Herselman

I don't see the the openv-switch package in pveversion -v.
This can have two reasons
1.) the package is not installed
2.) you have installed the Debian package what will not work with proxmox-ve.
 
Hi Wolfgang,

The 'pveversion -v' output from my test was prior to installing OVS. The purpose of my feedback was to essentially confirm that everything worked as expected and that my and most probably others problems were due to us installing OVS prior to adding the no-subscription repository.

Repeating the test in #16 above, with us running 'apt-get install openvswitch-switch', prior to adding the no-subscription repository, results in the same problem we experienced originally.

Symptoms:
  • Installing Debian openvswitch does not install net-tools. You can simply check whether or not 'ifconfig' works (it should if, it was the openvswitch package provided by Proxmox as they define it as a prerequisite and build the package)
  • Debian packages can be identified as:
    Code: 
    root@pve-test:~# dpkg -l | grep openvswitch
ii  openvswitch-common                   2.6.2~pre+git20161223-3        amd64        Open vSwitch common components
ii  openvswitch-switch                   2.6.2~pre+git20161223-3        amd64        Open vSwitch switch implementations

Herewith the output when installing the default Debian openvswitch packages:
Code: 
root@pve-test:~# apt-get install openvswitch-switch
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libatomic1 openvswitch-common uuid-runtime
Suggested packages:
  ethtool
The following NEW packages will be installed:
  libatomic1 openvswitch-common openvswitch-switch uuid-runtime


To subsequently fix the issue:
Code: 
dpkg --purge openvswitch-common openvswitch-switch;
echo -e "\n# PVE pve-no-subscription repository provided by proxmox.com, NOT recommended for production use\ndeb http://download.proxmox.com/debian stretch pve-no-subscription" >> /etc/apt/sources.list;
apt-get update; apt-get -y dist-upgrade; apt-get autoremove; apt-get autoclean;
apt-get install openvswitch-switch;

Herewith the output when installing the Proxmox openvswitch packages:
Code: 
The following additional packages will be installed:
  libatomic1 net-tools openvswitch-common
Suggested packages:
  ethtool openvswitch-datapath-module
The following NEW packages will be installed:
  libatomic1 net-tools openvswitch-common openvswitch-switch

NB: Note 'net-tools'...

And... everything works...


Herewith package information for the ones provided by Proxmox:
Code: 
root@pve-test:~# dpkg -l | grep openvswitch
ii  openvswitch-common                   2.7.0-2                        amd64        Open vSwitch common components
ii  openvswitch-switch                   2.7.0-2                        amd64        Open vSwitch switch implementations
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom