OVS bonding issue with interfaces not starting

[jtracy]

I am trying to get bonding to work under OVS and for the life of me can get the sub-interfaces to come up.

I have a small device with 6 ethernet ports on it that Im trying to get working correctly for a virutalized firewall. I orginally setup all of the networking in the web interface but have made changes in the CLI since, but nothing seems to work for bonding. Currently I have the firewall setup and running with the following network layout.

enp1s0  -  WAN connection
enp2s0  -  Switch 1
enp3s0  -  NAS
enp4s0
enp5s0  -  Switch2
enp6s0  -  Switch2

vlan3 -  Mangement Network (has the ip proxmox runs on)

bond0
  - enp5s0 enp6s0  

vmbr0  
  - enp2s0 enp3s0 enp4s0 vlan3 bond0

vmbr666
  - enp1s0

Here is the /etc/network/interfaces configuration:

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto enp1s0
iface enp1s0 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr666

iface wlx002243735475 inet manual

auto enp2s0
iface enp2s0 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr0

auto enp3s0
iface enp3s0 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr0

auto enp4s0
iface enp4s0 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr0

iface enp5s0 inet manual

iface enp6s0 inet manual

auto vlan3
iface vlan3 inet static
        address XX.XX.XX.XX/24
        gateway XX.XX.XX.XX
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=3

auto bond0
iface bond0 inet manual
        ovs_bonds enp5s0 enp6s0
        ovs_type OVSBond
        ovs_bridge vmbr0
        ovs_options bond_mode=balance-tcp other_config:lacp-time=fast lacp=active

auto vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports enp2s0 enp3s0 enp4s0 vlan3 bond0

auto vmbr666
iface vmbr666 inet manual
        ovs_type OVSBridge
        ovs_ports enp1s0
#WAN/Internet

I have tried different configurations but I always end up with the bond reporting this:

# ovs-appctl bond/show
---- bond0 ----
bond_mode: balance-tcp
bond may use recirculation: yes, Recirc-ID : 5
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
next rebalance: 2629 ms
lacp_status: configured
lacp_fallback_ab: false
active slave mac: 00:00:00:00:00:00(none)

slave enp5s0: disabled
  may_enable: false

slave enp6s0: disabled
  may_enable: false

Does anyone have an idea what might be going on here?

Spoiler: pveversion

pve-manager/6.2-11/22fb4983 (running kernel: 5.4.55-1-pve)
root@pve02:~# pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.55-1-pve)
pve-manager: 6.2-11 (running version: 6.2-11/22fb4983)
pve-kernel-5.4: 6.2-5
pve-kernel-helper: 6.2-5
pve-kernel-5.4.55-1-pve: 5.4.55-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve2
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-5
libpve-guest-common-perl: 3.1-2
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-6
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-10
pve-cluster: 6.1-8
pve-container: 3.1-12
pve-docs: 6.2-5
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-2
pve-ha-manager: 3.0-9
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-12
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-11
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve1

 

[RokaKen]

You seem to be missing critical keywords for OVS, e.g., auto vmbr0 should be allow-ovs vmbr0 , auto bond0 should be allow-vmbr0 bond0 , auto vlan3 should be allow-vmbr0 vlan3... see the wiki for Open vSwitch .

I see you've also installed ifupdown2, so perhaps @spirit can confirm.

 

[jtracy]

So that config was generated with the proxmox web gui. I will try and change them later and see if it fixes my issues. Thanks.

 

[spirit]

So that config was generated with the proxmox web gui. I will try and change them later and see if it fixes my issues. Thanks.

do you have installed ifupdown2 (for online reload), if yes,, the syntax is with "auto ..." .

If not, it should be "allow ..."



if you have installed ifupdown2, what is the current version (dpkg -l|grep ifupdown2) && can you send the result of "ifreload -a -d".

 

[spirit]

try to add "auto ..." to enp5s0 && enp6s0

auto enp5s0
iface enp5s0 inet manual

auto enp6s0
iface enp6s0 inet manual

Seem that ifupdown2 is used here, but without auto on physical interfaces, they are in the bond, but not up.

 

[RokaKen]

do you have installed ifupdown2 (for online reload), if yes,, the syntax is with "auto ..." .

If not, it should be "allow ..."

@spirit -- my apologies for giving incorrect advice. My account doesn't seem to work on the wiki. Could you add a note WRT ifupdown2 to the wiki when you have time?

 

[spirit]

@spirit -- my apologies for giving incorrect advice. My account doesn't seem to work on the wiki. Could you add a note WRT ifupdown2 to the wiki when you have time?

yes, I'll try to update official doc soon.