[SOLVED] No bond/ovs after 6 to 7 upgrade

[dashie]

Upgraded my latest 6.x to 7 and the networking part that was using a bond and openvswitch doesn't get up on reboot.

Networking on a direct interface does work, but not my main access. My setup is a bond0 on four interfaces eno{1,2,3,4} and tagged vlans on vmbr0.

Config is:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# Bond between the four interfaces
allow-vmbr0 eno1
iface eno1 inet manual

allow-vmbr0 eno2
iface eno2 inet manual

allow-vmbr0 eno3
iface eno3 inet manual

allow-vmbr0 eno4
iface eno4 inet manual

# Bond LACP
allow-vmbr0 bond0
iface bond0 inet manual
        ovs_bridge vmbr0
        ovs_type OVSBond
        ovs_bonds eno1 eno2 eno3 eno4
        ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast

# bridge for our bond and vlan virtual interfaces (the VMs will attach to this bridge too)
allow-ovs vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports bond0 vlan110 vlan140

allow-vmbr0 vlan110
iface vlan110 inet static
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=110

allow-vmbr0 vlan140
iface vlan140 inet static
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=140
        address 192.168.40.229
        netmask 255.255.255.0
        gateway 192.168.40.1

auto enp4s0f0
iface enp4s0f0 inet static
        address 10.42.42.2
        netmask 255.255.255.0
        mtu 9000

openvswitch/ifup etc. seems installed:

ii  ifupdown                             0.8.36                         amd64        high level tools to configure network interfaces
ii  openvswitch-common                   2.15.0+ds1-2                   amd64        Open vSwitch common components
ii  openvswitch-switch                   2.15.0+ds1-2                   amd64        Open vSwitch switch implementations

but no bond0/vmbr/vlans etc.:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether b8:ac:6f:13:3e:d7 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f0
    inet6 fe80::baac:6fff:fe13:3ed7/64 scope link
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether b8:ac:6f:13:3e:d9 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f1
    inet6 fe80::baac:6fff:fe13:3ed9/64 scope link
       valid_lft forever preferred_lft forever
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether b8:ac:6f:13:3e:db brd ff:ff:ff:ff:ff:ff
    altname enp2s0f0
    inet6 fe80::baac:6fff:fe13:3edb/64 scope link
       valid_lft forever preferred_lft forever
5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether b8:ac:6f:13:3e:dd brd ff:ff:ff:ff:ff:ff
    altname enp2s0f1
    inet6 fe80::baac:6fff:fe13:3edd/64 scope link
       valid_lft forever preferred_lft forever
6: enp4s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether a0:36:9f:13:96:94 brd ff:ff:ff:ff:ff:ff
    inet 10.42.42.2/24 brd 10.42.42.255 scope global enp4s0f0
       valid_lft forever preferred_lft forever
    inet6 fe80::a236:9fff:fe13:9694/64 scope link
       valid_lft forever preferred_lft forever
7: enp4s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a0:36:9f:13:96:96 brd ff:ff:ff:ff:ff:ff

any ideas what I can do to find the issue ?

Tried rebooting multiple times, ifup enoX or bond0 etc. but nothing wants to get up.

 

[dashie]

so hm I installed ifupdown2 which seemed to have caused a network config rewrite, network still doesn't work until I do a `systemctl restart networking` from the console.

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

allow-vmbr0 eno1

auto eno1
iface eno1 inet manual

allow-vmbr0 eno2

auto eno2
iface eno2 inet manual

allow-vmbr0 eno3

auto eno3
iface eno3 inet manual

allow-vmbr0 eno4

auto eno4
iface eno4 inet manual

auto enp4s0f0
iface enp4s0f0 inet static
        address 10.42.42.2/24
        mtu 9000
#10g interface for storage access

iface enp4s0f1 inet manual

auto vlan110
iface vlan110 inet static
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=110

auto vlan140
iface vlan140 inet static
        address 192.168.40.229/24
        gateway 192.168.40.1
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=140

auto bond0
iface bond0 inet manual
        ovs_bonds eno1 eno2 eno3 eno4
        ovs_type OVSBond
        ovs_bridge vmbr0
        ovs_options other_config:lacp-time=fast lacp=active bond_mode=balance-tcp
#bond between the 4 gig interfaces

auto vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports bond0 vlan110 vlan140

 

[spirit]

with ifupdown2, you should remove the "allow-*". (The upgrade script should have remove them, maybe it's a bug)

 

[dashie]

just tried that and removed the 4 allow-* occurences, but after reboot I still have to issue a "systemctl restart networking" to have everything getting up

 

[spirit]

it's possible to enable debug log at boot in /etc/default/networking

# Change the below to yes if you want verbose logging to be enabled
VERBOSE="tes"
 
# Change the below to yes if you want debug logging to be enabled
DEBUG="yes"

(I think it's logging in /var/log/daemon.log)

Maybe it could be a race when interfaces are starting or something specific.

 

[dashie]

thanks, I will try later today and report back.

 

[dashie]

Didn't found anything in logs with verbose and debug, but looking at journalctl -u networking then systemctl status networking, I saw the unit never triggered and was disabled.

For some reasons after the update the networking unit got disabled, I enabled it (systemctl enable networking) and now network starts at boot.

 

[Astraea]

After I upgraded my cluster from 6 to 7 I did not have any network connectivity. what ended up fixing the issues was logging into each node and running the following command

systemctl enable networking; reboot

After that, the node rebooted and came back online and everything was working again. Not sure why the networking stack got disabled during the update. This is on a 5 node cluster with multiple NICs with and without bonds that was using ifupdown2 before the upgrade.

 

[mfed]

Just wanted to share my experience.
I also had openvswitch configuration with a bond and vlans, and it also stopped working after the upgrade. Did not know about the 'systemctl restart networking' trick, but manually creating vlan interfaces and assigning ip addresses with 'ip' command worked. Ended up re-creating my configuration using Linux Bridge rather than openvswitch, and it just worked. Will try the 'systemctl enable networking' when I upgrade another node of the cluster...

 

[mfed]

I upgraded the other two nodes of the cluster, and there were no issues. But what I did differently is that I installed ifupdown2 before the upgrade. So I compared my /etc/network/interfaces, and found that the node that did not work had 'allow-vmbr0 <interface>' commands, but the working upgraded nodes had 'auto <interface>' commands at that place. After I replaced all 'allow-vmbr0' with 'auto', the network configuration started to work. I guess I should have read spirit's response more attentively, and possibly the bug is that without ifupdown2 installed the allow-* commands are not replaced (but still not working)...

 

[Astraea]

When I did the upgrade from 6 to 7 I had already been using ifupdown2 since I installed the 6 cluster and the interfaces did not work after the upgrade until I executed the command I shared previously. I wonder what is causing the network issues when upgrding. I will be doing a full reinstall of 7 hopefully shortly as I am going to be adding 2 more nodes and will be changing out the network design at the same time.

 

[spirit]

I upgraded the other two nodes of the cluster, and there were no issues. But what I did differently is that I installed ifupdown2 before the upgrade. So I compared my /etc/network/interfaces, and found that the node that did not work had 'allow-vmbr0 <interface>' commands, but the working upgraded nodes had 'auto <interface>' commands at that place. After I replaced all 'allow-vmbr0' with 'auto', the network configuration started to work. I guess I should have read spirit's response more attentively, and possibly the bug is that without ifupdown2 installed the allow-* commands are not replaced (but still not working)...

when ifupdown2 is installed, it's doing the changes of /etc/network/interfaces to /etc/network/interfaces.new, and it's replaced on next boot.
Maybe they are a bug when the distro is upgrade at the same time.

 

[jorsel]

I had the same issue after upgrading from 6 to 7.
What worked for me was changing the order in /etc/network/interfaces.
I have put the eno1 and eno2 lines below the rest. It seems like order does matter after all.