bridge on bond doesn't come up after reboot (using SR-IOV VF's)

K

kayson

New Member
Feb 13, 2024
13
1
3
I have PVE 8.3 installed on a machine with a dual port SFP+ NIC (X710-DA2). I've set up SR-IOV VF's using these udev rules:
Code: 
ACTION=="add", SUBSYSTEM=="net", ENV{INTERFACE}=="enp1s0f0np0", ATTR{device/sriov_numvfs}="3"
ACTION=="add", SUBSYSTEM=="net", ENV{INTERFACE}=="enp1s0f1np1", ATTR{device/sriov_numvfs}="3"

Using the PVE GUI, I've set up a bridge on a bond on two of the VF's resulting in the following interfaces file:
Code: 
auto lo
iface lo inet loopback

auto enp1s0f0v0
iface enp1s0f0v0 inet manual

auto enp1s0f1v0
iface enp1s0f1v0 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves enp1s0f0v0 enp1s0f1v0
        bond-miimon 100
        bond-mode active-backup
        bond-primary enp1s0f0v0

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        bridge-stp off
        bridge-fd 0

auto vmbr0.10
iface vmbr0.10 inet static
        address 10.7.0.20/24
        gateway 10.7.0.1

source /etc/network/interfaces.d/*

After applying the changes, the network works as expected. After a reboot, though, none of the interfaces come up:
Code: 
> ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 68:05:ca:9b:d6:84 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether ba:4e:ac:fb:fe:ef brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
3: enp1s0f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 68:05:ca:9b:d6:85 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether ba:4e:ac:fb:fe:ef brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
4: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 8c:16:45:92:88:9b brd ff:ff:ff:ff:ff:ff
    altname enp0s31f6
5: enp1s0f0v0: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN mode DEFAULT group default qlen 1000
    link/ether ba:4e:ac:fb:fe:ef brd ff:ff:ff:ff:ff:ff
6: enp1s0f1v0: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN mode DEFAULT group default qlen 1000
    link/ether ba:4e:ac:fb:fe:ef brd ff:ff:ff:ff:ff:ff
7: enp1s0f0v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 86:62:db:16:bf:ef brd ff:ff:ff:ff:ff:ff
8: enp1s0f1v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether e6:93:a7:a5:17:b9 brd ff:ff:ff:ff:ff:ff
9: enp1s0f1v2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether de:7d:9a:5c:a5:a3 brd ff:ff:ff:ff:ff:ff
10: enp1s0f0v2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 96:cd:ab:44:2c:57 brd ff:ff:ff:ff:ff:ff
11: bond0: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue master vmbr0 state DOWN mode DEFAULT group default qlen 1000
    link/ether ba:4e:ac:fb:fe:ef brd ff:ff:ff:ff:ff:ff
12: vmbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether ba:4e:ac:fb:fe:ef brd ff:ff:ff:ff:ff:ff
13: vmbr0.10@vmbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether ba:4e:ac:fb:fe:ef brd ff:ff:ff:ff:ff:ff

and I see the following in dmesg:
Code: 
[    6.891996] bond0: (slave enp1s0f0v0): Enslaving as a backup interface with a down link
[    6.895778] bond0: (slave enp1s0f1v0): Enslaving as a backup interface with a down link
[    6.911862] vmbr0: port 1(bond0) entered blocking state
[    6.911866] vmbr0: port 1(bond0) entered disabled state
[    6.911877] bond0: entered allmulticast mode

Strangely, the ifupdown2 debug logs show no errors and even show that it's running the commands to bring the VF's up:
Code: 
2025-01-10 12:32:50,576: MainThread: ifupdown: scheduler.py:105:run_iface_op(): debug: bond0: pre-up : running module bond
2025-01-10 12:32:50,576: MainThread: ifupdown.bond: bond.py:697:get_ifla_bond_attr_from_user_config(): info: bond0: set bond-mode active-backup
2025-01-10 12:32:50,576: MainThread: ifupdown.bond: bond.py:697:get_ifla_bond_attr_from_user_config(): info: bond0: set bond-miimon 100
2025-01-10 12:32:50,577: MainThread: ifupdown.bond: bond.py:697:get_ifla_bond_attr_from_user_config(): info: bond0: set bond-primary enp1s0f0v0
2025-01-10 12:32:50,577: MainThread: ifupdown2.NetlinkListenerWithCache: nlcache.py:3130:link_add_bond_with_info_data(): info: bond0: netlink: ip link add dev bond0 type bond (with attributes)
2025-01-10 12:32:50,577: MainThread: ifupdown2.NetlinkListenerWithCache: nlcache.py:3134:link_add_bond_with_info_data(): debug: attributes: OrderedDict([(1, 1), (3, 100), (11, 5)])
2025-01-10 12:32:50,577: MainThread: ifupdown2.NetlinkListenerWithCache: nlcache.py:2744:link_set_master(): info: enp1s0f0v0: netlink: ip link set dev enp1s0f0v0 master bond0
2025-01-10 12:32:50,579: MainThread: ifupdown2.NetlinkListenerWithCache: nlcache.py:2610:link_up_force(): info: enp1s0f0v0: netlink: ip link set dev enp1s0f0v0 up
2025-01-10 12:32:50,581: MainThread: ifupdown2.NetlinkListenerWithCache: nlcache.py:2744:link_set_master(): info: enp1s0f1v0: netlink: ip link set dev enp1s0f1v0 master bond0
2025-01-10 12:32:50,585: MainThread: ifupdown2.NetlinkListenerWithCache: nlcache.py:2610:link_up_force(): info: enp1s0f1v0: netlink: ip link set dev enp1s0f1v0 up
(full logs are attached)

Running ifreload -a or ifup enp1s0f0v0 enp1s0f1v0 succeed but don't actually bring the interfaces up. If I do an ifdown enp1s0f0v0 enp1s0f1v0, first however, then either command successfully brings the interfaces up, and once everything is up, the network behaves as expected.

As an experiment, I tried some other network configurations. For some reason, removing the bridge and statically assigning an IP to the bond fixes the problem, and all interfaces come up after reboot:
Code: 
auto lo
iface lo inet loopback

auto enp1s0f0v0
iface enp1s0f0v0 inet manual

auto enp1s0f1v0
iface enp1s0f1v0 inet manual

auto bond0
iface bond0 inet static
        bond-slaves enp1s0f0v0 enp1s0f1v0
        bond-miimon 100
        bond-mode active-backup
        bond-primary enp1s0f0v0
       address 10.1.0.20/24
       gateway 10.1.0.1

source /etc/network/interfaces.d/*
(the change in subnet is needed because of the switch VLAN config)

Code: 
> ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 68:05:ca:9b:d6:84 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 42:9d:5d:b1:61:e1 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
3: enp1s0f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 68:05:ca:9b:d6:85 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 42:9d:5d:b1:61:e1 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
4: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 8c:16:45:92:88:9b brd ff:ff:ff:ff:ff:ff
    altname enp0s31f6
5: enp1s0f1v0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 42:9d:5d:b1:61:e1 brd ff:ff:ff:ff:ff:ff
6: enp1s0f0v0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 42:9d:5d:b1:61:e1 brd ff:ff:ff:ff:ff:ff
7: enp1s0f0v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether c6:12:4a:95:c6:54 brd ff:ff:ff:ff:ff:ff
8: enp1s0f1v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether e6:8a:d1:a7:58:de brd ff:ff:ff:ff:ff:ff
9: enp1s0f0v2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether ce:ef:ad:81:8a:28 brd ff:ff:ff:ff:ff:ff
10: enp1s0f1v2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 72:f5:80:f1:b6:35 brd ff:ff:ff:ff:ff:ff
11: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 42:9d:5d:b1:61:e1 brd ff:ff:ff:ff:ff:ff

dmesg:
Code: 
[    6.720277] bond0: (slave enp1s0f0v0): Enslaving as a backup interface with a down link
[    6.724151] bond0: (slave enp1s0f1v0): Enslaving as a backup interface with a down link
[    6.805535] iavf 0000:02:02.0 enp1s0f0v0: NIC Link is Up Speed is 10 Gbps Full Duplex
[    6.809171] iavf 0000:02:0a.0 enp1s0f1v0: NIC Link is Up Speed is 10 Gbps Full Duplex
[    6.831866] bond0: (slave enp1s0f0v0): link status definitely up, 10000 Mbps full duplex
[    6.831876] bond0: (slave enp1s0f1v0): link status definitely up, 10000 Mbps full duplex
[    6.831878] bond0: (slave enp1s0f0v0): making interface the new active one
[    6.831893] bond0: active interface up!

ifupdown2 debug logs for this case are also attached. I'm not seeing any notable differences.

Initially I suspected some kind of race condition with the VF's getting created and the systemd unit that calls ifupdown2 (networking.service), but adding sleep calls doesn't make a difference. This is not very surprising because there is an ordering dependency on another systemd unit (ifupdown2-pre.service) which calls udevadm settle first.

Finally, if I don't use VF's, and use the interfaces proper (i.e. PF's) for the bond, it works fine.

Does anyone have any ideas what's going on? Is there a more "correct" way to set up a network with VF's at boot?

Thanks!
 

Attachments

You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom