[SOLVED] PVE 6.3 + OVS / Linux bridge + MLX312B SR-IOV + LACP

[mhaluska]

Hey folks,

I'm fighting with network setup, not sure if this config can work, so would be nice if you can share info or your config regarding similar setup.

• PVE: 6.3-3
• System: HPE ML350 Gen9
• Network card: HPE 546SFP+ (MLX312B) - 2-port SFP+ (part number: 779793-B21)
• Bridging: using OVS, bonding doesn't work for me on this card with linux bridge
• Switch: Mikrotik CRS317 with enabled RSTP

Everything is running fine with OVS without activating SR-IOV virtual adapters using mlx4_core driver option. Activating virtual adapters cause no network communication, tested using main ports (ens1+ens1d1) and also virtual ports (ens1v0+ens1d1v0 with probe_vf=1). I don't know if I can use LACP together with virtual adapters.

Ports + bonding + bridge config:

allow-vmbr0 ens1
auto ens1
iface ens1 inet manual
        mtu 9000

allow-vmbr0 ens1d1
auto ens1d1
iface ens1d1 inet manual
        mtu 9000

auto bond0
iface bond0 inet manual
        ovs_bonds ens1 ens1d1
        ovs_type OVSBond
        ovs_bridge vmbr0
        ovs_mtu 9000
        ovs_options vlan_mode=native-untagged tag=1 bond_mode=balance-tcp other_config:lacp-time=fast lacp=active

auto vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports bond0 vlan2001 vlan2003 vlan3004 vlan3005
        ovs_mtu 9000

Switch config (L2MTU = 10000):

 1  R ;;; MCX312B
      name="bonding2" mtu=9000 mac-address=<REMOVED> arp=enabled arp-timeout=auto slaves=sfp-sfpplus9,sfp-sfpplus10 mode=802.3ad primary=none link-monitoring=mii arp-interval=100ms arp-ip-targets="" mii-interval=100ms down-delay=100ms up-delay=100ms
      lacp-rate=1sec transmit-hash-policy=layer-2-and-3 min-links=1

Mellanox driver option (also tested probe_vf=1):

options mlx4_core num_vfs=16 port_type_array=2,2 probe_vf=0

Mellanox firmware config:

Device #1:
----------

Device type:    ConnectX3Pro
Device:         0000:04:00.0

Configurations:                              Next Boot
         SRIOV_EN                            True(1)
         NUM_OF_VFS                          16
         PHY_TYPE_P1                         XFI(2)
         XFI_MODE_P1                         _10G(0)
         FORCE_MODE_P1                       False(0)
         PHY_TYPE_P2                         XFI(2)
         XFI_MODE_P2                         _10G(0)
         FORCE_MODE_P2                       False(0)
         LOG_BAR_SIZE                        5
         BOOT_OPTION_ROM_EN_P1               True(1)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               True(1)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)
         STEER_FORCE_VLAN                    False(0)

Thanks for any tip.

 

[Alwin]

Everything is running fine with OVS without activating SR-IOV virtual adapters using mlx4_core driver option. Activating virtual adapters cause no network communication, tested using main ports (ens1+ens1d1) and also virtual ports (ens1v0+ens1d1v0 with probe_vf=1). I don't know if I can use LACP together with virtual adapters.

Not possible, see here.
https://community.mellanox.com/s/article/howto-configure-lag-in-the-vm-over-sr-iov

 

[mhaluska]

Not possible, see here.
https://community.mellanox.com/s/article/howto-configure-lag-in-the-vm-over-sr-iov

Ok, so if I understand correctly. Based on link above, if I activate SR-IOV and VF, I cannot use bonding on eth2 and eth3, even if I use probe_vf=1 and bond VF on PVE, am I right?
Edit: just asking because I didn't use VF in VM yet, just curious that LACP wasn't working on PVE while no VF has been attached to VM.

 

[Alwin]

1. LACP will not work in such case, as more than 1 VM can be configured with bond.

2. You cannot configure on VM with bond and another VM with regular one port connectivity - It will not work (basic networking)

It is even more complicated. To think about, since VF are a subset of the NIC, bond modes that involve the switch (eg. LACP) will not work. The switch doesn't have any knowledge about the VF. Though the guide mentions the use of a port channel on the switch. I suppose they mean a passive channel (no LACPDUs). In any case, this seems very inflexible.

Why do you want to achieve with SR-IOV?

 

[mhaluska]

SR-IOV just for passing thru VF to two virtual routers (so without LACP), nothing else. LACP on PVE - lazy to calculate/configure ideal balance per port
But now I understand, I'll just use LACP without SR-IOV, easier way.
Thanks Alwin

 

[Alwin]

SR-IOV just for passing thru VF to two virtual routers (so without LACP)

Well, if use the VMs as routers, then you could employ RSTP, routing/switching costs or probably other protocols like ospf, rip, bgp.

 

[mhaluska]

Well, if use the VMs as routers, then you could employ RSTP, routing/switching costs or probably other protocols like ospf, rip, bgp.

For simplicity I'll just use LACP and ignore VF (SR-IOV), don't want to complicate my simple homelab
Thanks for hints Alwin.