proxmox 7 latest dist-upgrade causes emulex 10Gb nic no carrier

[Guy]

Hello all,

I've upgraded to Proxmox 7, and use openvswitch, and ifupdown2 as well as several bonded interfaces. All was working just fine until yesterday when I did a "apt-get dist-upgrade". Since then the interface is reporting "no carrier" and not working

I've another 2 identical proxmox system in the same blade centre, 1 I've also updated and it also fails straight away, the other I've left alone and it continues to work. All have similar network configurations, just changes to IP Address.

i'm stumped here, to be it looks like a package upgrade has broken something, maybe the driver or something in openvswitch.

Can anyone give any suggestions on further debugging I can do or indeed provide a solution?

Cheers,
--Guy

4: enp2s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 10:1f:74:32:8f:a0 brd ff:ff:ff:ff:ff:ff
5: enp2s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 10:1f:74:32:8f:a4 brd ff:ff:ff:ff:ff:ff
6: enp2s0f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 10:1f:74:32:8f:a1 brd ff:ff:ff:ff:ff:ff
7: enp2s0f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 10:1f:74:32:8f:a5 brd ff:ff:ff:ff:ff:ff

~# modinfo be2net
filename:       /lib/modules/5.11.22-3-pve/kernel/drivers/net/ethernet/emulex/benet/be2net.ko
license:        GPL
author:         Emulex Corporation
description:    Emulex OneConnect NIC Driver
srcversion:     2E58CB3BB7D030C3EBC9154
alias:          pci:v000010DFd00000728sv*sd*bc*sc*i*
alias:          pci:v000010DFd00000720sv*sd*bc*sc*i*
alias:          pci:v000010DFd0000E228sv*sd*bc*sc*i*
alias:          pci:v000010DFd0000E220sv*sd*bc*sc*i*
alias:          pci:v000019A2d00000710sv*sd*bc*sc*i*
alias:          pci:v000019A2d00000221sv*sd*bc*sc*i*
alias:          pci:v000019A2d00000700sv*sd*bc*sc*i*
alias:          pci:v000019A2d00000211sv*sd*bc*sc*i*
depends:       
retpoline:      Y
intree:         Y
name:           be2net
vermagic:       5.11.22-3-pve SMP mod_unload modversions
parm:           num_vfs:Number of PCI VFs to initialize (uint)
parm:           rx_frag_size:Size of a fragment that holds rcvd data. (ushort)

ovs-vsctl show
1e01a3b1-9ccf-4cc2-9247-007ec255d4bf
    Bridge vmbr1
        Port vmbr1
            Interface vmbr1
                type: internal
        Port storage
            Interface storage
                type: internal
    Bridge vmbr0
        Port vmbr0
            Interface vmbr0
                type: internal
        Port vlan1230
            tag: 1230
            Interface vlan1230
                type: internal
    ovs_version: "2.15.0"

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto enp2s0f0
allow-vmbr0 enp2s0f0
iface enp2s0f0 inet manual
  ovs_bridge vmbr0

auto enp2s0f1
allow-vmbr0 enp2s0f1
iface enp2s0f1 inet manual
  ovs_bridge vmbr0

auto enp2s0f2
allow-vmbr1 enp2s0f2
iface enp2s0f2 inet manual
  ovs_bridge vmbr1
  ovs_mtu 9000

auto enp2s0f3
allow-vmbr1 enp2s0f3
iface enp2s0f3 inet manual
  ovs_bridge vmbr1
  ovs_mtu 9000

iface enp2s0f4 inet manual

iface enp2s0f5 inet manual

iface enp2s0f6 inet manual

iface enp2s0f7 inet manual

auto enp10s4f0
iface enp10s4f0 inet static
    address 10.1.2.6/24
#Ring 0v2

auto enp10s4f1
iface enp10s4f1 inet static
    address 10.1.3.6/24
#Ring 1v2

auto vmbr0
allow-ovs vmbr0
iface vmbr0 inet manual
    ovs_type OVSBridge
    ovs_ports enp2s0f0 vlan1230
    ovs_mtu 1500

auto vmbr1
allow-ovs vmbr1
iface vmbr1 inet manual
    ovs_type OVSBridge
    ovs_ports enp2s0f2 storage
    ovs_mtu 9000

auto vlan1230
allow-vmbr0 vlan1230
iface vlan1230 inet static
    address 192.168.230.210/24
    gateway 192.168.230.254
    ovs_type OVSIntPort
    ovs_bridge vmbr0
    ovs_options tag=1230
    ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
#ServerNet

auto storage
allow-vmbr1 storage
iface storage inet static
    address 10.1.1.210/24
    ovs_type OVSIntPort
    ovs_bridge vmbr1
    ovs_mtu 9000
#Storage

# auto bond0
# allow-vmbr0 bond0
# iface bond0 inet manual
#     ovs_bonds enp2s0f0 enp2s0f1
#     ovs_type OVSBond
#     ovs_bridge vmbr0
#     ovs_options bond_mode=balance-slb lacp=active vlan_mode=native-untagged
#     pre-up ip link set enp2s0f0 up && ip link set enp2s0f1 up
#
# auto bond1
# allow-vmbr1 bond1
# iface bond1 inet manual
#     ovs_bonds enp2s0f2 enp2s0f3
#     ovs_type OVSBond
#     ovs_bridge vmbr1
#     ovs_mtu 9000
#     ovs_options bond_mode=balance-slb lacp=active vlan_mode=native-untagged
#     pre-up ip link set enp2s0f2 up && ip link set enp2s0f3 up

 

[Guy]

Never mind.. looks like the upstream switch decided to block the ports and the failover switch had hung in a strange way which meant it thought it was there. Anyway a reboot of the secondary switch seems to have restored my worlds.