[SOLVED] Upgrade HP Z840 6.4 -> 7.0-13 networking failure

G

GumShoeNoir

Member
Nov 7, 2021
9
1
8
I upgraded two Proxmox 6.4 servers to version 7 yesterday. One went fine. The other on an HP Z840 has some trouble.

I tried commenting out auto vmbr0 and added mac hwaddress b1:d7:33:35:94 found these suggestions in my research. Neither helped.

Any ideas?

From syslog

Code: 
Nov  7 10:13:01 pve2 kernel: [19787.157091] igb 0000:05:00.0 enp5s0: igb: enp5s0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX
Nov  7 10:13:01 pve2 kernel: [19787.157392] igb 0000:05:00.0 enp5s0: Link Speed was downgraded by SmartSpeed
Nov  7 10:13:01 pve2 kernel: [19787.157727] vmbr0: port 1(enp5s0) entered blocking state
Nov  7 10:13:01 pve2 kernel: [19787.158725] vmbr0: port 1(enp5s0) entered forwarding state
Nov  7 10:13:02 pve2 kernel: [19787.828701] igb 0000:05:00.0 enp5s0: igb: enp5s0 NIC Link is Down
Nov  7 10:13:02 pve2 kernel: [19788.180619] vmbr0: port 1(enp5s0) entered disabled state
[...]
Nov  7 10:13:33 pve2 kernel: [19819.632961] igb 0000:05:00.0 enp5s0: igb: enp5s0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX
Nov  7 10:13:33 pve2 kernel: [19819.633294] igb 0000:05:00.0 enp5s0: Link Speed was downgraded by SmartSpeed
Nov  7 10:13:33 pve2 kernel: [19819.633652] vmbr0: port 1(enp5s0) entered blocking state
Nov  7 10:13:33 pve2 kernel: [19819.634826] vmbr0: port 1(enp5s0) entered forwarding state
Nov  7 10:13:34 pve2 kernel: [19820.376587] igb 0000:05:00.0 enp5s0: igb: enp5s0 NIC Link is Down
Nov  7 10:13:34 pve2 kernel: [19820.660535] vmbr0: port 1(enp5s0) entered disabled state
[...]
Nov  7 10:14:00 pve2 systemd[1]: Starting Proxmox VE replication runner...
Nov  7 10:14:01 pve2 systemd[1]: pvesr.service: Succeeded.
Nov  7 10:14:01 pve2 systemd[1]: Finished Proxmox VE replication runner.
Nov  7 10:14:08 pve2 kernel: [19854.276835] igb 0000:05:00.0 enp5s0: igb: enp5s0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX
Nov  7 10:14:08 pve2 kernel: [19854.277185] igb 0000:05:00.0 enp5s0: Link Speed was downgraded by SmartSpeed
Nov  7 10:14:08 pve2 kernel: [19854.277589] vmbr0: port 1(enp5s0) entered blocking state
Nov  7 10:14:08 pve2 kernel: [19854.278882] vmbr0: port 1(enp5s0) entered forwarding state
Nov  7 10:14:09 pve2 kernel: [19855.020470] igb 0000:05:00.0 enp5s0: igb: enp5s0 NIC Link is Down
Nov  7 10:14:09 pve2 pvestatd[2433]: storage 'debbie_cifs' is not online
Nov  7 10:14:09 pve2 kernel: [19855.284379] vmbr0: port 1(enp5s0) entered disabled state
[...]
Nov  7 10:15:00 pve2 systemd[1]: Starting Proxmox VE replication runner...
Nov  7 10:15:01 pve2 systemd[1]: pvesr.service: Succeeded.
Nov  7 10:15:01 pve2 systemd[1]: Finished Proxmox VE replication runner.
Nov  7 10:15:09 pve2 pvestatd[2433]: storage 'debbie_cifs' is not online
[...]
Nov  7 10:16:00 pve2 systemd[1]: Starting Proxmox VE replication runner...
Nov  7 10:16:01 pve2 systemd[1]: pvesr.service: Succeeded.
Nov  7 10:16:01 pve2 systemd[1]: Finished Proxmox VE replication runner.
Nov  7 10:16:09 pve2 pvestatd[2433]: storage 'debbie_cifs' is not online

/etc/network/interfaces

Code: 
auto lo
iface lo inet loopback

# auto enp5s0
iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.1.192/24
    # hwaddress b1:d7:33:35:94
    gateway 192.168.1.1
    bridge-ports enp5s0
    bridge-stp off
    bridge-fd 0

iface eno1 inet manual


# lshw -class network

Code: 
  *-network DISABLED
       description: Ethernet interface
       product: Ethernet Connection (2) I218-LM
       vendor: Intel Corporation
       physical id: 19
       bus info: pci@0000:00:19.0
       logical name: eno1
       version: 05
       serial: ec:b1:d7:33:35:93
       capacity: 1Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=e1000e driverversion=5.11.22-5-pve firmware=0.1-4 latency=0 link=no multicast=yes port=twisted pair
       resources: irq:39 memory:d0400000-d041ffff memory:d043b000-d043bfff ioport:6040(size=32)
  *-network
       description: Ethernet interface
       product: I210 Gigabit Network Connection
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:05:00.0
       logical name: enp5s0
       version: 03
       serial: ec:b1:d7:33:35:94
       capacity: 1Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=igb driverversion=5.11.22-5-pve firmware=3.25, 0x800005fa latency=0 link=no multicast=yes port=twisted pair
       resources: irq:16 memory:d0100000-d017ffff ioport:2000(size=32) memory:d0180000-d0183fff
  *-network
       description: Ethernet interface
       physical id: 1
       logical name: vmbr0
       serial: ec:b1:d7:33:35:94
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A ip=192.168.1.192 link=no multicast=yes
 
Last edited:
Try updating the firmware of the NIC and the BIOS to the latest version.
 
GumShoeNoir said:
I upgraded two Proxmox 6.4 servers to version 7 yesterday. One went fine. The other on an HP Z840 has some trouble.
Click to expand...

Thank you @mira

Just as I was about to swap in a Win 10 disk to attempt firmware upgrade, magically appearing on my front porch this afternoon, a "1.25G Gigabit Ethernet Converged Network Card, with Intel 82576 Chip, Dual RJ45 Ports, PCI Express 2.1 X1" $45.88 delivered.

I installed the new card swapping device names in /etc/network/interfaces and the server came up with network connectivity and both VMs too. With network connectivity restored I installed some tools that I missed while disconnected; ethtool,...

The BIOS firmware was/is current on the HP Z840.

Code: 
# dmidecode 3.3
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.
105 structures occupying 5619 bytes.
Table at 0xAB35B000.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
        Vendor: Hewlett-Packard
        Version: M60 v02.57
        Release Date: 05/21/2021
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 16 MB
        Characteristics:
                PCI is supported
                PNP is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                EDD is supported
                Print screen service is supported (int 5h)
                8042 keyboard services are supported (int 9h)
                Serial services are supported (int 14h)
                Printer services are supported (int 17h)
                ACPI is supported
                USB legacy is supported
                BIOS boot specification is supported
                Function key-initiated network boot is supported
                Targeted content distribution is supported
                UEFI is supported
        BIOS Revision: 2.57

Code: 
root@pve2:~# ethtool -i enp5s0
driver: igb
version: 5.11.22-5-pve
firmware-version: 3.25, 0x800005fa
expansion-rom-version:
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@pve2:~#

HP Customer Support - Software and Driver Downloads referring to win 10 has a firmware 3.30 Rev.A dated March 4, 2021 filename sp112877.exe

I'm inclined to leave it alone and move on to other projects, perhaps some update in the future will fix the problem with the i210 firmware.

Well guess I'll take a stab at installing that .exe with a FreeDOS USB stick. FreeDOS says the sp112877.exe program can't be run in DOS mode. I tried unsuccessfully booting a Win10 disk. It occurs to me that when I updated the BIOS a couple months ago I used FreeDOS, I wonder why FreeDOS didn't work for the ethernet firmware update...

Although I've solved my connectivity problem by throwing additional hardware at it, the issue of this post is not resolved hence I'm not marking it solved. It seems like a bug to me if a working Proxmox 6.4 upgraded to version 7 breaks it's networking.

Maybe I'll try to contact the art of server, he has been helpful with several HP Z840 firmware explanations.

Would any ProxMox folks want me to enter a bug at https://bugzilla.proxmox.com/ ?
 
Kernel upgrade from 5.4 to 5.11 means that there were changes to the driver, and support for older firmware versions might be worse now.
If the firmware is provided by HP, who knows what changes they apply to the official Intel firmware. So hard to say what the issue is.
But I'd suggest upgrading to the latest firmware for now and see if it fixes the issue.
 
mira said:
Kernel upgrade from 5.4 to 5.11 means that there were changes to the driver, and support for older firmware versions might be worse now.
If the firmware is provided by HP, who knows what changes they apply to the official Intel firmware. So hard to say what the issue is.
But I'd suggest upgrading to the latest firmware for now and see if it fixes the issue.
Click to expand...

@mira unSuccess

This morning I was able to bring up a Windows 10 HD on the HP Z840. I unsuccessfully ran the sp11827.exe firmware update. Here's a screen capture.

I tried twice, removing the 82576 I installed yesterday for the 2nd try, with same results minus that new card.

I'm assuming the version shown in the screen capture is the firmware version discovered in the current state. ??

FWIW although the firmware update shows March 2021 the updater shows Copyright 2013 - 2020

WAG; Maybe a motherboard revision issue?

I'm willing to try other suggestions if it will help Proxmox team, otherwise I'll just use my 82576 solution. I am curious....

i210 update screen capture.png

Please pardon the double attachment, can't figure out how to delete the duplicate.
 

Attachments

  • i210 update screen capture.png
    i210 update screen capture.png
    200.5 KB · Views: 1
FWIW I booted from a debian bullseye live USB. No problem with ip connectivity. I've read a few recent reports of bullseye with the i210 drivers... This live USB kernel is a bit older though...

Code: 
uname -a

Linux debian 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) x86_64 GNU/Linux

Code: 
ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether ec:b1:d7:33:35:94 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.161/24 brd 192.168.1.255 scope global dynamic noprefixroute enp5s0
       valid_lft 86226sec preferred_lft 86226sec
    inet6 fe80::2984:a118:2cd6:100a/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether ec:b1:d7:33:35:93 brd ff:ff:ff:ff:ff:ff
    altname enp0s25
 
@mira check this out. I got a working pve7 on the HP Z840 in question. I burnt a usb stick with proxmox-ve_7.0-2.iso installing onto a single spare Kintgston SSD - the problem child in this post was a pair of 1g Samsung in a zfs mirror installed with version 6 media, then upgraded to 7 where all the networking issues started.

The pve 7 install media has 5.11.22-4-pve The pve 6->7 upgrade has 5.11.22-5-pve

I'm willing to downgrade (or upgrade) the kernel to help identify the problem if you can assist with download and install.

pve2 is the problem child. wonder is the test fresh pve 7 install

Eventually I would like to use the onboard i210 ethernet on pve2, I have some configuration that I would rather not wipe. I do have connectivity through the dual port 82576 card so I can download / install from the internet.

Code: 
root@wonder:~# uname -a
Linux wonder 5.11.22-4-pve #1 SMP PVE 5.11.22-8 (Fri, 27 Aug 2021 11:51:34 +0200) x86_64 GNU/Linux
root@wonder:~#

Code: 
root@wonder:~# lshw -class network
  *-network DISABLED       
       description: Ethernet interface
       product: Ethernet Connection (2) I218-LM
       vendor: Intel Corporation
       physical id: 19
       bus info: pci@0000:00:19.0
       logical name: eno1
       version: 05
       serial: ec:b1:d7:33:35:93
       capacity: 1Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=e1000e driverversion=5.11.22-4-pve firmware=0.1-4 latency=0 link=no multicast=yes port=twisted pair
       resources: irq:39 memory:d0400000-d041ffff memory:d043b000-d043bfff ioport:6040(size=32)
  *-network
       description: Ethernet interface
       product: I210 Gigabit Network Connection
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:05:00.0
       logical name: enp5s0
       version: 03
       serial: ec:b1:d7:33:35:94
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=igb driverversion=5.11.22-4-pve duplex=full firmware=3.25, 0x800005fa latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
       resources: irq:16 memory:d0100000-d017ffff ioport:2000(size=32) memory:d0180000-d0183fff
  *-network
       description: Ethernet interface
       physical id: 1
       logical name: vmbr0
       serial: ec:b1:d7:33:35:94
       size: 1Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A ip=192.168.1.161 link=yes multicast=yes speed=1Gbit/s
root@wonder:~#

Code: 
root@wonder:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether ec:b1:d7:33:35:94 brd ff:ff:ff:ff:ff:ff
3: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ec:b1:d7:33:35:93 brd ff:ff:ff:ff:ff:ff
    altname enp0s25
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ec:b1:d7:33:35:94 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.161/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::eeb1:d7ff:fe33:3594/64 scope link
       valid_lft forever preferred_lft forever
root@wonder:~#
 
Success i210 Ethernet on HP Z840 working again.

I was about to try Linux Kernel 5.13, ZFS 2.1 for Proxmox VE from a post by @t.lamprecht but first I saw there were some updates available including a kernel Linux 5.11.22-7-pve I applied them and rebooted. The enp5s0 interface came up steady, so changed /etc/network/interfaces back to reflect the onboard ethernet.

@t.lamprecht I'm curious will future users that may try a 6.4 -> 7.0-13 upgrade on their HP Z840 (possibly others) experience networking failure as I did, or will the update process install 5.11.22-7 or newer rather than the 5.11.22-5 that apparently caused me this grief with network connectivity. I'm assuming the update process will use the most current kernel.

Is there a tag for 'solved'?

cc: @mira
 
Last edited:
  • Like
Reactions: mira
GumShoeNoir said:
Success i210 Ethernet on HP Z840 working again.

I was about to try Linux Kernel 5.13, ZFS 2.1 for Proxmox VE from a post by @t.lamprecht but first I saw there were some updates available including a kernel Linux 5.11.22-7-pve I applied them and rebooted. The enp5s0 interface came up steady, so changed /etc/network/interfaces back to reflect the onboard ethernet.
Click to expand...
great to hear!

GumShoeNoir said:
@t.lamprecht I'm curious will future users that may try a 6.4 -> 7.0-13 upgrade on their HP Z840 (possibly others) experience networking failure as I did, or will the update process install 5.11.22-7 or newer rather than the 5.11.22-5 that apparently caused me this grief with network connectivity. I'm assuming the update process will use the most current kernel.
Click to expand...
Yes, the newest kernel from the repositories configured will get pulled in on upgrade.

GumShoeNoir said:
Is there a tag for 'solved'?
Click to expand...
You can set such a prefix by using the Edit Thread button on top.
 
You must log in or register to reply here.
Top Bottom