HP StorageWorks CN1000E Dual 10Gb Fiber NIC

[mrdmadev]

I'm trying to get the NIC mentioned in the subject line to work for more than a few minutes in Proxmox. I've done a lot of testing and interrogating, so I don't know what info to include. I realize too much information can be a bad thing, but bare with me. The initial symptom was accessing the Proxmox GUI and after x-amount of time, no longer having access via the GUI or ping. After reboot, could ping the Proxmox host for about 3 minutes then the NIC would stop responding. So, I jumped on the host and tried to ping out to both the gateway and www.google.com and it would only ping them, or any other host 12 times, then give me an error for each remaining ping: "ping: sendmsg: No buffer space available". I can ping it's IP address (ping itself) at the host and it pings forever which normal and expected.

I can reboot, and start all over again but I get the eventual same result as above.

This problem DOES NOT exist on Debian Live Boot USB on same machine. The kernel version was 4.19.some#. The driver loaded is be2net and version 12.0.0.0 via $lshw. Ping works for hours.
This problem DOES NOT exist on Fedora Live Boot USB on same machine. The kernel version wasn't looked for. The driver loaded is be2net and version 12.0.0.0 via $lshw. Ping works for hours

* * * * * * * *
/etc/network/interfaces

auto lo
iface lo inet loopback
iface enp3s0 inet manual
iface enp2s0f0 inet manual
iface enp2s0f1 inet manual
auto vmbr0
iface vmbr0 inet static
address 192.168.0.201/24
gateway 192.168.0.1
bridge-ports enp2s0f0
bridge-stp off
bridge-fd 0

* * * * * * * *

The same driver and version is loaded in Proxmox as it was in the Debian Live Boot and Fedora Live Boot via $lshw, kernel version is 5.4.106-1-pve, and I get the results/symptoms mentioned in the opening paragraph.

Some digging led me to https://www.cyberciti.biz/faq/linux-tcp-tuning/ where I adjusted the recommended settings for /etc/sysctl.conf at that site. Still the same problem.

I'm not sure what to try next. Any direction would be appreciated.

PS. I almost hate mentioning this, because this has no bearing on Fedora or Debian (mentioned above) on the same machine, but the firmware for the NIC is at 2.X and the latest version is 4.X. I am working on a way to update the firmware, but I would rather not hassle with that, as the update utilities only seem to be available in openSUSE Enterprise, RedHat Enterprise, and older Windows Server editions.

Thanks in advance for any ideas that I can try.

EDIT: Added some clarity to driver and version in Proxmox.
EDIT 2: This NIC is installed on a consumer grade Gigabyte motherboard, not HP hardware. Which again, has no bearing on other distros working fine.

 

[mira]

Can you try the latest Ubuntu 20.04 as well? The PVE kernel is based on the one from Ubuntu 20.04.
You could also opt-in for the 5.11 kernel by installing it via apt install pve-kernel-5.11

 

[mrdmadev]

Can you try the latest Ubuntu 20.04 as well? The PVE kernel is based on the one from Ubuntu 20.04.
You could also opt-in for the 5.11 kernel by installing it via apt install pve-kernel-5.11

The Ubuntu 20.04.02 LTS live desktop image doesn’t allow me to ping out or in to the gateway or urls. The port does get assigned an IP address, but no ingress or egress. The driver that gets loaded is be2net version 5.8.0-43-generic.

Should I go ahead with opting in on the pve-kernel-5.11 and give it a spin?

 

[mira]

I'd rather recommend updating the firmware. It could be that the newer driver versions are not compatible with the old firmware.

 

[mrdmadev]

I'd rather recommend updating the firmware. It could be that the newer driver versions are not compatible with the old firmware.

I spent at least half a day for the last few days trying to find the right combination of supported OS and flash utility tools on HPs site and I finally found the right combination and got the firmware flashed on both cards.

Same outcome. It loads the same driver as before. It can ping itself (the Proxmox host machine) from its CLI forever. When I ping the network gateway with it, it pings it 12 times, pauses, then comes back with "ping: sendmsg: No buffer space available" every second thereafter. The only positive is that it doesn't complain at boot that the firmware is too old like it was doing previously.

All other distros I have tested work fine except Ubuntu Desktop Live USB (loads a generic driver) and Proxmox (loads be2net v12.0.0.0, which is correct).

Edit: using modprobe, the version (12.0.0.0) is the same, but the src: is different.

Should I hit up some Linux forums for something deeper? Like maybe swapping/transplanting .ko files? Or somehow preventing the loading of the kernel driver Proxmox provides, and forcing a different one to load if I can bring a different one in? Is that even possible? There has to be a way for me to get this to work.

PS. For anyone finding this thread by accident because they want to update firmware in these cards, the only combo I got to work was downloading and temporarily installing Windows Server 2012R2, downloading and installing the 1G/10G drivers, the FCoE drivers, and the flash utility from the HP support site.

 

[mira]

Which distributions did you try and which kernel versions did they ship?
Any kernel newer than 5.4?

PS. For anyone finding this thread by accident because they want to update firmware in these cards, the only combo I got to work was downloading and temporarily installing Windows Server 2012R2, downloading and installing the 1G/10G drivers, the FCoE drivers, and the flash utility from the HP support site.

That sounds terrible.

 

[mrdmadev]

Which distributions did you try and which kernel versions did they ship?
Any kernel newer than 5.4?


That sounds terrible.

Works	Distro	Kernel Release	Kernel Version	Driver Name / Description	Driver srcversion via modinfo
Yes	Debian LXCE Live Boot ISO	4.19.0-16-AMD64	#1 SMP Debian 4.19.181-1(2021-03-18)	be2net / Emulex OneConnect NIC Driver 12.0.0.0	C3E8B2COAFECBDBC992AF14
"ping: sendmsg: No buffer space available after" 12th ping. Ctrl+c out of ping, and then ping 12 times again, lather, rinse, repeat.	Ubuntu	5.8.0-43-Generic	#49-20.04.1-ubuntu SMP Fri Feb 5 09:57 UTC 2021	be2net / Emulex OneConnect NIC Driver (note: driver version omitted)	C04C6D8180A7523FC5A3180
"ping: sendmsg: No buffer space available after" 12th ping. Ctrl+c out of ping, and then ping 12 times again, lather, rinse, repeat.	Proxmox	Whatever is most recent
Yes	Fedora Desktop Live	5.6.6-300.FC32.x86_64	#1 SMP TUE APR 21 13:44:19 UTC 2020	be2net / Emulex OneConnect NIC Driver 12.0.0.0	7650220B852644D8C0FF39

What are my options? I'm so far down the rabbit hole on this, I don't mind compiling/maintaining a driver if it's even possible. I don't need to know line-by-line commands if it is possible, but a description of what I need to do on the macro level would be nice. I've never worked with the kernel and/or drivers, but it is something I have always wanted to learn, so I'm game to do whatever needs to be done. Is it as "simple" as pulling source code for the driver out of the debian kernel (since it has been the most reliable) and compiling against the kernel used for Proxmox?

Thank you for helping me this far.

 

[mrdmadev]

I’d like to bump this once since it has been 5 days. Anyone here willing to at least provide me a direction to go?

Thank you!

 

[mira]

I'd suggest trying the pve-kernel-5.11 package.
If it works -> Great!
If it doesn't work, try Ubuntu 21.04 Live ISO. It contains the 5.11 kernel our kernel is based on.
If the Live ISO with kernel 5.11 doesn't work, I'd suggest trying the mainline kernel in Ubuntu: https://wiki.ubuntu.com/Kernel/MainlineBuilds
If the mainline kernel works, but the Ubuntu one doesn't, we can then narrow it down further.