Chelsio T520-SO-CR freezes Proxmox during installation !!

M@trox · Feb 27, 2019

Hi,

When trying to install a fresh Proxmox 5.3 it is freezing during new installation with a NIC Chelsio T520-SO-CR.
I have flashed the card with latest firmware v1.21.5.0.
This is the only card connected to my new build.

It is halting with error->

could not find config file cxgb4/t5-config.txt

Is the driver cxgb4 for this card in Proxmox 5.3 not updated/compatible with kernel 4.15?

XCp-ng 7.6.0 with kernel 4.4 had a similar problem. I installed it without the T520 NIC, connected it afterwards and the interface did not work. After they helped me to install a new rebuild of the driver we managed to get it to work after we upgraded the kernel to 4.9. I believe we also blacklisted

csiostor

from kernel bootup.

Chelsio support recommended me to use a newer kernel, since there is an issue with reading VPD from the card in older kernels. Because of this I thought Proxmox would be the one to use since it is using a much newer kernel than XCP-ng or XenServer.

Is there someone who can help me fix this or even better if Proxmox team could update their driver for this card?

LnxBil · Feb 27, 2019

Try to install Debian Stretch and then PVE on top, maybe this solves your issue:

https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Stretch

M@trox · Feb 28, 2019

Thank you for the suggestion.
After installation of Debian Stretch 9.8 on kernel 4.9, the Chelsio T520-SO-CR NIC was working straight away. But after I installed Proxmox VE 5.3 on top of that and the system upgraded it self to kernel 4.15, the behaviour is the same as installing Proxmox iso? The T520 NIC is not initializing.
Has anyone got his Chelsio T520-SO-CR working on Proxmox VE 5.3 ?
Does anyone have any hint or suggestion of what I can try?

# lspci -k

Code:

65:00.0 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Subsystem: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Kernel driver in use: cxgb4
    Kernel modules: cxgb4
65:00.1 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Subsystem: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Kernel driver in use: cxgb4
    Kernel modules: cxgb4
65:00.2 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Subsystem: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Kernel modules: cxgb4
65:00.3 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Subsystem: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Kernel modules: cxgb4
65:00.4 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Subsystem: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Kernel modules: cxgb4
65:00.5 SCSI storage controller: Chelsio Communications Inc T520-SO Unified Wire Storage Controller
    Subsystem: Chelsio Communications Inc T520-SO Unified Wire Storage Controller
65:00.6 Fibre Channel: Chelsio Communications Inc T520-SO Unified Wire Storage Controller
    Subsystem: Chelsio Communications Inc T520-SO Unified Wire Storage Controller
    Kernel driver in use: csiostor
    Kernel modules: csiostor

# dmesg

Code:

[    1.620979] Chelsio T4/T5/T6 Network Driver - version 2.0.0-ko
[    1.621570] csiostor: Chelsio FCoE driver 1.0.0-ko
[    1.649452] csiostor 0000:65:00.6: Firmware revision: 1.21.5.0
[    1.654418] csiostor 0000:65:00.6: PF: 6, Coming up as MASTER, HW state: Initializing

[    2.218725] csiostor 0000:65:00.6: Direct firmware load for cxgb4/t5-config.txt failed with error -2
[    2.218727] csiostor 0000:65:00.6: could not find config file cxgb4/t5-config.txt, err: 0
[    2.253598] csiostor 0000:65:00.6: No FCoE Initiator capability in the firmware.
[    2.253710] csiostor 0000:65:00.6: Configuration file error -22
[    2.253711] csiostor 0000:65:00.6: Could not initialize adapter, error-22

[  242.652887]  cxgb4_init_module+0x4a/0x1000 [cxgb4]
[  363.506669]  cxgb4_init_module+0x4a/0x1000 [cxgb4]
[  484.351139]  cxgb4_init_module+0x4a/0x1000 [cxgb4]

tman222 · Apr 13, 2019

I realize this thread is already a couple months old now, but I wanted to chime in here with a solution since I also had quite a bit of trouble recently getting a Chelsio network card to work under Proxmox 5.3. In my case I was using the T520's bigger brother, the 4 port Chelsio T540-SO-CR.

Similar to the OP, after installing the card (T540) it was not properly recognized and configured during system boot (even though the pve-firmware package was installed). I decided to go Chelsio's website to download drivers and saw that I would have to compile them from source (i.e. there was no pre-compiled Debian package available). So I went ahead and downloaded the latest Chelsio Unified Wire software (3.10 at the time) and attempted to compile the drivers. After a couple hours of trial and error I finally got the driver to compile and install. Here are some important notes from that time:

1) You'll need to install the Promox Linux kernel header files (i.e. pve-headers) if you haven't done so already.
2) Inside the Chelsio driver Makefile, right at the stop, set the "DEBIAN" variable manually to 1. From what I can tell, the make script tries to identify which version of Linux (Debian variant) is on the system but has trouble with Proxmox.
3) If the driver compile still fails after completing 1) and 2) check the logs - there may additional dependencies (packages) that need to be installed.

Once the driver was installed and I rebooted the system, the card was recognized and I was able configure everything just fine. 10Gbit performance is great - using the virtio_net driver I am able to achieve north 9Gbit/s when running a simple iperf3 test between a bare metal Linux host and Linux VM running on top of Proxmox.

Now, the only downside to these steps is that the driver will need to recompiled again after a kernel upgrade. Unfortunately, I have not quite figured out yet how to get DKMS to work. Anyhow, I hope this helps others which may be facing similar troubles.

-----------------------------

EDIT 7/20/19 to reflect correct Unified Wire software version.

joachim · Jul 16, 2019

tman222 said:
After a couple hours of trial and error I finally got the driver to compile and install.

Did you encounter the compile errors caused by actual errors in the code? (see below). Proxmox 5.3 with the 3.11 driver from Chelsio.

Code:

##################################################
#          Building Network Drivers              #
##################################################
Preparing for build ...
#### Preparing cxgb4 ####
make[3]: Nothing to be done for 'prep'.
#######################
####  cxgb4        ####
#######################
  CC [M]  /root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.o
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c: In function ‘cxgb_select_queue’:
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:2015:9: error: too many arguments to function ‘fallback’
  return fallback(dev, skb, NULL) % dev->real_num_tx_queues;
         ^~~~~~~~
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c: At top level:
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:7926:26: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
  .ndo_select_queue     = cxgb_select_queue,
                          ^~~~~~~~~~~~~~~~~
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:7926:26: note: (near initialization for ‘cxgb4_netdev_ops.ndo_select_queue’)
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c: In function ‘init_one’:
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:9092:2: error: implicit declaration of function ‘pcie_print_link_status’ [-Werror=implicit-function-declaration]
  pcie_print_link_status(adapter->pdev);
  ^~~~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
scripts/Makefile.build:322: recipe for target '/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.o' failed
make[5]: *** [/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.o] Error 1
Makefile:1552: recipe for target '_module_/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4' failed
make[4]: *** [_module_/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4] Error 2
Makefile:86: recipe for target 'build' failed
make[3]: *** [build] Error 2
Makefile:1261: recipe for target 'cxgb4' failed
make[2]: *** [cxgb4] Error 2

levifig · Jul 20, 2019

Does the UnifiedWire even install on Proxmox 6? I can't seem to have it recognize the headers, and I wonder if it's just not ready for kernel 5.x… :\

tman222 · Jul 21, 2019

joachim said:

Did you encounter the compile errors caused by actual errors in the code? (see below). Proxmox 5.3 with the 3.11 driver from Chelsio.

Code:

##################################################
#          Building Network Drivers              #
##################################################
Preparing for build ...
#### Preparing cxgb4 ####
make[3]: Nothing to be done for 'prep'.
#######################
####  cxgb4        ####
#######################
  CC [M]  /root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.o
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c: In function ‘cxgb_select_queue’:
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:2015:9: error: too many arguments to function ‘fallback’
  return fallback(dev, skb, NULL) % dev->real_num_tx_queues;
         ^~~~~~~~
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c: At top level:
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:7926:26: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
  .ndo_select_queue     = cxgb_select_queue,
                          ^~~~~~~~~~~~~~~~~
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:7926:26: note: (near initialization for ‘cxgb4_netdev_ops.ndo_select_queue’)
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c: In function ‘init_one’:
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:9092:2: error: implicit declaration of function ‘pcie_print_link_status’ [-Werror=implicit-function-declaration]
  pcie_print_link_status(adapter->pdev);
  ^~~~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
scripts/Makefile.build:322: recipe for target '/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.o' failed
make[5]: *** [/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.o] Error 1
Makefile:1552: recipe for target '_module_/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4' failed
make[4]: *** [_module_/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4] Error 2
Makefile:86: recipe for target 'build' failed
make[3]: *** [build] Error 2
Makefile:1261: recipe for target 'cxgb4' failed
make[2]: *** [cxgb4] Error 2

Hi joachim - My apologies, I just noticed that I made a mistake in my prior post. It was actually the Chelsio 3.10 drivers that I was using and installed, not the 3.11'.s The 3.10's compile fine under the current version of Promox, but when I tried to compile the 3.11's recently I got the same errors you did. I also managed to get DKMS to work recently - I'd be happy share my notes on that if there is interest.

tman222 · Jul 21, 2019

levifig said:
Does the UnifiedWire even install on Proxmox 6? I can't seem to have it recognize the headers, and I wonder if it's just not ready for kernel 5.x… :\

Hi levifig -- how are you trying to install the drivers? Are you compiling from source, following the Python install script, or trying to use DKMS?

levifig · Jul 21, 2019

tman222 said:
Hi levifig -- how are you trying to install the drivers? Are you compiling from source, following the Python install script, or trying to use DKMS?

Downloaded the drivers, unpacked the tar.gz, ran ./install.py. When it reached the part where it needed the "kernel source", I installed the pve-headers package for my current kernel, and it still seemed to fail to find the sources… :\

Code:

 Building for kernel 5.0.15-1-pve                                                                                                                         │
 │ /bin/bash: rpm: command not found                                                                                                                        │
 │ ##################################################                                                                                                       │
 │ #          Cleaning Source/Build                 #                                                                                                       │
 │ ##################################################                                                                                                       │
 │ Building for kernel 5.0.15-1-pve                                                                                                                         │
 │                                                                                                                                                          │
 │ /bin/bash: rpm: command not found                                                                                                                        │
 │ ##################################################                                                                                                       │
 │ #          Cleaning Source/Build/RPMDir          #                                                                                                       │
 │ ##################################################                                                                                                       │
 │ Building for kernel 5.0.15-1-pve                                                                                                                         │
 │ Makefile:620: When using KOBJ=<path>, the KSRC=<path> must also be defined.                                                                              │
 │ Makefile:621: Use KDIR=<path> when KSRC and KOBJ are the same.                                                                                           │
 │ Makefile:622: *** ERROR: kernel source path not specified.  Stop.                                                                                        │
 │ make: *** [Makefile:4585: removeallPrevious] Error 1

I also find it odd that it's looking for the rpm package, but then again: it could be trying to install the kernel sources... :\

tman222 · Jul 21, 2019

levifig said:

Downloaded the drivers, unpacked the tar.gz, ran ./install.py. When it reached the part where it needed the "kernel source", I installed the pve-headers package for my current kernel, and it still seemed to fail to find the sources… :\

Code:

 Building for kernel 5.0.15-1-pve                                                                                                                         │
 │ /bin/bash: rpm: command not found                                                                                                                        │
 │ ##################################################                                                                                                       │
 │ #          Cleaning Source/Build                 #                                                                                                       │
 │ ##################################################                                                                                                       │
 │ Building for kernel 5.0.15-1-pve                                                                                                                         │
 │                                                                                                                                                          │
 │ /bin/bash: rpm: command not found                                                                                                                        │
 │ ##################################################                                                                                                       │
 │ #          Cleaning Source/Build/RPMDir          #                                                                                                       │
 │ ##################################################                                                                                                       │
 │ Building for kernel 5.0.15-1-pve                                                                                                                         │
 │ Makefile:620: When using KOBJ=<path>, the KSRC=<path> must also be defined.                                                                              │
 │ Makefile:621: Use KDIR=<path> when KSRC and KOBJ are the same.                                                                                           │
 │ Makefile:622: *** ERROR: kernel source path not specified.  Stop.                                                                                        │
 │ make: *** [Makefile:4585: removeallPrevious] Error 1

I also find it odd that it's looking for the rpm package, but then again: it could be trying to install the kernel sources... :\

Just to confirm, did you set Debian = 1 in the Makefile?

levifig · Jul 21, 2019

tman222 said:
Just to confirm, did you set Debian = 1 in the Makefile?

Holy heck! I scoured the manual and couldn't find a single pointer about the debrules folder and how to use it instead!!

It's installed now! THANK YOU!!

PS: I remember glossing over a DEBIAN variable on this thread, but apparently I kept looking in the install.py not the Makefile... #sillyme :X

tman222 · Jul 21, 2019

levifig said:
Holy heck! I scoured the manual and couldn't find a single pointer about the debrules folder and how to use it instead!!

It's installed now! THANK YOU!!

PS: I remember glossing over a DEBIAN variable on this thread, but apparently I kept looking in the install.py not the Makefile... #sillyme :X

That's great news! Out of curiosity, which version of the Unified Wire software drivers did you install - was it the 3.10's or 3.11's? I'm currently running Proxmox 5.4 and the 3.11's throw some compile errors though the 3.10's work fine (see above). Would be great if the 3.11's now worked worked fine with Proxmox 6.0 though.

levifig · Jul 21, 2019

tman222 said:
That's great news! Out of curiosity, which version of the Unified Wire software drivers did you install - was it the 3.10's or 3.11's? I'm currently running Proxmox 5.4 and the 3.11's throw some compile errors though the 3.10's work fine (see above). Would be great if the 3.11's now worked worked fine with Proxmox 6.0 though.

ChelsioUwire-3.11.0.0 was what I used.

tman222 · Aug 11, 2019

Decided to upgrade my my Proxmox box from 5.4 to 6.0 recently and was surprised that after the upgrade the Chelsio card was still recognized and working fine. At first I thought that the 3.10 driver based DKMS module that I had working and built under Proxmox 5.4 had just been compiled for Proxmox 6.0 and 5.x kernel. However, after further investigation it turns out that during the upgrade the cxgb4 module out of the 5.x kernel tree was actually used instead (vs. the DKMS module that been built from the 3.10 drivers). So now I'm a bit perplexed why the card was not recognized under Proxmox 5. Could it be simply that because the module was running already during the upgrade from Proxmox 5 to 6 the installer recognized the card and installed the cxgb4 module out of the 5.x kernel tree? Or maybe has all this been fixed with Proxmox 6 and there is no longer a need to manually install the Chelsio drivers? Thanks in advance for any input.

M K · Aug 12, 2019

tman222 said:
Just to confirm, did you set Debian = 1 in the Makefile?

This works. Was able to get the Unified Wire 3.11 drivers installed on Proxmox 6.
DKMS is an issue however as updating the kernel required me to install the Unified Wire drivers again.

Rphoton · Oct 31, 2019

Hello, I'm a new user here. T520-CR on Proxmox 6.0 like the previous posters.
After setting Debian=1, installing kernel files, etc, I got to the point of a succesfull install, according to the installer. However, I am unable to find cxgbtool anywhere in my system.
man cxgbtool works and it displays the help page but I can't access the binary.
Am I doing something obvious wrong?

Rphoton · Nov 20, 2019

I downloaded the 3.12 drivers from the beta site, repeated the steps, and now it seems to work ok and cxgbtool does work.

Dave.r · Dec 10, 2019

Hi Guys ,
I have downloaded the latest beta version from ChelsioUwire-3.12.0.1.tar.gz ( 3.12.0.1 version ) with latest proxmox 6.1 and it says " Kernel not supported " .
This is so frustrating...

pveversion -V
proxmox-ve: 6.1-2 (running kernel: 5.3.13-1-pve)
pve-manager: 6.1-3 (running version: 6.1-3/37248ce6)
pve-kernel-5.3: 6.1-1
pve-kernel-helper: 6.1-1
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-2
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.1-2
pve-container: 3.0-14
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-3
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

bummster · Feb 27, 2020

Got mine (T422-CR) almost working.
In my scenario my T422-CR was on firmware 1.3 and dmesg was complaining about firmware and how it couldn't flash it. error -2
Proxmox version is 6.1 running Kernel 5.3.18.
downloaded the tar.gz from chelsio. 3.12.
Then i hit the compile problem of having too new of a kernel.
Did the apt dance and got pve-kernel 5.0 onto my pve host. I couldn't uninstall via apt-get my 5.3 kernel because it was trying to remove pve with it.
So instead, I rebooted it to grub, arrowed down to advanced, and selected the 5.0 kernel i installed above.
Then did the python install.py dance with the Debian=1 in the MakeFile.
Rebooted back to kernel 5.3 and my card picked up its new firmware, 1.24.11. confirmed with an ethtool -i <interface mine is enp1s0f4> i can talk to it just fine via cli.

of course, now my dang card won't bring any interfaces up. All four interfaces appear in system->network with a link state of down no matter what cable, port, or flavor of nic conf i try. but it's progress? =)
I noticed that the 3.12 chelsio driver does not list my card in the supported device list, but 3.11 does. so i'm headed down that path now.

Chelsio T520-SO-CR freezes Proxmox during installation !!

Member

Distinguished Member

Member

Member

Renowned Member

Active Member

Member

Member

Active Member

Member

Active Member

Member

Active Member

Member

Member

New Member

New Member

Member

New Member