Chelsio T520-SO-CR freezes Proxmox during installation !!

M@trox

New Member
Feb 27, 2019
3
0
1
39
Hi,

When trying to install a fresh Proxmox 5.3 it is freezing during new installation with a NIC Chelsio T520-SO-CR.
I have flashed the card with latest firmware v1.21.5.0.
This is the only card connected to my new build.

It is halting with error->
could not find config file cxgb4/t5-config.txt
Proxmox freeze installation_T520.png

Is the driver cxgb4 for this card in Proxmox 5.3 not updated/compatible with kernel 4.15?

XCp-ng 7.6.0 with kernel 4.4 had a similar problem. I installed it without the T520 NIC, connected it afterwards and the interface did not work. After they helped me to install a new rebuild of the driver we managed to get it to work after we upgraded the kernel to 4.9. I believe we also blacklisted
from kernel bootup.

Chelsio support recommended me to use a newer kernel, since there is an issue with reading VPD from the card in older kernels. Because of this I thought Proxmox would be the one to use since it is using a much newer kernel than XCP-ng or XenServer.

Is there someone who can help me fix this or even better if Proxmox team could update their driver for this card?
 

M@trox

New Member
Feb 27, 2019
3
0
1
39
Thank you for the suggestion.
After installation of Debian Stretch 9.8 on kernel 4.9, the Chelsio T520-SO-CR NIC was working straight away. But after I installed Proxmox VE 5.3 on top of that and the system upgraded it self to kernel 4.15, the behaviour is the same as installing Proxmox iso? The T520 NIC is not initializing.
Has anyone got his Chelsio T520-SO-CR working on Proxmox VE 5.3 ?
Does anyone have any hint or suggestion of what I can try?

# lspci -k
Code:
65:00.0 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Subsystem: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Kernel driver in use: cxgb4
    Kernel modules: cxgb4
65:00.1 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Subsystem: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Kernel driver in use: cxgb4
    Kernel modules: cxgb4
65:00.2 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Subsystem: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Kernel modules: cxgb4
65:00.3 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Subsystem: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Kernel modules: cxgb4
65:00.4 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Subsystem: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Kernel modules: cxgb4
65:00.5 SCSI storage controller: Chelsio Communications Inc T520-SO Unified Wire Storage Controller
    Subsystem: Chelsio Communications Inc T520-SO Unified Wire Storage Controller
65:00.6 Fibre Channel: Chelsio Communications Inc T520-SO Unified Wire Storage Controller
    Subsystem: Chelsio Communications Inc T520-SO Unified Wire Storage Controller
    Kernel driver in use: csiostor
    Kernel modules: csiostor
# dmesg
Code:
[    1.620979] Chelsio T4/T5/T6 Network Driver - version 2.0.0-ko
[    1.621570] csiostor: Chelsio FCoE driver 1.0.0-ko
[    1.649452] csiostor 0000:65:00.6: Firmware revision: 1.21.5.0
[    1.654418] csiostor 0000:65:00.6: PF: 6, Coming up as MASTER, HW state: Initializing

[    2.218725] csiostor 0000:65:00.6: Direct firmware load for cxgb4/t5-config.txt failed with error -2
[    2.218727] csiostor 0000:65:00.6: could not find config file cxgb4/t5-config.txt, err: 0
[    2.253598] csiostor 0000:65:00.6: No FCoE Initiator capability in the firmware.
[    2.253710] csiostor 0000:65:00.6: Configuration file error -22
[    2.253711] csiostor 0000:65:00.6: Could not initialize adapter, error-22

[  242.652887]  cxgb4_init_module+0x4a/0x1000 [cxgb4]
[  363.506669]  cxgb4_init_module+0x4a/0x1000 [cxgb4]
[  484.351139]  cxgb4_init_module+0x4a/0x1000 [cxgb4]
 

tman222

New Member
Apr 13, 2019
6
0
1
I realize this thread is already a couple months old now, but I wanted to chime in here with a solution since I also had quite a bit of trouble recently getting a Chelsio network card to work under Proxmox 5.3. In my case I was using the T520's bigger brother, the 4 port Chelsio T540-SO-CR.

Similar to the OP, after installing the card (T540) it was not properly recognized and configured during system boot (even though the pve-firmware package was installed). I decided to go Chelsio's website to download drivers and saw that I would have to compile them from source (i.e. there was no pre-compiled Debian package available). So I went ahead and downloaded the latest Chelsio Unified Wire software (3.10 at the time) and attempted to compile the drivers. After a couple hours of trial and error I finally got the driver to compile and install. Here are some important notes from that time:

1) You'll need to install the Promox Linux kernel header files (i.e. pve-headers) if you haven't done so already.
2) Inside the Chelsio driver Makefile, right at the stop, set the "DEBIAN" variable manually to 1. From what I can tell, the make script tries to identify which version of Linux (Debian variant) is on the system but has trouble with Proxmox.
3) If the driver compile still fails after completing 1) and 2) check the logs - there may additional dependencies (packages) that need to be installed.

Once the driver was installed and I rebooted the system, the card was recognized and I was able configure everything just fine. 10Gbit performance is great - using the virtio_net driver I am able to achieve north 9Gbit/s when running a simple iperf3 test between a bare metal Linux host and Linux VM running on top of Proxmox.

Now, the only downside to these steps is that the driver will need to recompiled again after a kernel upgrade. Unfortunately, I have not quite figured out yet how to get DKMS to work. Anyhow, I hope this helps others which may be facing similar troubles.

-----------------------------

EDIT 7/20/19 to reflect correct Unified Wire software version.
 
Last edited:

joachim

New Member
Nov 19, 2015
11
0
1
After a couple hours of trial and error I finally got the driver to compile and install.
Did you encounter the compile errors caused by actual errors in the code? (see below). Proxmox 5.3 with the 3.11 driver from Chelsio.


Code:
##################################################
#          Building Network Drivers              #
##################################################
Preparing for build ...
#### Preparing cxgb4 ####
make[3]: Nothing to be done for 'prep'.
#######################
####  cxgb4        ####
#######################
  CC [M]  /root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.o
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c: In function ‘cxgb_select_queue’:
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:2015:9: error: too many arguments to function ‘fallback’
  return fallback(dev, skb, NULL) % dev->real_num_tx_queues;
         ^~~~~~~~
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c: At top level:
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:7926:26: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
  .ndo_select_queue     = cxgb_select_queue,
                          ^~~~~~~~~~~~~~~~~
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:7926:26: note: (near initialization for ‘cxgb4_netdev_ops.ndo_select_queue’)
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c: In function ‘init_one’:
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:9092:2: error: implicit declaration of function ‘pcie_print_link_status’ [-Werror=implicit-function-declaration]
  pcie_print_link_status(adapter->pdev);
  ^~~~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
scripts/Makefile.build:322: recipe for target '/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.o' failed
make[5]: *** [/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.o] Error 1
Makefile:1552: recipe for target '_module_/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4' failed
make[4]: *** [_module_/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4] Error 2
Makefile:86: recipe for target 'build' failed
make[3]: *** [build] Error 2
Makefile:1261: recipe for target 'cxgb4' failed
make[2]: *** [cxgb4] Error 2
 

levifig

New Member
Feb 13, 2018
13
0
1
levifig.com
Does the UnifiedWire even install on Proxmox 6? I can't seem to have it recognize the headers, and I wonder if it's just not ready for kernel 5.x… :\
 

tman222

New Member
Apr 13, 2019
6
0
1
Did you encounter the compile errors caused by actual errors in the code? (see below). Proxmox 5.3 with the 3.11 driver from Chelsio.


Code:
##################################################
#          Building Network Drivers              #
##################################################
Preparing for build ...
#### Preparing cxgb4 ####
make[3]: Nothing to be done for 'prep'.
#######################
####  cxgb4        ####
#######################
  CC [M]  /root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.o
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c: In function ‘cxgb_select_queue’:
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:2015:9: error: too many arguments to function ‘fallback’
  return fallback(dev, skb, NULL) % dev->real_num_tx_queues;
         ^~~~~~~~
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c: At top level:
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:7926:26: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
  .ndo_select_queue     = cxgb_select_queue,
                          ^~~~~~~~~~~~~~~~~
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:7926:26: note: (near initialization for ‘cxgb4_netdev_ops.ndo_select_queue’)
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c: In function ‘init_one’:
/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:9092:2: error: implicit declaration of function ‘pcie_print_link_status’ [-Werror=implicit-function-declaration]
  pcie_print_link_status(adapter->pdev);
  ^~~~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
scripts/Makefile.build:322: recipe for target '/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.o' failed
make[5]: *** [/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.o] Error 1
Makefile:1552: recipe for target '_module_/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4' failed
make[4]: *** [_module_/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4] Error 2
Makefile:86: recipe for target 'build' failed
make[3]: *** [build] Error 2
Makefile:1261: recipe for target 'cxgb4' failed
make[2]: *** [cxgb4] Error 2
Hi joachim - My apologies, I just noticed that I made a mistake in my prior post. It was actually the Chelsio 3.10 drivers that I was using and installed, not the 3.11'.s The 3.10's compile fine under the current version of Promox, but when I tried to compile the 3.11's recently I got the same errors you did. I also managed to get DKMS to work recently - I'd be happy share my notes on that if there is interest.
 
Last edited:

tman222

New Member
Apr 13, 2019
6
0
1
Does the UnifiedWire even install on Proxmox 6? I can't seem to have it recognize the headers, and I wonder if it's just not ready for kernel 5.x… :\
Hi levifig -- how are you trying to install the drivers? Are you compiling from source, following the Python install script, or trying to use DKMS?
 

levifig

New Member
Feb 13, 2018
13
0
1
levifig.com
Hi levifig -- how are you trying to install the drivers? Are you compiling from source, following the Python install script, or trying to use DKMS?
Downloaded the drivers, unpacked the tar.gz, ran ./install.py. When it reached the part where it needed the "kernel source", I installed the pve-headers package for my current kernel, and it still seemed to fail to find the sources… :\

Code:
 Building for kernel 5.0.15-1-pve                                                                                                                         │
 │ /bin/bash: rpm: command not found                                                                                                                        │
 │ ##################################################                                                                                                       │
 │ #          Cleaning Source/Build                 #                                                                                                       │
 │ ##################################################                                                                                                       │
 │ Building for kernel 5.0.15-1-pve                                                                                                                         │
 │                                                                                                                                                          │
 │ /bin/bash: rpm: command not found                                                                                                                        │
 │ ##################################################                                                                                                       │
 │ #          Cleaning Source/Build/RPMDir          #                                                                                                       │
 │ ##################################################                                                                                                       │
 │ Building for kernel 5.0.15-1-pve                                                                                                                         │
 │ Makefile:620: When using KOBJ=<path>, the KSRC=<path> must also be defined.                                                                              │
 │ Makefile:621: Use KDIR=<path> when KSRC and KOBJ are the same.                                                                                           │
 │ Makefile:622: *** ERROR: kernel source path not specified.  Stop.                                                                                        │
 │ make: *** [Makefile:4585: removeallPrevious] Error 1
I also find it odd that it's looking for the rpm package, but then again: it could be trying to install the kernel sources... :\
 

tman222

New Member
Apr 13, 2019
6
0
1
Downloaded the drivers, unpacked the tar.gz, ran ./install.py. When it reached the part where it needed the "kernel source", I installed the pve-headers package for my current kernel, and it still seemed to fail to find the sources… :\

Code:
 Building for kernel 5.0.15-1-pve                                                                                                                         │
 │ /bin/bash: rpm: command not found                                                                                                                        │
 │ ##################################################                                                                                                       │
 │ #          Cleaning Source/Build                 #                                                                                                       │
 │ ##################################################                                                                                                       │
 │ Building for kernel 5.0.15-1-pve                                                                                                                         │
 │                                                                                                                                                          │
 │ /bin/bash: rpm: command not found                                                                                                                        │
 │ ##################################################                                                                                                       │
 │ #          Cleaning Source/Build/RPMDir          #                                                                                                       │
 │ ##################################################                                                                                                       │
 │ Building for kernel 5.0.15-1-pve                                                                                                                         │
 │ Makefile:620: When using KOBJ=<path>, the KSRC=<path> must also be defined.                                                                              │
 │ Makefile:621: Use KDIR=<path> when KSRC and KOBJ are the same.                                                                                           │
 │ Makefile:622: *** ERROR: kernel source path not specified.  Stop.                                                                                        │
 │ make: *** [Makefile:4585: removeallPrevious] Error 1
I also find it odd that it's looking for the rpm package, but then again: it could be trying to install the kernel sources... :\
Just to confirm, did you set Debian = 1 in the Makefile?
 

levifig

New Member
Feb 13, 2018
13
0
1
levifig.com
Just to confirm, did you set Debian = 1 in the Makefile?
Holy heck! I scoured the manual and couldn't find a single pointer about the debrules folder and how to use it instead!!

It's installed now! THANK YOU!! :eek:

PS: I remember glossing over a DEBIAN variable on this thread, but apparently I kept looking in the install.py not the Makefile... #sillyme :X
 

tman222

New Member
Apr 13, 2019
6
0
1
Holy heck! I scoured the manual and couldn't find a single pointer about the debrules folder and how to use it instead!!

It's installed now! THANK YOU!! :eek:

PS: I remember glossing over a DEBIAN variable on this thread, but apparently I kept looking in the install.py not the Makefile... #sillyme :X
That's great news! Out of curiosity, which version of the Unified Wire software drivers did you install - was it the 3.10's or 3.11's? I'm currently running Proxmox 5.4 and the 3.11's throw some compile errors though the 3.10's work fine (see above). Would be great if the 3.11's now worked worked fine with Proxmox 6.0 though.
 

levifig

New Member
Feb 13, 2018
13
0
1
levifig.com
That's great news! Out of curiosity, which version of the Unified Wire software drivers did you install - was it the 3.10's or 3.11's? I'm currently running Proxmox 5.4 and the 3.11's throw some compile errors though the 3.10's work fine (see above). Would be great if the 3.11's now worked worked fine with Proxmox 6.0 though.
ChelsioUwire-3.11.0.0 was what I used. :)
 

tman222

New Member
Apr 13, 2019
6
0
1
Decided to upgrade my my Proxmox box from 5.4 to 6.0 recently and was surprised that after the upgrade the Chelsio card was still recognized and working fine. At first I thought that the 3.10 driver based DKMS module that I had working and built under Proxmox 5.4 had just been compiled for Proxmox 6.0 and 5.x kernel. However, after further investigation it turns out that during the upgrade the cxgb4 module out of the 5.x kernel tree was actually used instead (vs. the DKMS module that been built from the 3.10 drivers). So now I'm a bit perplexed why the card was not recognized under Proxmox 5. Could it be simply that because the module was running already during the upgrade from Proxmox 5 to 6 the installer recognized the card and installed the cxgb4 module out of the 5.x kernel tree? Or maybe has all this been fixed with Proxmox 6 and there is no longer a need to manually install the Chelsio drivers? Thanks in advance for any input.
 

M K

New Member
Mar 21, 2019
1
0
1
Just to confirm, did you set Debian = 1 in the Makefile?
This works. Was able to get the Unified Wire 3.11 drivers installed on Proxmox 6.
DKMS is an issue however as updating the kernel required me to install the Unified Wire drivers again.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!