Chelsio T520-SO-CR freezes Proxmox during installation !!

Discussion in 'Proxmox VE: Networking and Firewall' started by M@trox, Feb 27, 2019.

  1. M@trox

    M@trox New Member

    Joined:
    Feb 27, 2019
    Messages:
    3
    Likes Received:
    0
    Hi,

    When trying to install a fresh Proxmox 5.3 it is freezing during new installation with a NIC Chelsio T520-SO-CR.
    I have flashed the card with latest firmware v1.21.5.0.
    This is the only card connected to my new build.

    It is halting with error->
    Proxmox freeze installation_T520.png

    Is the driver cxgb4 for this card in Proxmox 5.3 not updated/compatible with kernel 4.15?

    XCp-ng 7.6.0 with kernel 4.4 had a similar problem. I installed it without the T520 NIC, connected it afterwards and the interface did not work. After they helped me to install a new rebuild of the driver we managed to get it to work after we upgraded the kernel to 4.9. I believe we also blacklisted
    from kernel bootup.

    Chelsio support recommended me to use a newer kernel, since there is an issue with reading VPD from the card in older kernels. Because of this I thought Proxmox would be the one to use since it is using a much newer kernel than XCP-ng or XenServer.

    Is there someone who can help me fix this or even better if Proxmox team could update their driver for this card?
     
  2. LnxBil

    LnxBil Well-Known Member

    Joined:
    Feb 21, 2015
    Messages:
    3,937
    Likes Received:
    365
  3. M@trox

    M@trox New Member

    Joined:
    Feb 27, 2019
    Messages:
    3
    Likes Received:
    0
    Thank you for the suggestion.
    After installation of Debian Stretch 9.8 on kernel 4.9, the Chelsio T520-SO-CR NIC was working straight away. But after I installed Proxmox VE 5.3 on top of that and the system upgraded it self to kernel 4.15, the behaviour is the same as installing Proxmox iso? The T520 NIC is not initializing.
    Has anyone got his Chelsio T520-SO-CR working on Proxmox VE 5.3 ?
    Does anyone have any hint or suggestion of what I can try?

    # lspci -k
    Code:
    65:00.0 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
        Subsystem: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
        Kernel driver in use: cxgb4
        Kernel modules: cxgb4
    65:00.1 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
        Subsystem: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
        Kernel driver in use: cxgb4
        Kernel modules: cxgb4
    65:00.2 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
        Subsystem: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
        Kernel modules: cxgb4
    65:00.3 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
        Subsystem: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
        Kernel modules: cxgb4
    65:00.4 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
        Subsystem: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
        Kernel modules: cxgb4
    65:00.5 SCSI storage controller: Chelsio Communications Inc T520-SO Unified Wire Storage Controller
        Subsystem: Chelsio Communications Inc T520-SO Unified Wire Storage Controller
    65:00.6 Fibre Channel: Chelsio Communications Inc T520-SO Unified Wire Storage Controller
        Subsystem: Chelsio Communications Inc T520-SO Unified Wire Storage Controller
        Kernel driver in use: csiostor
        Kernel modules: csiostor
    # dmesg
    Code:
    [    1.620979] Chelsio T4/T5/T6 Network Driver - version 2.0.0-ko
    [    1.621570] csiostor: Chelsio FCoE driver 1.0.0-ko
    [    1.649452] csiostor 0000:65:00.6: Firmware revision: 1.21.5.0
    [    1.654418] csiostor 0000:65:00.6: PF: 6, Coming up as MASTER, HW state: Initializing
    
    [    2.218725] csiostor 0000:65:00.6: Direct firmware load for cxgb4/t5-config.txt failed with error -2
    [    2.218727] csiostor 0000:65:00.6: could not find config file cxgb4/t5-config.txt, err: 0
    [    2.253598] csiostor 0000:65:00.6: No FCoE Initiator capability in the firmware.
    [    2.253710] csiostor 0000:65:00.6: Configuration file error -22
    [    2.253711] csiostor 0000:65:00.6: Could not initialize adapter, error-22
    
    [  242.652887]  cxgb4_init_module+0x4a/0x1000 [cxgb4]
    [  363.506669]  cxgb4_init_module+0x4a/0x1000 [cxgb4]
    [  484.351139]  cxgb4_init_module+0x4a/0x1000 [cxgb4]
     
  4. tman222

    tman222 New Member

    Joined:
    Apr 13, 2019
    Messages:
    6
    Likes Received:
    0
    I realize this thread is already a couple months old now, but I wanted to chime in here with a solution since I also had quite a bit of trouble recently getting a Chelsio network card to work under Proxmox 5.3. In my case I was using the T520's bigger brother, the 4 port Chelsio T540-SO-CR.

    Similar to the OP, after installing the card (T540) it was not properly recognized and configured during system boot (even though the pve-firmware package was installed). I decided to go Chelsio's website to download drivers and saw that I would have to compile them from source (i.e. there was no pre-compiled Debian package available). So I went ahead and downloaded the latest Chelsio Unified Wire software (3.10 at the time) and attempted to compile the drivers. After a couple hours of trial and error I finally got the driver to compile and install. Here are some important notes from that time:

    1) You'll need to install the Promox Linux kernel header files (i.e. pve-headers) if you haven't done so already.
    2) Inside the Chelsio driver Makefile, right at the stop, set the "DEBIAN" variable manually to 1. From what I can tell, the make script tries to identify which version of Linux (Debian variant) is on the system but has trouble with Proxmox.
    3) If the driver compile still fails after completing 1) and 2) check the logs - there may additional dependencies (packages) that need to be installed.

    Once the driver was installed and I rebooted the system, the card was recognized and I was able configure everything just fine. 10Gbit performance is great - using the virtio_net driver I am able to achieve north 9Gbit/s when running a simple iperf3 test between a bare metal Linux host and Linux VM running on top of Proxmox.

    Now, the only downside to these steps is that the driver will need to recompiled again after a kernel upgrade. Unfortunately, I have not quite figured out yet how to get DKMS to work. Anyhow, I hope this helps others which may be facing similar troubles.

    -----------------------------

    EDIT 7/20/19 to reflect correct Unified Wire software version.
     
    #4 tman222, Apr 13, 2019
    Last edited: Jul 21, 2019
  5. joachim

    joachim New Member

    Joined:
    Nov 19, 2015
    Messages:
    11
    Likes Received:
    0
    Did you encounter the compile errors caused by actual errors in the code? (see below). Proxmox 5.3 with the 3.11 driver from Chelsio.


    Code:
    ##################################################
    #          Building Network Drivers              #
    ##################################################
    Preparing for build ...
    #### Preparing cxgb4 ####
    make[3]: Nothing to be done for 'prep'.
    #######################
    ####  cxgb4        ####
    #######################
      CC [M]  /root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.o
    /root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c: In function ‘cxgb_select_queue’:
    /root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:2015:9: error: too many arguments to function ‘fallback’
      return fallback(dev, skb, NULL) % dev->real_num_tx_queues;
             ^~~~~~~~
    /root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c: At top level:
    /root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:7926:26: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
      .ndo_select_queue     = cxgb_select_queue,
                              ^~~~~~~~~~~~~~~~~
    /root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:7926:26: note: (near initialization for ‘cxgb4_netdev_ops.ndo_select_queue’)
    /root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c: In function ‘init_one’:
    /root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.c:9092:2: error: implicit declaration of function ‘pcie_print_link_status’ [-Werror=implicit-function-declaration]
      pcie_print_link_status(adapter->pdev);
      ^~~~~~~~~~~~~~~~~~~~~~
    cc1: some warnings being treated as errors
    scripts/Makefile.build:322: recipe for target '/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.o' failed
    make[5]: *** [/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4/cxgb4_main.o] Error 1
    Makefile:1552: recipe for target '_module_/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4' failed
    make[4]: *** [_module_/root/ChelsioUwire-3.11.0.0/build/src/network/cxgb4] Error 2
    Makefile:86: recipe for target 'build' failed
    make[3]: *** [build] Error 2
    Makefile:1261: recipe for target 'cxgb4' failed
    make[2]: *** [cxgb4] Error 2
     
  6. levifig

    levifig New Member

    Joined:
    Feb 13, 2018
    Messages:
    13
    Likes Received:
    0
    Does the UnifiedWire even install on Proxmox 6? I can't seem to have it recognize the headers, and I wonder if it's just not ready for kernel 5.x… :\
     
  7. tman222

    tman222 New Member

    Joined:
    Apr 13, 2019
    Messages:
    6
    Likes Received:
    0
    Hi joachim - My apologies, I just noticed that I made a mistake in my prior post. It was actually the Chelsio 3.10 drivers that I was using and installed, not the 3.11'.s The 3.10's compile fine under the current version of Promox, but when I tried to compile the 3.11's recently I got the same errors you did. I also managed to get DKMS to work recently - I'd be happy share my notes on that if there is interest.
     
    #7 tman222, Jul 21, 2019
    Last edited: Jul 21, 2019
  8. tman222

    tman222 New Member

    Joined:
    Apr 13, 2019
    Messages:
    6
    Likes Received:
    0
    Hi levifig -- how are you trying to install the drivers? Are you compiling from source, following the Python install script, or trying to use DKMS?
     
  9. levifig

    levifig New Member

    Joined:
    Feb 13, 2018
    Messages:
    13
    Likes Received:
    0
    Downloaded the drivers, unpacked the tar.gz, ran ./install.py. When it reached the part where it needed the "kernel source", I installed the pve-headers package for my current kernel, and it still seemed to fail to find the sources… :\

    Code:
     Building for kernel 5.0.15-1-pve                                                                                                                         │
     │ /bin/bash: rpm: command not found                                                                                                                        │
     │ ##################################################                                                                                                       │
     │ #          Cleaning Source/Build                 #                                                                                                       │
     │ ##################################################                                                                                                       │
     │ Building for kernel 5.0.15-1-pve                                                                                                                         │
     │                                                                                                                                                          │
     │ /bin/bash: rpm: command not found                                                                                                                        │
     │ ##################################################                                                                                                       │
     │ #          Cleaning Source/Build/RPMDir          #                                                                                                       │
     │ ##################################################                                                                                                       │
     │ Building for kernel 5.0.15-1-pve                                                                                                                         │
     │ Makefile:620: When using KOBJ=<path>, the KSRC=<path> must also be defined.                                                                              │
     │ Makefile:621: Use KDIR=<path> when KSRC and KOBJ are the same.                                                                                           │
     │ Makefile:622: *** ERROR: kernel source path not specified.  Stop.                                                                                        │
     │ make: *** [Makefile:4585: removeallPrevious] Error 1

    I also find it odd that it's looking for the rpm package, but then again: it could be trying to install the kernel sources... :\
     
  10. tman222

    tman222 New Member

    Joined:
    Apr 13, 2019
    Messages:
    6
    Likes Received:
    0
    Just to confirm, did you set Debian = 1 in the Makefile?
     
  11. levifig

    levifig New Member

    Joined:
    Feb 13, 2018
    Messages:
    13
    Likes Received:
    0
    Holy heck! I scoured the manual and couldn't find a single pointer about the debrules folder and how to use it instead!!

    It's installed now! THANK YOU!! :eek:

    PS: I remember glossing over a DEBIAN variable on this thread, but apparently I kept looking in the install.py not the Makefile... #sillyme :X
     
  12. tman222

    tman222 New Member

    Joined:
    Apr 13, 2019
    Messages:
    6
    Likes Received:
    0
    That's great news! Out of curiosity, which version of the Unified Wire software drivers did you install - was it the 3.10's or 3.11's? I'm currently running Proxmox 5.4 and the 3.11's throw some compile errors though the 3.10's work fine (see above). Would be great if the 3.11's now worked worked fine with Proxmox 6.0 though.
     
  13. levifig

    levifig New Member

    Joined:
    Feb 13, 2018
    Messages:
    13
    Likes Received:
    0
    ChelsioUwire-3.11.0.0 was what I used. :)
     
  14. tman222

    tman222 New Member

    Joined:
    Apr 13, 2019
    Messages:
    6
    Likes Received:
    0
    Decided to upgrade my my Proxmox box from 5.4 to 6.0 recently and was surprised that after the upgrade the Chelsio card was still recognized and working fine. At first I thought that the 3.10 driver based DKMS module that I had working and built under Proxmox 5.4 had just been compiled for Proxmox 6.0 and 5.x kernel. However, after further investigation it turns out that during the upgrade the cxgb4 module out of the 5.x kernel tree was actually used instead (vs. the DKMS module that been built from the 3.10 drivers). So now I'm a bit perplexed why the card was not recognized under Proxmox 5. Could it be simply that because the module was running already during the upgrade from Proxmox 5 to 6 the installer recognized the card and installed the cxgb4 module out of the 5.x kernel tree? Or maybe has all this been fixed with Proxmox 6 and there is no longer a need to manually install the Chelsio drivers? Thanks in advance for any input.
     
  15. M K

    M K New Member

    Joined:
    Mar 21, 2019
    Messages:
    1
    Likes Received:
    0
    This works. Was able to get the Unified Wire 3.11 drivers installed on Proxmox 6.
    DKMS is an issue however as updating the kernel required me to install the Unified Wire drivers again.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice