Dual SFP+ Card only showing 1 Nic

Discussion in 'Proxmox VE: Networking and Firewall' started by 1nerdyguy, Aug 31, 2017.

  1. 1nerdyguy

    1nerdyguy Member

    Joined:
    Apr 17, 2014
    Messages:
    118
    Likes Received:
    2
    Just installed some Chelsio cards, dual 10GB SFP. When I log into the GUI, I only see 1 new nic called enp66s0 to configure.

    Logging into the console and doin ga lspci | grep 'Ethernet' shows my 4 onboard gigabits, and 1 "Chelsio Communications Inc T320 10GbE Dual Port Adapter"

    What am i missing to be able to use both ports?

    EDIT:
    Think I figured out part of the problem. The Chelsio T3 appears to be using the wrong kernel driver, which is only enabling 1 port.

    The correct driver doesn't want to compile on a modern kernel, but there is a pre-compiled back in jessie's non-free packages

    but that package conflicts with the PVE-Firmware package...any suggestions?
     
    #1 1nerdyguy, Aug 31, 2017
    Last edited: Aug 31, 2017
  2. TheDevouringOne

    TheDevouringOne New Member

    Joined:
    Aug 21, 2017
    Messages:
    9
    Likes Received:
    0
    I am having the same issue tried rebuilding the cluster from the ground up to address this and no luck =(
     
  3. 1nerdyguy

    1nerdyguy Member

    Joined:
    Apr 17, 2014
    Messages:
    118
    Likes Received:
    2
    Got it working by installing bare debian, installing the non-free-firmware package, and then installing proxmox on top of it.
     
  4. TheDevouringOne

    TheDevouringOne New Member

    Joined:
    Aug 21, 2017
    Messages:
    9
    Likes Received:
    0
    #4 TheDevouringOne, Sep 2, 2017
    Last edited: Sep 2, 2017
  5. Clive Austin

    Clive Austin New Member
    Proxmox VE Subscriber

    Joined:
    Mar 14, 2017
    Messages:
    8
    Likes Received:
    0
    +1 ... I'm having exactly the same issue, but can't shed any more light I'm afraid! I wonder if we should report it as a "bug" and hope for a kernel update?
     
  6. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,129
    Likes Received:
    478
    if you provide some more details (e.g., which driver/module with which firmware in which version works) we can try to find out whether it can be included/ported to 4.10
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. Clive Austin

    Clive Austin New Member
    Proxmox VE Subscriber

    Joined:
    Mar 14, 2017
    Messages:
    8
    Likes Received:
    0
    Hi Fabian,

    Thanks for reaching out.... what extra info can I provide?

    I'm running Proxmox Virtual Environment 5.0-31 (Debian 9)

    I have installed 3 x Chelsio N320E T320 dual-port SFP+ 10Gb cards. One card has been installed in each of my two Proxmox servers and one in my OpenNAS server. The OpenNAS server recognises both ports, but each of my Proxmox servers only recognise one port on these cards.

    If I run "lspci" on the Proxmox servers, we see that the card seems to recognised...

    # lspci | grep -i chelsio
    0c:00.0 Ethernet controller: Chelsio Communications Inc T320 10GbE Dual Port Adapter​

    If I run "ifconfig" on the Proxmox servers though, it only shows a single interface...

    # ifconfig | grep ens
    ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
    ether 00:07:43:0a:27:e2 txqueuelen 1000 (Ethernet)
    RX packets 69478 bytes 7740606 (7.3 MiB)
    RX errors 0 dropped 0 overruns 0 frame 0
    TX packets 64346 bytes 6528292 (6.2 MiB)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
    device interrupt 31 memory 0xfdff0000-fdff0fff​

    Running "ifconfig" on my FreeNAS server shows both interfaces...

    # ifconfig | grep cxgb
    cxgb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6c03bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:07:43:0a:ff:8e
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet 10Gbase-Twinax <full-duplex>
    status: active
    cxgb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6c03bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:07:43:0a:ff:8f
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet 10Gbase-Twinax <full-duplex>
    status: active​

    If I run "lsmod" on the Proxmox servers I can see that seem to have loaded the cxgb3 and mdio kernel modules...

    # lsmod | grep cx
    cxgb3 151552 0
    mdio 16384 1 cxgb3
    I'm certainly not an expert here, but it feels as though Proxmox may have incorrectly identified the card (as single-port h/w) and loaded the wrong kernel module?​

    FWIW, these are the (file) references to cxgb3 that I've found on the system...

    #locate cxgb3
    /lib/firmware/cxgb3
    /lib/firmware/cxgb3/ael2005_opt_edc.bin
    /lib/firmware/cxgb3/ael2005_twx_edc.bin
    /lib/firmware/cxgb3/ael2020_twx_edc.bin
    /lib/firmware/cxgb3/t3b_psram-1.1.0.bin
    /lib/firmware/cxgb3/t3c_psram-1.1.0.bin
    /lib/firmware/cxgb3/t3fw-7.10.0.bin
    /lib/firmware/cxgb3/t3fw-7.12.0.bin
    /lib/firmware/cxgb3/t3fw-7.4.0.bin
    /lib/modules/4.10.15-1-pve/kernel/drivers/infiniband/hw/cxgb3
    /lib/modules/4.10.15-1-pve/kernel/drivers/infiniband/hw/cxgb3/iw_cxgb3.ko
    /lib/modules/4.10.15-1-pve/kernel/drivers/net/ethernet/chelsio/cxgb3
    /lib/modules/4.10.15-1-pve/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko
    /lib/modules/4.10.15-1-pve/kernel/drivers/scsi/cxgbi/cxgb3i
    /lib/modules/4.10.15-1-pve/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko
    /lib/modules/4.10.17-2-pve/kernel/drivers/infiniband/hw/cxgb3
    /lib/modules/4.10.17-2-pve/kernel/drivers/infiniband/hw/cxgb3/iw_cxgb3.ko
    /lib/modules/4.10.17-2-pve/kernel/drivers/net/ethernet/chelsio/cxgb3
    /lib/modules/4.10.17-2-pve/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko
    /lib/modules/4.10.17-2-pve/kernel/drivers/scsi/cxgbi/cxgb3i
    /lib/modules/4.10.17-2-pve/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko
    /lib/modules/4.10.17-3-pve/kernel/drivers/infiniband/hw/cxgb3
    /lib/modules/4.10.17-3-pve/kernel/drivers/infiniband/hw/cxgb3/iw_cxgb3.ko
    /lib/modules/4.10.17-3-pve/kernel/drivers/net/ethernet/chelsio/cxgb3
    /lib/modules/4.10.17-3-pve/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko
    /lib/modules/4.10.17-3-pve/kernel/drivers/scsi/cxgbi/cxgb3i
    /lib/modules/4.10.17-3-pve/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko
    /usr/include/rdma/cxgb3-abi.h​

    Hope that helps ... please let me know if I can do anything more to assist in any way...

    Best Regards,
     
  8. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,129
    Likes Received:
    478
    does the log contain any lines mentioning cxgb ? if so, please post them.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. Clive Austin

    Clive Austin New Member
    Proxmox VE Subscriber

    Joined:
    Mar 14, 2017
    Messages:
    8
    Likes Received:
    0
    Hi,
    Ok, I've just loked in messages, syslog and kern...

    # grep cxgb /var/log/messages


    /var/log/messages:Sep 25 12:37:40 pm28 kernel: [137856.307353] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

    /var/log/messages:Sep 25 12:58:55 pm28 kernel: [139131.148561] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

    /var/log/messages:Sep 25 14:05:58 pm28 kernel: [143153.339167] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

    /var/log/messages:Sep 25 15:23:23 pm28 kernel: [147798.652869] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

    /var/log/messages:Sep 26 00:26:28 pm28 kernel: [180381.479858] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

    /var/log/messages:Sep 26 06:11:40 pm28 kernel: [201092.095015] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

    /var/log/messages:Sep 26 08:21:26 pm28 kernel: [208877.310810] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

    /var/log/messages:Sep 26 09:46:44 pm28 kernel: [213994.841538] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes
    # grep cxgb /var/log/syslog

    /var/log/syslog:Sep 26 08:21:26 pm28 kernel: [208877.310810] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

    /var/log/syslog:Sep 26 09:46:44 pm28 kernel: [213994.841538] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes
    # grep cxgb /var/log/kern.log

    /var/log/kern.log:Sep 25 12:37:40 pm28 kernel: [137856.307353] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

    /var/log/kern.log:Sep 25 12:58:55 pm28 kernel: [139131.148561] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

    /var/log/kern.log:Sep 25 14:05:58 pm28 kernel: [143153.339167] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

    /var/log/kern.log:Sep 25 15:23:23 pm28 kernel: [147798.652869] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

    /var/log/kern.log:Sep 26 00:26:28 pm28 kernel: [180381.479858] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

    /var/log/kern.log:Sep 26 06:11:40 pm28 kernel: [201092.095015] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

    /var/log/kern.log:Sep 26 08:21:26 pm28 kernel: [208877.310810] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

    /var/log/kern.log:Sep 26 09:46:44 pm28 kernel: [213994.841538] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes​
     
  10. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,129
    Likes Received:
    478
    that does not help unfortunately. if you find a module that does work as expected, please give the exact version so we can investigate further (or contact support of the vendor).
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. streat90

    streat90 New Member

    Joined:
    Oct 5, 2017
    Messages:
    2
    Likes Received:
    0
    Did we ever get this working?
     
  12. Clive Austin

    Clive Austin New Member
    Proxmox VE Subscriber

    Joined:
    Mar 14, 2017
    Messages:
    8
    Likes Received:
    0
    No it's still an issue... but it's just been over-taken by other (more urgent) issues so I haven't looked at it for a while. Interestingly though, one of the most annoying things is that after a re-boot the server seems to arbitrarily re-assign ens5 to the other of the 2 interfaces on the card so that I have to pull the module and plug in to the other SFP+ socket from time-to-time. Go figure! At least it proves both ports are working...
     
  13. lastb0isct

    lastb0isct Member

    Joined:
    Dec 29, 2015
    Messages:
    61
    Likes Received:
    0
    I'm seeing this issue as well on three of my machines. Is there a way to get the chelsio driver updated?

    I was thinking of trying to compile the driver from source and installing it on one of the proxmox servers but not sure if that will resolve the issue...any advice would be welcomed!
     
  14. lastb0isct

    lastb0isct Member

    Joined:
    Dec 29, 2015
    Messages:
    61
    Likes Received:
    0
    What do you mean by non-free-firmware package? Do you have that available?
     
  15. Donitoza

    Donitoza New Member

    Joined:
    Jan 4, 2018
    Messages:
    8
    Likes Received:
    0
    It`s helped me (Proxmox 5.1):

    find link named as "renameX"
    # ip link list
    ....
    2: rename3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 00:07:43:0a:7b:60 brd ff:ff:ff:ff:ff:ff
    3: ens1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 00:07:43:0a:7b:61 brd ff:ff:ff:ff:ff:ff
    ....


    copy MACs and add in /etc/udev/rules.d/10-persistent-network.rules (create if needed):

    # cat >> /etc/udev/rules.d/10-persistent-network.rules
    SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:07:43:0a:7b:60", NAME="ens1s0"
    SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:07:43:0a:7b:61", NAME="ens1s1"


    then change interface name in /etc/network/interfaces

    reboot
     
  16. Bahjons

    Bahjons New Member

    Joined:
    Aug 8, 2018
    Messages:
    2
    Likes Received:
    0
    Anyone find a fix for the T320 cards? I have had them running fine on Proxmox 4.4, but having issues with them seeing the cable on Proxmox 5.2.
     
  17. Bahjons

    Bahjons New Member

    Joined:
    Aug 8, 2018
    Messages:
    2
    Likes Received:
    0
    I just booted to a non-PVE kernel (4.9.0-7-amd64) and it sees the link on the T320 and working fine with both interfaces. Soon as I boot to PVE Kernel (4.15.18-1-pve) I'm back to no link on either interface. Seems to me the PVE Kernel is missing something...
     
  18. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,129
    Likes Received:
    478
    please provide "modinfo cxgb" output for both working and non-working kernels - maybe we can find something starting there.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice