Dual SFP+ Card only showing 1 Nic

1

1nerdyguy

Active Member
Apr 17, 2014
119
2
38
Just installed some Chelsio cards, dual 10GB SFP. When I log into the GUI, I only see 1 new nic called enp66s0 to configure.

Logging into the console and doin ga lspci | grep 'Ethernet' shows my 4 onboard gigabits, and 1 "Chelsio Communications Inc T320 10GbE Dual Port Adapter"

What am i missing to be able to use both ports?

EDIT:
Think I figured out part of the problem. The Chelsio T3 appears to be using the wrong kernel driver, which is only enabling 1 port.

The correct driver doesn't want to compile on a modern kernel, but there is a pre-compiled back in jessie's non-free packages

but that package conflicts with the PVE-Firmware package...any suggestions?
 
Last edited:
Got it working by installing bare debian, installing the non-free-firmware package, and then installing proxmox on top of it.
 
+1 ... I'm having exactly the same issue, but can't shed any more light I'm afraid! I wonder if we should report it as a "bug" and hope for a kernel update?
 
if you provide some more details (e.g., which driver/module with which firmware in which version works) we can try to find out whether it can be included/ported to 4.10
 
Hi Fabian,

Thanks for reaching out.... what extra info can I provide?

I'm running Proxmox Virtual Environment 5.0-31 (Debian 9)

I have installed 3 x Chelsio N320E T320 dual-port SFP+ 10Gb cards. One card has been installed in each of my two Proxmox servers and one in my OpenNAS server. The OpenNAS server recognises both ports, but each of my Proxmox servers only recognise one port on these cards.

If I run "lspci" on the Proxmox servers, we see that the card seems to recognised...

# lspci | grep -i chelsio
0c:00.0 Ethernet controller: Chelsio Communications Inc T320 10GbE Dual Port Adapter​

If I run "ifconfig" on the Proxmox servers though, it only shows a single interface...

# ifconfig | grep ens
ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 00:07:43:0a:27:e2 txqueuelen 1000 (Ethernet)
RX packets 69478 bytes 7740606 (7.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 64346 bytes 6528292 (6.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 31 memory 0xfdff0000-fdff0fff​

Running "ifconfig" on my FreeNAS server shows both interfaces...

# ifconfig | grep cxgb
cxgb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6c03bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:07:43:0a:ff:8e
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet 10Gbase-Twinax <full-duplex>
status: active
cxgb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6c03bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:07:43:0a:ff:8f
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet 10Gbase-Twinax <full-duplex>
status: active​

If I run "lsmod" on the Proxmox servers I can see that seem to have loaded the cxgb3 and mdio kernel modules...

# lsmod | grep cx
cxgb3 151552 0
mdio 16384 1 cxgb3
I'm certainly not an expert here, but it feels as though Proxmox may have incorrectly identified the card (as single-port h/w) and loaded the wrong kernel module?​

FWIW, these are the (file) references to cxgb3 that I've found on the system...

#locate cxgb3
/lib/firmware/cxgb3
/lib/firmware/cxgb3/ael2005_opt_edc.bin
/lib/firmware/cxgb3/ael2005_twx_edc.bin
/lib/firmware/cxgb3/ael2020_twx_edc.bin
/lib/firmware/cxgb3/t3b_psram-1.1.0.bin
/lib/firmware/cxgb3/t3c_psram-1.1.0.bin
/lib/firmware/cxgb3/t3fw-7.10.0.bin
/lib/firmware/cxgb3/t3fw-7.12.0.bin
/lib/firmware/cxgb3/t3fw-7.4.0.bin
/lib/modules/4.10.15-1-pve/kernel/drivers/infiniband/hw/cxgb3
/lib/modules/4.10.15-1-pve/kernel/drivers/infiniband/hw/cxgb3/iw_cxgb3.ko
/lib/modules/4.10.15-1-pve/kernel/drivers/net/ethernet/chelsio/cxgb3
/lib/modules/4.10.15-1-pve/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko
/lib/modules/4.10.15-1-pve/kernel/drivers/scsi/cxgbi/cxgb3i
/lib/modules/4.10.15-1-pve/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko
/lib/modules/4.10.17-2-pve/kernel/drivers/infiniband/hw/cxgb3
/lib/modules/4.10.17-2-pve/kernel/drivers/infiniband/hw/cxgb3/iw_cxgb3.ko
/lib/modules/4.10.17-2-pve/kernel/drivers/net/ethernet/chelsio/cxgb3
/lib/modules/4.10.17-2-pve/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko
/lib/modules/4.10.17-2-pve/kernel/drivers/scsi/cxgbi/cxgb3i
/lib/modules/4.10.17-2-pve/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko
/lib/modules/4.10.17-3-pve/kernel/drivers/infiniband/hw/cxgb3
/lib/modules/4.10.17-3-pve/kernel/drivers/infiniband/hw/cxgb3/iw_cxgb3.ko
/lib/modules/4.10.17-3-pve/kernel/drivers/net/ethernet/chelsio/cxgb3
/lib/modules/4.10.17-3-pve/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko
/lib/modules/4.10.17-3-pve/kernel/drivers/scsi/cxgbi/cxgb3i
/lib/modules/4.10.17-3-pve/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko
/usr/include/rdma/cxgb3-abi.h​

Hope that helps ... please let me know if I can do anything more to assist in any way...

Best Regards,
 
does the log contain any lines mentioning cxgb ? if so, please post them.
 
Hi,
Ok, I've just loked in messages, syslog and kern...

# grep cxgb /var/log/messages

/var/log/messages:Sep 25 12:37:40 pm28 kernel: [137856.307353] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

/var/log/messages:Sep 25 12:58:55 pm28 kernel: [139131.148561] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

/var/log/messages:Sep 25 14:05:58 pm28 kernel: [143153.339167] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

/var/log/messages:Sep 25 15:23:23 pm28 kernel: [147798.652869] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

/var/log/messages:Sep 26 00:26:28 pm28 kernel: [180381.479858] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

/var/log/messages:Sep 26 06:11:40 pm28 kernel: [201092.095015] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

/var/log/messages:Sep 26 08:21:26 pm28 kernel: [208877.310810] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

/var/log/messages:Sep 26 09:46:44 pm28 kernel: [213994.841538] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes
​
# grep cxgb /var/log/syslog

/var/log/syslog:Sep 26 08:21:26 pm28 kernel: [208877.310810] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

/var/log/syslog:Sep 26 09:46:44 pm28 kernel: [213994.841538] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes
​
# grep cxgb /var/log/kern.log

/var/log/kern.log:Sep 25 12:37:40 pm28 kernel: [137856.307353] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

/var/log/kern.log:Sep 25 12:58:55 pm28 kernel: [139131.148561] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

/var/log/kern.log:Sep 25 14:05:58 pm28 kernel: [143153.339167] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

/var/log/kern.log:Sep 25 15:23:23 pm28 kernel: [147798.652869] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

/var/log/kern.log:Sep 26 00:26:28 pm28 kernel: [180381.479858] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

/var/log/kern.log:Sep 26 06:11:40 pm28 kernel: [201092.095015] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

/var/log/kern.log:Sep 26 08:21:26 pm28 kernel: [208877.310810] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes

/var/log/kern.log:Sep 26 09:46:44 pm28 kernel: [213994.841538] hid_generic usbkbd usbmouse usbhid hid psmouse pata_acpi i2c_piix4 pata_atiixp hpsa ahci cxgb3 libahci scsi_transport_sas bnx2 mdio fjes​
 
that does not help unfortunately. if you find a module that does work as expected, please give the exact version so we can investigate further (or contact support of the vendor).
 
No it's still an issue... but it's just been over-taken by other (more urgent) issues so I haven't looked at it for a while. Interestingly though, one of the most annoying things is that after a re-boot the server seems to arbitrarily re-assign ens5 to the other of the 2 interfaces on the card so that I have to pull the module and plug in to the other SFP+ socket from time-to-time. Go figure! At least it proves both ports are working...
 
I'm seeing this issue as well on three of my machines. Is there a way to get the chelsio driver updated?

I was thinking of trying to compile the driver from source and installing it on one of the proxmox servers but not sure if that will resolve the issue...any advice would be welcomed!
 
It`s helped me (Proxmox 5.1):

find link named as "renameX"
# ip link list
....
2: rename3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
link/ether 00:07:43:0a:7b:60 brd ff:ff:ff:ff:ff:ff
3: ens1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
link/ether 00:07:43:0a:7b:61 brd ff:ff:ff:ff:ff:ff
....

copy MACs and add in /etc/udev/rules.d/10-persistent-network.rules (create if needed):

# cat >> /etc/udev/rules.d/10-persistent-network.rules
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:07:43:0a:7b:60", NAME="ens1s0"
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:07:43:0a:7b:61", NAME="ens1s1"

then change interface name in /etc/network/interfaces

reboot
 
Anyone find a fix for the T320 cards? I have had them running fine on Proxmox 4.4, but having issues with them seeing the cable on Proxmox 5.2.
 
I just booted to a non-PVE kernel (4.9.0-7-amd64) and it sees the link on the T320 and working fine with both interfaces. Soon as I boot to PVE Kernel (4.15.18-1-pve) I'm back to no link on either interface. Seems to me the PVE Kernel is missing something...
 
please provide "modinfo cxgb" output for both working and non-working kernels - maybe we can find something starting there.
 
Im facing the same issue here, however I cant test this with an older (pve) kernel because then it will not import the ZFS pool

Atleast this is the modinfo from a non-working kernel
Code: 
root@americano:~# modinfo cxgb
filename:       /lib/modules/4.15.18-9-pve/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko
license:        GPL
author:         Chelsio Communications
description:    Chelsio 10Gb Ethernet Driver
srcversion:     A26CC06B1694A060A12DA3D
alias:          pci:v00001425d00000010sv*sd00000001bc*sc*i*
alias:          pci:v00001425d0000000Esv*sd00000001bc*sc*i*
alias:          pci:v00001425d0000000Bsv*sd00000001bc*sc*i*
alias:          pci:v00001425d0000000Asv*sd00000001bc*sc*i*
alias:          pci:v00001425d00000007sv*sd00000000bc*sc*i*
alias:          pci:v00001425d00000008sv*sd00000001bc*sc*i*
alias:          pci:v00001425d00000008sv*sd00000000bc*sc*i*
depends:        mdio
retpoline:      Y
intree:         Y
name:           cxgb
vermagic:       4.15.18-9-pve SMP mod_unload modversions
parm:           copybreak:Receive copy threshold (int)
parm:           dflt_msg_enable:Chelsio T1 default message enable bitmap (int)
parm:           t1powersave:Enable/Disable T1 powersaving mode (int)
parm:           disable_msi:Disable Message Signaled Interrupt (MSI) (int)
 
Hi,
same problem here. I've got a T320 card and the server has the 4.15.18-9-pve kernel.
I've tried renaming the interfaces to cxgb0 and cxgb1 by creating a file at "/etc/udev/rules.d/" and modifying the vmbr0 configuration at /etc/network/interfaces accordingly.
After the system reboots I can access to it via the ip address of vmbr0 (bridge port cxgb0) but I cannot start any VM using that bridge and I get the following error:
Code: 
no physical interface on bridge 'vmbr0'
kvm: network script /var/lib/qemu-server/pve-bridge failed with status 6400
I've also noticed at the web interface (System->Network) that both ports are listed as "Unknown" type, instead of "Network device".
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom