Setup with Intel 10 Gigabit X710-DA2 SFP+ Dual Port

H

hape

Renowned Member
Jun 10, 2013
75
5
73
Hello all,

we want to setup two proxmox 4.4 installations with a "Intel 10 Gigabit X710-DA2 SFP+ Dual Port" for a CEPH Network on supermicro mainboards. The two ports are visible but if i connect them with a 10g switch the link won't come up.

Do i need a special driver for that card?

What am i doing wrong?

Do i need special tools to setup the 10g-nic?

Regards

Hans-Peter Straub
 
I have the same NIC in our blades and they are working out of the box.

Code: 
root@pmxn1:~# lspci | grep Eth
01:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
01:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
02:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
02:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
03:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
03:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
40:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
40:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
 
hec said:
I have the same NIC in our blades and they are working out of the box.

Code: 
root@pmxn1:~# lspci | grep Eth
01:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
01:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
02:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
02:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
03:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
03:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
40:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
40:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
Click to expand...


What kind of SFP+-Modules (manufacturer) have you installed on the card?
 
OK i've ordered now original INTEL "E10GSFPSR", to be sure the modules are compatible.
 
Maybe i find some time tomorrow and can test the x710 with some SFP+. But i need to check which SFPs i have for testing in the lab.

I give you an update tomorrow.

bg
Gregor
 
Now i've installed INTELs "E10GSFPSR". The link is now going up, but i cannot access this interface. When i show the statistics on this interface i see, that there is not 1 byte going out on this NIC ->(eth3).

------------------------
eth0 Link encap:Ethernet HWaddr 0c:c4:7a:ab:61:22
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1642 errors:0 dropped:0 overruns:0 frame:0
TX packets:929 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:478554 (467.3 KiB) TX bytes:1131722 (1.0 MiB)

eth3 Link encap:Ethernet HWaddr 3c:fd:fe:a3:02:85
inet addr:10.81.0.10 Bcast:10.81.0.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:10 errors:0 dropped:0 overruns:0 frame:0
TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:2778 (2.7 KiB) TX bytes:2778 (2.7 KiB)

vmbr0 Link encap:Ethernet HWaddr 0c:c4:7a:ab:61:22
inet addr:172.16.224.186 Bcast:172.16.224.255 Mask:255.255.255.0
inet6 addr: fe80::ec4:7aff:feab:6122/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1639 errors:0 dropped:0 overruns:0 frame:0
TX packets:363 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:455259 (444.5 KiB) TX bytes:1094366 (1.0 MiB)
------------------------

"ethtool -i eth3" shows the following:

------------------------
root@pmx-mm-01:~# ethtool -i eth3
driver: i40e
version: 1.4.25-k
firmware-version: 5.04 0x80002530 0.0.0
bus-info: 0000:02:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
------------------------

What is wrong with the Setup of this 10G-NIC?

I have this problem on two machines at the same time, a hardware-defect is not such realistic.

Regards

Hans-Peter
 
The link is showed with "state DOWN" in ip link. Why?

Here is the output of "ip link":

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop portid 3cfdfea30284 state DOWN mode DEFAULT group default qlen 1000
link/ether 3c:fd:fe:a3:02:84 brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:ab:61:22 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:ab:61:23 brd ff:ff:ff:ff:ff:ff
5: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq portid 3cfdfea30285 state DOWN mode DEFAULT group default qlen 1000
link/ether 3c:fd:fe:a3:02:85 brd ff:ff:ff:ff:ff:ff
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:ab:61:22 brd ff:ff:ff:ff:ff:ff
 
hec said:
I have the same NIC in our blades and they are working out of the box.

Code: 
root@pmxn1:~# lspci | grep Eth
01:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
01:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
02:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
02:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
03:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
03:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
40:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
40:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
Click to expand...


Have you done a special setup of the card, and what type of switch do you have?
 
Arista 7150s a real nice ulta low latency switch.

edit:
as i see i have an older fw version but this should not be a problem:

Code: 
root@pmxn1:~# ethtool -i eth3
driver: i40e
version: 1.4.25-k
firmware-version: 4.53 0x80001dc0 0.0.0
bus-info: 0000:40:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

do you see anything in the switch?

maybe spanning tree?
 
Last edited:
It looks that this cards doesn't work with the current Proxmox-Version and the used Supermicro Mainboards.

Does anybody knows a functional 10G DualPort 10G SFP+ NIC that works well with Proxmox 4.4-xx?
 
Ah i think you need to check the bios settings there is an option i had to set with i think x520 maybe the same for x710

I think this was map memory above 4g
 
After all other tries failed, my last attempt to get the NICs working is to compile the newest driver from intel (1.6.24).

When i start the make in the src directory i get the following error:

----------------------------
root@pmx-mm-01:/usr/src/i40e-1.6.42/src# make
make[1]: Entering directory '/usr/src/linux-headers-4.4.40-1-pve'
CC [M] /usr/src/i40e-1.6.42/src/i40e_main.o
In file included from /usr/src/i40e-1.6.42/src/i40e.h:52:0,
from /usr/src/i40e-1.6.42/src/i40e_main.c:25:
/usr/src/i40e-1.6.42/src/kcompat.h:5277:20: error: static declaration of ‘napi_consume_skb’ follows non-static declaration
static inline void napi_consume_skb(struct sk_buff *skb,
^
In file included from include/linux/tcp.h:21:0,
from include/net/tcp.h:24,
from /usr/src/i40e-1.6.42/src/i40e.h:27,
from /usr/src/i40e-1.6.42/src/i40e_main.c:25:
include/linux/skbuff.h:2346:6: note: previous declaration of ‘napi_consume_skb’ was here
void napi_consume_skb(struct sk_buff *skb, int budget);
^
scripts/Makefile.build:258: recipe for target '/usr/src/i40e-1.6.42/src/i40e_main.o' failed
make[2]: *** [/usr/src/i40e-1.6.42/src/i40e_main.o] Error 1
Makefile:1420: recipe for target '_module_/usr/src/i40e-1.6.42/src' failed
make[1]: *** [_module_/usr/src/i40e-1.6.42/src] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-4.4.40-1-pve'
Makefile:69: recipe for target 'default' failed
make: *** [default] Error 2
----------------------------

Does anybody have any idea?
 
Did you ever get the cards working? I am considering using the SuperMicro AOC-STG-b4S 4port 10GbaseT, Intel XL710 and X557 NIC's and want to know if it will work before we purchase.
 
You must log in or register to reply here.
Top Bottom