Setup with Intel 10 Gigabit X710-DA2 SFP+ Dual Port

hape

Renowned Member
Jun 10, 2013
70
3
73
Hello all,

we want to setup two proxmox 4.4 installations with a "Intel 10 Gigabit X710-DA2 SFP+ Dual Port" for a CEPH Network on supermicro mainboards. The two ports are visible but if i connect them with a 10g switch the link won't come up.

Do i need a special driver for that card?

What am i doing wrong?

Do i need special tools to setup the 10g-nic?

Regards

Hans-Peter Straub
 
I have the same NIC in our blades and they are working out of the box.

Code:
root@pmxn1:~# lspci | grep Eth
01:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
01:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
02:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
02:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
03:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
03:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
40:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
40:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
 
I have the same NIC in our blades and they are working out of the box.

Code:
root@pmxn1:~# lspci | grep Eth
01:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
01:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
02:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
02:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
03:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
03:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
40:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
40:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)


What kind of SFP+-Modules (manufacturer) have you installed on the card?
 
OK i've ordered now original INTEL "E10GSFPSR", to be sure the modules are compatible.
 
Maybe i find some time tomorrow and can test the x710 with some SFP+. But i need to check which SFPs i have for testing in the lab.

I give you an update tomorrow.

bg
Gregor
 
Now i've installed INTELs "E10GSFPSR". The link is now going up, but i cannot access this interface. When i show the statistics on this interface i see, that there is not 1 byte going out on this NIC ->(eth3).

------------------------
eth0 Link encap:Ethernet HWaddr 0c:c4:7a:ab:61:22
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1642 errors:0 dropped:0 overruns:0 frame:0
TX packets:929 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:478554 (467.3 KiB) TX bytes:1131722 (1.0 MiB)

eth3 Link encap:Ethernet HWaddr 3c:fd:fe:a3:02:85
inet addr:10.81.0.10 Bcast:10.81.0.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:10 errors:0 dropped:0 overruns:0 frame:0
TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:2778 (2.7 KiB) TX bytes:2778 (2.7 KiB)

vmbr0 Link encap:Ethernet HWaddr 0c:c4:7a:ab:61:22
inet addr:172.16.224.186 Bcast:172.16.224.255 Mask:255.255.255.0
inet6 addr: fe80::ec4:7aff:feab:6122/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1639 errors:0 dropped:0 overruns:0 frame:0
TX packets:363 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:455259 (444.5 KiB) TX bytes:1094366 (1.0 MiB)
------------------------

"ethtool -i eth3" shows the following:

------------------------
root@pmx-mm-01:~# ethtool -i eth3
driver: i40e
version: 1.4.25-k
firmware-version: 5.04 0x80002530 0.0.0
bus-info: 0000:02:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
------------------------

What is wrong with the Setup of this 10G-NIC?

I have this problem on two machines at the same time, a hardware-defect is not such realistic.

Regards

Hans-Peter
 
The link is showed with "state DOWN" in ip link. Why?

Here is the output of "ip link":

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop portid 3cfdfea30284 state DOWN mode DEFAULT group default qlen 1000
link/ether 3c:fd:fe:a3:02:84 brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:ab:61:22 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:ab:61:23 brd ff:ff:ff:ff:ff:ff
5: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq portid 3cfdfea30285 state DOWN mode DEFAULT group default qlen 1000
link/ether 3c:fd:fe:a3:02:85 brd ff:ff:ff:ff:ff:ff
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:ab:61:22 brd ff:ff:ff:ff:ff:ff
 
I have the same NIC in our blades and they are working out of the box.

Code:
root@pmxn1:~# lspci | grep Eth
01:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
01:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
02:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
02:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
03:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
03:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Backplane Connection (rev 02)
40:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
40:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)


Have you done a special setup of the card, and what type of switch do you have?
 
Arista 7150s a real nice ulta low latency switch.

edit:
as i see i have an older fw version but this should not be a problem:

Code:
root@pmxn1:~# ethtool -i eth3
driver: i40e
version: 1.4.25-k
firmware-version: 4.53 0x80001dc0 0.0.0
bus-info: 0000:40:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

do you see anything in the switch?

maybe spanning tree?
 
Last edited:
It looks that this cards doesn't work with the current Proxmox-Version and the used Supermicro Mainboards.

Does anybody knows a functional 10G DualPort 10G SFP+ NIC that works well with Proxmox 4.4-xx?
 
Ah i think you need to check the bios settings there is an option i had to set with i think x520 maybe the same for x710

I think this was map memory above 4g
 
After all other tries failed, my last attempt to get the NICs working is to compile the newest driver from intel (1.6.24).

When i start the make in the src directory i get the following error:

----------------------------
root@pmx-mm-01:/usr/src/i40e-1.6.42/src# make
make[1]: Entering directory '/usr/src/linux-headers-4.4.40-1-pve'
CC [M] /usr/src/i40e-1.6.42/src/i40e_main.o
In file included from /usr/src/i40e-1.6.42/src/i40e.h:52:0,
from /usr/src/i40e-1.6.42/src/i40e_main.c:25:
/usr/src/i40e-1.6.42/src/kcompat.h:5277:20: error: static declaration of ‘napi_consume_skb’ follows non-static declaration
static inline void napi_consume_skb(struct sk_buff *skb,
^
In file included from include/linux/tcp.h:21:0,
from include/net/tcp.h:24,
from /usr/src/i40e-1.6.42/src/i40e.h:27,
from /usr/src/i40e-1.6.42/src/i40e_main.c:25:
include/linux/skbuff.h:2346:6: note: previous declaration of ‘napi_consume_skb’ was here
void napi_consume_skb(struct sk_buff *skb, int budget);
^
scripts/Makefile.build:258: recipe for target '/usr/src/i40e-1.6.42/src/i40e_main.o' failed
make[2]: *** [/usr/src/i40e-1.6.42/src/i40e_main.o] Error 1
Makefile:1420: recipe for target '_module_/usr/src/i40e-1.6.42/src' failed
make[1]: *** [_module_/usr/src/i40e-1.6.42/src] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-4.4.40-1-pve'
Makefile:69: recipe for target 'default' failed
make: *** [default] Error 2
----------------------------

Does anybody have any idea?
 
Did you ever get the cards working? I am considering using the SuperMicro AOC-STG-b4S 4port 10GbaseT, Intel XL710 and X557 NIC's and want to know if it will work before we purchase.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!