Kernel 4.15.18-9 incompatible with 10 Gbps card
- Thread starter tuantv
- Start date
tuantv
I have no such issue
I have no such issue
Code:
root@pve3:~# pveversion -v
proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-5 (running version: 5.3-5/97ae681d)
pve-kernel-4.15: 5.2-12
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-7-pve: 4.15.18-27
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-3-pve: 4.15.17-14
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph: 12.2.8-pve1
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-33
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-5
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
openvswitch-switch: 2.7.0-3
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-31
pve-container: 2.0-31
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-5
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-43
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1
Here are the parameters on my proxmox, please support.
Code:
root@node03:~# ethtool -i enp4s0f1 | grep firmware-version
firmware-version: 0x30030001
root@node03:~# lspci | grep SFP
04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
root@node03:~# ip r
default via 103.254.12.1 dev eno1 onlink
10.200.0.0/16 dev enp4s0f1 proto kernel scope link src 10.200.23.254 linkdown
103.254.12.0/25 dev eno1 proto kernel scope link src 103.254.12.23
103.254.13.0/25 dev vmbr0 proto kernel scope link src 103.254.13.93
103.254.14.0/25 dev vmbr1 proto kernel scope link src 103.254.14.93
192.168.108.0/24 dev eno2 proto kernel scope link src 192.168.108.13
root@node03:~# pveversion -v
proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-5 (running version: 5.3-5/97ae681d)
pve-kernel-4.15: 5.2-12
pve-kernel-4.15.18-9-pve: 4.15.18-30
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-33
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-5
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-31
pve-container: 2.0-31
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-5
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-43
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1
root@node03:~# dme
dmesg dmeventd
root@node03:~# dmesg | grep enp4s0f1
[ 2.776473] ixgbe 0000:04:00.1 enp4s0f1: renamed from eth1
[ 14.308918] ixgbe 0000:04:00.1: registered PHC device on enp4s0f1
[ 14.416792] IPv6: ADDRCONF(NETDEV_UP): enp4s0f1: link is not ready
[1538600.142842] ixgbe 0000:04:00.1: removed PHC on enp4s0f1
[1538653.591932] ixgbe 0000:04:00.1: registered PHC device on enp4s0f1
[1538653.699109] IPv6: ADDRCONF(NETDEV_UP): enp4s0f1: link is not ready
[1538888.554902] ixgbe 0000:04:00.1: removed PHC on enp4s0f1
[1538893.222490] ixgbe 0000:04:00.1: registered PHC device on enp4s0f1
[1538893.329832] IPv6: ADDRCONF(NETDEV_UP): enp4s0f1: link is not readysometimes the dmesg messages don't contain the interface-name (actually in that case IIRC the rename to predictable names is one of the last things that happens) - please post the complete output of dmesg (or at least some more context)
Please see info bellow for help.
Code:
auto lo
iface lo inet loopback
auto eno2
iface eno2 inet static
address 192.168.108.13
netmask 255.255.255.0
auto enp4s0f1
iface enp4s0f1 inet static
address 10.200.23.254
netmask 255.255.0.0
hwaddress ether 00:1b:21:8a:c6:79
auto eno1
iface eno1 inet static
address 103.254.12.23
netmask 255.255.255.128
gateway 103.254.12.1
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp
auto vmbr0
iface vmbr0 inet static
address 103.254.13.93
netmask 255.255.255.128
bridge-ports none
bridge-stp off
bridge-fd 0
auto vmbr1
iface vmbr1 inet static
address 103.254.14.93
netmask 255.255.255.128
bridge-ports none
bridge-stp off
bridge-fd 0
root@node03:~# ifdown enp4s0f1
root@node03:~# ifup enp4s0f1
root@node03:~# ip r
default via 103.254.12.1 dev eno1 onlink
10.200.0.0/16 dev enp4s0f1 proto kernel scope link src 10.200.23.254 linkdown
103.254.12.0/25 dev eno1 proto kernel scope link src 103.254.12.23
103.254.13.0/25 dev vmbr0 proto kernel scope link src 103.254.13.93
103.254.14.0/25 dev vmbr1 proto kernel scope link src 103.254.14.93
192.168.108.0/24 dev eno2 proto kernel scope link src 192.168.108.13hmm - nothing specific in the dmesg (basically looks like the NIC simply isn't plugged in):
* where is the nic plugged in? - does the switch/other side detect a link?
* is the other sided configured to have the interface up?
from the /etc/network/interfaces you posted:
* why do you need the `hwaddress ether` line for the NIC? - do you get a link if you remove that line?
`ethtool` can yield some helpful information - try to see if you can find an issue with it:
* ethtool $ifname
* ethtool -S $ifname
* ethtool -i $ifname
* ethtool --phy-statistics $ifname
should get you started
if all above doesn't resolve the issue:
the ixgbe module has a debug-parameter - maybe load the module with debug enabled - it might show some information
hope this helps!
* where is the nic plugged in? - does the switch/other side detect a link?
* is the other sided configured to have the interface up?
from the /etc/network/interfaces you posted:
* why do you need the `hwaddress ether` line for the NIC? - do you get a link if you remove that line?
`ethtool` can yield some helpful information - try to see if you can find an issue with it:
* ethtool $ifname
* ethtool -S $ifname
* ethtool -i $ifname
* ethtool --phy-statistics $ifname
should get you started
if all above doesn't resolve the issue:
the ixgbe module has a debug-parameter - maybe load the module with debug enabled - it might show some information
hope this helps!
* NIC is attached via PCI slot
* I have added hwaddress ether` line to the NIC, delete this line, the nic will still report linkdown.
See me for info as below with ethtool.
I have implemented 4 proxmox builds with 4 different 10Gb cards of this intel and all have linkdown errors. Can you support us via ssh? If possible, we can pay for support and subscribe to your products.
* I have added hwaddress ether` line to the NIC, delete this line, the nic will still report linkdown.
See me for info as below with ethtool.
I have implemented 4 proxmox builds with 4 different 10Gb cards of this intel and all have linkdown errors. Can you support us via ssh? If possible, we can pay for support and subscribe to your products.
Code:
root@node03:~# ethtool enp4s0f1
Settings for enp4s0f1:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Speed: Unknown!
Duplex: Unknown! (255)
Port: Other
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: no
root@node03:~# ethtool -S Link detected: no
ethtool: bad command line argument(s)
For more information run ethtool -h
root@node03:~# ethtool -S enp4s0f1
NIC statistics:
rx_packets: 0
tx_packets: 0
rx_bytes: 0
tx_bytes: 0
rx_pkts_nic: 0
tx_pkts_nic: 0
rx_bytes_nic: 0
tx_bytes_nic: 0
lsc_int: 0
tx_busy: 0
non_eop_descs: 0
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 0
broadcast: 0
rx_no_buffer_count: 0
collisions: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
hw_rsc_aggregated: 0
hw_rsc_flushed: 0
fdir_match: 0
fdir_miss: 0
fdir_overflow: 0
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
tx_flow_control_xon: 0
rx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_flow_control_xoff: 0
rx_csum_offload_errors: 0
alloc_rx_page: 49056
alloc_rx_page_failed: 0
alloc_rx_buff_failed: 0
rx_no_dma_resources: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_skipped: 0
rx_hwtstamp_cleared: 0
fcoe_bad_fccrc: 0
rx_fcoe_dropped: 0
rx_fcoe_packets: 0
rx_fcoe_dwords: 0
fcoe_noddp: 0
fcoe_noddp_ext_buff: 0
tx_fcoe_packets: 0
tx_fcoe_dwords: 0
tx_queue_0_packets: 0
tx_queue_0_bytes: 0
tx_queue_1_packets: 0
tx_queue_1_bytes: 0
tx_queue_2_packets: 0
tx_queue_2_bytes: 0
tx_queue_3_packets: 0
tx_queue_3_bytes: 0
tx_queue_4_packets: 0
tx_queue_4_bytes: 0
tx_queue_5_packets: 0
tx_queue_5_bytes: 0
tx_queue_6_packets: 0
tx_queue_6_bytes: 0
tx_queue_7_packets: 0
tx_queue_7_bytes: 0
tx_queue_8_packets: 0
tx_queue_8_bytes: 0
tx_queue_9_packets: 0
tx_queue_9_bytes: 0
tx_queue_10_packets: 0
tx_queue_10_bytes: 0
tx_queue_11_packets: 0
tx_queue_11_bytes: 0
tx_queue_12_packets: 0
tx_queue_12_bytes: 0
tx_queue_13_packets: 0
tx_queue_13_bytes: 0
tx_queue_14_packets: 0
tx_queue_14_bytes: 0
tx_queue_15_packets: 0
tx_queue_15_bytes: 0
tx_queue_16_packets: 0
tx_queue_16_bytes: 0
tx_queue_17_packets: 0
tx_queue_17_bytes: 0
tx_queue_18_packets: 0
tx_queue_18_bytes: 0
tx_queue_19_packets: 0
tx_queue_19_bytes: 0
tx_queue_20_packets: 0
tx_queue_20_bytes: 0
tx_queue_21_packets: 0
tx_queue_21_bytes: 0
tx_queue_22_packets: 0
tx_queue_22_bytes: 0
tx_queue_23_packets: 0
tx_queue_23_bytes: 0
tx_queue_24_packets: 0
tx_queue_24_bytes: 0
tx_queue_25_packets: 0
tx_queue_25_bytes: 0
tx_queue_26_packets: 0
tx_queue_26_bytes: 0
tx_queue_27_packets: 0
tx_queue_27_bytes: 0
tx_queue_28_packets: 0
tx_queue_28_bytes: 0
tx_queue_29_packets: 0
tx_queue_29_bytes: 0
tx_queue_30_packets: 0
tx_queue_30_bytes: 0
tx_queue_31_packets: 0
tx_queue_31_bytes: 0
tx_queue_32_packets: 0
tx_queue_32_bytes: 0
tx_queue_33_packets: 0
tx_queue_33_bytes: 0
tx_queue_34_packets: 0
tx_queue_34_bytes: 0
tx_queue_35_packets: 0
tx_queue_35_bytes: 0
tx_queue_36_packets: 0
tx_queue_36_bytes: 0
tx_queue_37_packets: 0
tx_queue_37_bytes: 0
tx_queue_38_packets: 0
tx_queue_38_bytes: 0
tx_queue_39_packets: 0
tx_queue_39_bytes: 0
tx_queue_40_packets: 0
tx_queue_40_bytes: 0
tx_queue_41_packets: 0
tx_queue_41_bytes: 0
tx_queue_42_packets: 0
tx_queue_42_bytes: 0
tx_queue_43_packets: 0
tx_queue_43_bytes: 0
tx_queue_44_packets: 0
tx_queue_44_bytes: 0
tx_queue_45_packets: 0
tx_queue_45_bytes: 0
tx_queue_46_packets: 0
tx_queue_46_bytes: 0
tx_queue_47_packets: 0
tx_queue_47_bytes: 0
tx_queue_48_packets: 0
tx_queue_48_bytes: 0
tx_queue_49_packets: 0
tx_queue_49_bytes: 0
tx_queue_50_packets: 0
tx_queue_50_bytes: 0
tx_queue_51_packets: 0
tx_queue_51_bytes: 0
tx_queue_52_packets: 0
tx_queue_52_bytes: 0
tx_queue_53_packets: 0
tx_queue_53_bytes: 0
tx_queue_54_packets: 0
tx_queue_54_bytes: 0
tx_queue_55_packets: 0
tx_queue_55_bytes: 0
tx_queue_56_packets: 0
tx_queue_56_bytes: 0
tx_queue_57_packets: 0
tx_queue_57_bytes: 0
tx_queue_58_packets: 0
tx_queue_58_bytes: 0
tx_queue_59_packets: 0
tx_queue_59_bytes: 0
tx_queue_60_packets: 0
tx_queue_60_bytes: 0
tx_queue_61_packets: 0
tx_queue_61_bytes: 0
tx_queue_62_packets: 0
tx_queue_62_bytes: 0
tx_queue_63_packets: 0
tx_queue_63_bytes: 0
rx_queue_0_packets: 0
rx_queue_0_bytes: 0
rx_queue_1_packets: 0
rx_queue_1_bytes: 0
rx_queue_2_packets: 0
rx_queue_2_bytes: 0
rx_queue_3_packets: 0
rx_queue_3_bytes: 0
rx_queue_4_packets: 0
rx_queue_4_bytes: 0
rx_queue_5_packets: 0
rx_queue_5_bytes: 0
rx_queue_6_packets: 0
rx_queue_6_bytes: 0
rx_queue_7_packets: 0
rx_queue_7_bytes: 0
rx_queue_8_packets: 0
rx_queue_8_bytes: 0
rx_queue_9_packets: 0
rx_queue_9_bytes: 0
rx_queue_10_packets: 0
rx_queue_10_bytes: 0
rx_queue_11_packets: 0
rx_queue_11_bytes: 0
rx_queue_12_packets: 0
rx_queue_12_bytes: 0
rx_queue_13_packets: 0
rx_queue_13_bytes: 0
rx_queue_14_packets: 0
rx_queue_14_bytes: 0
rx_queue_15_packets: 0
rx_queue_15_bytes: 0
rx_queue_16_packets: 0
rx_queue_16_bytes: 0
rx_queue_17_packets: 0
rx_queue_17_bytes: 0
rx_queue_18_packets: 0
rx_queue_18_bytes: 0
rx_queue_19_packets: 0
rx_queue_19_bytes: 0
rx_queue_20_packets: 0
rx_queue_20_bytes: 0
rx_queue_21_packets: 0
rx_queue_21_bytes: 0
rx_queue_22_packets: 0
rx_queue_22_bytes: 0
rx_queue_23_packets: 0
rx_queue_23_bytes: 0
rx_queue_24_packets: 0
rx_queue_24_bytes: 0
rx_queue_25_packets: 0
rx_queue_25_bytes: 0
rx_queue_26_packets: 0
rx_queue_26_bytes: 0
rx_queue_27_packets: 0
rx_queue_27_bytes: 0
rx_queue_28_packets: 0
rx_queue_28_bytes: 0
rx_queue_29_packets: 0
rx_queue_29_bytes: 0
rx_queue_30_packets: 0
rx_queue_30_bytes: 0
rx_queue_31_packets: 0
rx_queue_31_bytes: 0
rx_queue_32_packets: 0
rx_queue_32_bytes: 0
rx_queue_33_packets: 0
rx_queue_33_bytes: 0
rx_queue_34_packets: 0
rx_queue_34_bytes: 0
rx_queue_35_packets: 0
rx_queue_35_bytes: 0
rx_queue_36_packets: 0
rx_queue_36_bytes: 0
rx_queue_37_packets: 0
rx_queue_37_bytes: 0
rx_queue_38_packets: 0
rx_queue_38_bytes: 0
rx_queue_39_packets: 0
rx_queue_39_bytes: 0
rx_queue_40_packets: 0
rx_queue_40_bytes: 0
rx_queue_41_packets: 0
rx_queue_41_bytes: 0
rx_queue_42_packets: 0
rx_queue_42_bytes: 0
rx_queue_43_packets: 0
rx_queue_43_bytes: 0
rx_queue_44_packets: 0
rx_queue_44_bytes: 0
rx_queue_45_packets: 0
rx_queue_45_bytes: 0
rx_queue_46_packets: 0
rx_queue_46_bytes: 0
rx_queue_47_packets: 0
rx_queue_47_bytes: 0
rx_queue_48_packets: 0
rx_queue_48_bytes: 0
rx_queue_49_packets: 0
rx_queue_49_bytes: 0
rx_queue_50_packets: 0
rx_queue_50_bytes: 0
rx_queue_51_packets: 0
rx_queue_51_bytes: 0
rx_queue_52_packets: 0
rx_queue_52_bytes: 0
rx_queue_53_packets: 0
rx_queue_53_bytes: 0
rx_queue_54_packets: 0
rx_queue_54_bytes: 0
rx_queue_55_packets: 0
rx_queue_55_bytes: 0
rx_queue_56_packets: 0
rx_queue_56_bytes: 0
rx_queue_57_packets: 0
rx_queue_57_bytes: 0
rx_queue_58_packets: 0
rx_queue_58_bytes: 0
rx_queue_59_packets: 0
rx_queue_59_bytes: 0
rx_queue_60_packets: 0
rx_queue_60_bytes: 0
rx_queue_61_packets: 0
rx_queue_61_bytes: 0
rx_queue_62_packets: 0
rx_queue_62_bytes: 0
rx_queue_63_packets: 0
rx_queue_63_bytes: 0
tx_pb_0_pxon: 0
tx_pb_0_pxoff: 0
tx_pb_1_pxon: 0
tx_pb_1_pxoff: 0
tx_pb_2_pxon: 0
tx_pb_2_pxoff: 0
tx_pb_3_pxon: 0
tx_pb_3_pxoff: 0
tx_pb_4_pxon: 0
tx_pb_4_pxoff: 0
tx_pb_5_pxon: 0
tx_pb_5_pxoff: 0
tx_pb_6_pxon: 0
tx_pb_6_pxoff: 0
tx_pb_7_pxon: 0
tx_pb_7_pxoff: 0
rx_pb_0_pxon: 0
rx_pb_0_pxoff: 0
rx_pb_1_pxon: 0
rx_pb_1_pxoff: 0
rx_pb_2_pxon: 0
rx_pb_2_pxoff: 0
rx_pb_3_pxon: 0
rx_pb_3_pxoff: 0
rx_pb_4_pxon: 0
rx_pb_4_pxoff: 0
rx_pb_5_pxon: 0
rx_pb_5_pxoff: 0
rx_pb_6_pxon: 0
rx_pb_6_pxoff: 0
rx_pb_7_pxon: 0
rx_pb_7_pxoff: 0
root@node03:~#
root@node03:~# ethtool -i enp4s0f1
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x30030001
expansion-rom-version:
bus-info: 0000:04:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@node03:~# ethtool --phy-stats enp4s0f1
ethtool: bad command line argument(s)
For more information run ethtool -h
root@node03:~# ethtool -h
ethtool version 4.8
Usage:Just out of curiousity, have you ever used these nics with the cables/switch that you have them connected to now? I'd go back and double check the known working config.
you use dac or transceiver ?
If you are using transceiver,
you may try allow_unsupported_sfp=1 option for the ixgbe driver.
If you are using transceiver,
you may try allow_unsupported_sfp=1 option for the ixgbe driver.
Last edited:
@tuantv
here is my dmesg on my ixgbe device
I lookup your log and your SFP+ were never detected and that should be the direction you need to look at.
here is my dmesg on my ixgbe device
Code:
[ 2.046280] ixgbe 0000:05:00.0 enp5s0f0: renamed from eth1
[ 7.369699] ixgbe 0000:05:00.0 enp5s0f0: changing MTU from 1500 to 9000
[ 7.543537] ixgbe 0000:05:00.0: registered PHC device on enp5s0f0
[ 7.648574] IPv6: ADDRCONF(NETDEV_UP): enp5s0f0: link is not ready
[ 7.716031] ixgbe 0000:05:00.0 enp5s0f0: detected SFP+: 3
[ 7.964053] ixgbe 0000:05:00.0 enp5s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 7.964169] IPv6: ADDRCONF(NETDEV_CHANGE): enp5s0f0: link becomes readyI lookup your log and your SFP+ were never detected and that should be the direction you need to look at.