[Solved] Intel X540-T2 - second port not sending or receiving packets

[Aterfax]

Hi all,

I have 3 boxes, 2x Proxmox 6.3-6 (1x Xeon E3-1220V2 1x FX 8350) latest intel ixgbe driver and 1x Unraid 6.9.1 built in ixgbe driver (1x Xeon E3-1220V2) all of which seem to have the same issue with the 2nd port being detected, the link state being detected and negotiated to 10GBase correctly but the second ports on all 3 do not appear to send or recieve packets despite having static IPs set.

I'm running with PCIE passthrough of other components on Proxmox but not these 10GBase cards though I doubt IOMMU state is having an effect on things and I cannot turn passthrough off on the Proxmox boxes.

I've installed the latest Intel X540 driver and I have tried connecting various ports from each machine to the other with straight through and cross over cables - makes no difference - the first ports work but none of the second ports do.

I suspect something is wrong with the ixgbe drivers as I have a forth card running in a Windows 10 machine which is fully functional on both ports (and I have swapped cards through the Windows 10 machine and they work fully.)

Anyone got any ideas? I have no idea how to debug this as from the lspci and networking perspective the second ports appear to be fully functional. And the Unraid machine having the same issue is making me think ixgbe is probably the base cause.

 

[Stoiko Ivanov]

Anything interesting pointing to what causes the issue in the journal or `dmesg` output after booting up?

Regarding using the latest driver - I assume you mean the out-of-tree driver provided by Intel - I have had mixed experiences with those (meaning that they helped in some situation, but did break things in others) - Maybe try to use the ixgbe module shipped by the PVE kernel.

do the packets you sent out show up in a tcpdump on the machine?

I hope this helps!

 

[Aterfax]

There's nothing in the dmesg output or journal files which gives a hint.

Yes I do mean the latest out of tree by Intel - initially I was trying with the built in driver but was seeing the same thing.

Going to have an explore with tcpdump now.

Thanks!

 

[Aterfax]

Admittedly I haven't had enough time to dig into this during the week but it looks like the second interfaces were failing to set their IPs correctly (doh) - forcing static IPs again on the command line and I have now got symmetric 9.4Gbit going between all devices.

Thanks for the idea with tcpdump as the NICs were clearly getting something which made me look at the NICs in a bit more detail (they had not got the IPs I had told them to have.)

Much appreciated!

 

[Stoiko Ivanov]

Glad that helped

Please edit the thread and mark it as 'SOLVED'
Thanks!

 

[snpz]

Admittedly I haven't had enough time to dig into this during the week but it looks like the second interfaces were failing to set their IPs correctly (doh) - forcing static IPs again on the command line and I have now got symmetric 9.4Gbit going between all devices.

Thanks for the idea with tcpdump as the NICs were clearly getting something which made me look at the NICs in a bit more detail (they had not got the IPs I had told them to have.)

Much appreciated!

And it works as expected after a server reboot?
I still cant find the reason why randomly one or more interfaces are not detected correctly during bootup!
https://forum.proxmox.com/threads/no-network-after-proxmox-kernel-upgrade.86216/

 

[Aterfax]

I haven't rebooted yet, will respond again if I do encounter that issue.