[SOLVED] Installation of NIC removes onboard network interfaces

[duckerwhite]

BLUF: The onboard interfaces register in linux/Proxmox as well as an additional NIC but just not at the same time.

I have a X9DRD-LF-TW008 that I aim to use as an external server/firewall with Proxmox hosting pfSense among other things. The motherboard comes with two interfaces and an interface for IPMI. I picked up a NIC for the PCI-E slot to add an additional interface. I am unable to get Proxmox to recognize all the interfaces at the same time.

Without the NIC installed, Proxmox will recognize the two onboard interfaces and the IPMI works as well.
With the NIC installed, Proxmox will recognize the NIC but not the two onboard interfaces.

I have also tried installing Prox without the NIC installed and then installing the NIC after a successful installation of Prox. Once the NIC is installed both onboard interfaces go away but Prox will see and use the new NIC.

(ip a) shows the same results as mentioned above.

Any help or thoughts are appreciated.

 

[Stoiko Ivanov]

if this is due to a hardware problem you should be able to see that in the `dmesg` output (when it is not working)

else please post the output of
* `ip -detail link show`
(both with and without the NIC installed)

I hope this helps!

 

[duckerwhite]

As requested, below are the txt dumps for both the (dmesg) and (ip -d link show) commands both with and without the NIC installed.

It appears that the hardware isn't seeing the onboard interfaces once the NIC is installed.

Differences with and without the NIC installed

(With the NIC installed)
[ 2.033531] r8169 0000:82:00.0 eth0: RTL8168evl/8111evl, 00:e0:63:35:13:55, XID 2c9, IRQ 35
[ 2.033532] r8169 0000:82:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[ 2.060963] r8169 0000:82:00.0 enp130s0: renamed from eth0
[ 5.550641] RTL8211E Gigabit Ethernet r8169-8200:00: attached PHY driver [RTL8211E Gigabit Ethernet] (mii_bushy_addr=r8169-8200:00, irq=IGNORE)

(Without the NIC installed)
[ 2.064795] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
[ 2.127181] igb 0000:05:00.0: added PHC on eth0
[ 2.127183] igb 0000:05:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 2.127185] igb 0000:05:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 0c:c4:7a:34:14:ca
[ 2.127259] igb 0000:05:00.0: eth0: PBA No: 104900-000
[ 2.127260] igb 0000:05:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[ 2.182940] igb 0000:05:00.1: added PHC on eth1
[ 2.182942] igb 0000:05:00.1: Intel(R) Gigabit Ethernet Network Connection
[ 2.182943] igb 0000:05:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 0c:c4:7a:34:14:cb
[ 2.183017] igb 0000:05:00.1: eth1: PBA No: 104900-000
[ 2.183020] igb 0000:05:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[ 2.187601] igb 0000:05:00.0 enp5s0f0: renamed from eth0
[ 2.266627] igb 0000:05:00.1 enp5s0f1: renamed from eth1

 

[Stoiko Ivanov]

hmm - this is really odd - with the external NIC installed the pci-ids of the devices shift around (and the mgag (vga device) gets the pci-bus-id which was used before by the onboard NIC)

* you could try to update the BIOS/FIrmware of the mainboard?
* additionally check the BIOS settings related to the pci-bus (e.g above 4G decoding) and try changing a setting and then rebooting.

have not seen such a case before...

EDIT: a colleague just pointed me to this:

(Both CPUs need to be installed for full access to PCI-E slots and onboard controllers. See manual block diagram for details.)

from : https://www.supermicro.com/products/motherboard/xeon/c600/x9drd-lf.cfm

I hope this helps!

 

[duckerwhite]

So I appreciate the assistance, sometimes you just need to walk away from a situation and come back with fresh eyes.

Buried in the BIOS was an option for the LAN controller to disable the onboard interfaces if a NIC was installed in the PCI-E slot.

Sorry for dragging you all into my nightmare but I seriously appreciate the responsive replies and assistance. You guys rule and also make kick ass software, I love Proxmox.

 

[Stoiko Ivanov]

Glad you round the issue and managed to resolve it

Please edit the thread and mark it as 'SOLVED' - this should help others with a similar setup
Thanks!

 

[trisweb]

Glad you round the issue and managed to resolve it

Please edit the thread and mark it as 'SOLVED' - this should help others with a similar setup
Thanks!

So I appreciate the assistance, sometimes you just need to walk away from a situation and come back with fresh eyes.

Buried in the BIOS was an option for the LAN controller to disable the onboard interfaces if a NIC was installed in the PCI-E slot.

Sorry for dragging you all into my nightmare but I seriously appreciate the responsive replies and assistance. You guys rule and also make kick ass software, I love Proxmox.

Just wanted to say thanks for this from down the line, I was banging my head against the wall trying to figure out why I couldn’t use the other interfaces and here was the clear answer. Solved again. Thanks!