Loss of Network

[Kryten73]

Hi All,

Looking for help in the dark art of networking. I have built my own server using a SuperMicro X10SRH-CLN4F (4 network ports plus 1 IPMI port). Server has been running perfectly for several years and multiple upgrades. (docker/truenas scale with PCI passthroug, Win, Arch).

Issue is that few days ago (and I can only relate to putting in a GFX card and then removing it) the network links eno1-4 not showing up.

my network cable has always been in port 1 and responded on 192.168.1.4 - interesting enough the IPMI also worked on the same port but with 192.168.1.5 so had assumed some 'fancy' subnetting/NAT was taking place on the MoBo.

Anyhow I'm able to access the server IPMI on 192.168.1.5 which has allowed me to get the contents of /etc/network/interfaces and ip link and ip add (Attached).

It appears because Proxmox cannot find eno1 then my vmbr0 does not start.

All the VMs are up and running so I'm loathed to go through an entire rebuild due to my lack of networking knowledge - appreciate if any could review and let me know thoughts.

Thank you.

 

[shrdlicka]

Hi,

have you checked lspci -v if your network card is showing up there?

 

[Kryten73]

yes they are there

 

[shrdlicka]

That is all the info you are getting from "lspci -v"?

Mine looks like this:

00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-V (rev 10)
    DeviceName: Onboard - Ethernet
    Subsystem: ASRock Incorporation Ethernet Connection (7) I219-V
    Flags: bus master, fast devsel, latency 0, IRQ 125, IOMMU group 8
    Memory at a2200000 (32-bit, non-prefetchable) [size=128K]
    Capabilities: <access denied>
    Kernel driver in use: e1000e
    Kernel modules: e1000e

Can you find entries in your journalctl -b 0 log for one of the adresses (i.e. "04:00.1", "04.:00.2" ...)?

 

[Kryten73]

sorry all the detail.... this is the first two ports, second two ports are similar

 

[shrdlicka]

Can you attach the output of journalctl -b0 as a file here? Maybe there is something in there that explains the missing Ethernet interfaces.

 

[Neobin]

Your NICs are using the vfio-pci driver, which you only want for devices that you actually PCIe-passthrough.

As you mentioned, you are using PCIe-passthrough and installed/removed devices lastly, you should check your whole PCIe-passthrough-configuration.
On some point, you unintentionally bind your NICs to the vfio-pci driver.

 

[Kryten73]

I'm unable to transfer files from the machine - only have a console via IPMI - anything specific to search for?

 

[Neobin]

Have a look into the files in: /etc/modprobe.d/ for: options vfio-pci ids=XXXX:XXXX [1].
Afaik you can also bind devices per kernel command line; so have a look there too: [2]. (Viewing currently used one: cat /proc/cmdline.)
Compare those IDs [1] with the ones from your NICs; they should not be in there.
Also check, if there is something (especially an Intel NIC driver) blacklisted [1], that should not be.

[1] https://pve.proxmox.com/wiki/PCI(e)_Passthrough#_host_device_passthrough
[2] https://pve.proxmox.com/wiki/Host_Bootloader#sysboot_edit_kernel_cmdline

 

[Kryten73]

Thank you very much for the advice - away from home for a bit, will try the suggestions as I think this gives me enough to investigate further.

Really appreciate your time,

 

[Kryten73]

Thanks all for advice - managed to work it out. Somehow my 4 port on board network card was passing through to my TrueNas VM rather than my SAS/LSI storage card - no idea how that happened given I was moving in and out a GFX card on another VM. Anyway solved so thanks for the help.

Cheers!