Ethernet port disappeared in Proxmox; no connectivity on host, but VMs are still reachable?

[rprichard]

I have had Proxmox up and running with no issues for over a year. Today I decided to move the Proxmox machine to a different ethernet port on my router (separate pfSense box) to put it in a DMZ. To make things easier, I configured the new interface in pfSense with all the same settings (interface address, DHCP range, etc). However, certain settings like static mappings were lost. Everything else remaining the same, I figured the move would be seamless, as Proxmox would retain the same address.

However, it has been anything but. Despite many hours of troubleshooting and countless reboots, I cannot get Proxmox to get an IP address. Pinging the gateway results in "destination host unreachable", and running an nmap on the whole address range it belongs to returns just itself and one VM that is exposed. Curiously, that VM responds to pings from the rest of my network despite the fact that the host does not have a working connection.

I'm pretty new to Linux/GNU and to networking, so bear with me.

I have hooked my PC to the network port that Proxmox is now on, and the port worked perfectly. That port's interface on my router has an address of 172.20.0.1/24, and the DHCP server is configured to provide 172.20.0.100-172.20.0.200. My Proxmox machine is configured to use 172.20.0.100, so there should be no issue there. Here is my /etc/network/interfaces file:

Spoiler

It seems problematic that vmbr0 is configured to use enp2s0, when `ip link` shows no port with that name. Perhaps my understanding is lacking. As an aside, it seems that vmbr1-4 are artifacts from the quad-port NIC I had installed at one point. If this seems like a reasonable deduction, can they be removed from my interfaces file?
Anyway, here's `ip link`:

Spoiler

If relevant, here's `lspci | grep Ethernet`:

Spoiler

Finally, here's what happens when I try to ping the gateway:

Spoiler

I've spent ~3 hours trying different things (different IPs, different ports on the router, different ethernet cables, attempted to set Proxmox to DHCP, etc.) and I'm out of ideas. I would greatly appreciate if someone more knowledgeable in these things could point me in the right direction[/ISPOILER]

 

[rprichard]

Had energy to do a bit more digging today. Running `ifreload -a` returns "Error: vmbr0: bridge port enp2s0 does not exist."

So it seems like Proxmox is not seeing the physical ethernet port, though I'm not sure how a VM would have connectivity if this were the case. I read on another forum post that enabling the network stack in the BIOS may help (though I can't see how), but there was no effect.

Against my better judgment that is telling me to change as few things as possible, I removed vmbr1-4 from /etc/network/interfaces, as the port numbers associated with those are certainly from my old network card.

Somehow I cannot find a single thread anywhere describing this situation, where the ethernet port is not visible in the output of `ip link`.

I am about an hour away from just grabbing all my backups (can't wait to learn how to mount a USB drive in the CLI) and reinstalling the OS. Really really hoping to not come to that.

Edit: Curious. I checked system logs with `dmesg | grep enp`, there's a message: "[1.878...] r8169 0000:01:00.0 enp1s0: renamed from eth0".
So apparently at least at some point the system is seeing a network port? I'm not certain why it's "enp1s0", when the interfaces file references "enp2s0" - as I understand it, the changed number refers to the bus the port is on which certainly has not changed. At any rate, modifying /etc/network/interfaces to reference enp1s0 instead has the same result.

However, rebooting after modifying interfaces produced a greater quantity of log messages. This is encouraging. (Cleaned up the OCR below as best I could)

root@pve:~# dmesg | grep enp
[2.390310] r8169 0000:01:00.0 enp1s0: renamed from etho
[8.666990] vmbr0: port 1 (enp1s0) entered blocking state
[8.666995] vmbr0: port 1 (enp1s0) entered disabled state
[8.667052] device enpiso entered promiscuous mode
[8.891097] r8169 0000:01:00.0 enp1s0: Link is Down
[8.893857] vmbr0: port 1(enp1s0) entered blocking state
[8.893861] vmbr0: port 1(enp1s0) entered forwarding state
[9.697424] vmbr0: port 1 (enp1s0) entered disabled state
[29.360260] vmbr0: port 1 (enp1s0) entered disabled state
[29.360378] device enp1s0 left promiscuous mode
[29.360383] vmbr0: port 1 (enp1s0) entered disabled state
root@pve:~# ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data. From 172.20.0.100 icmp_seq=1 Destination Host Unreachable

"Link is down" makes me think there's an external issue. However, nothing outside the machine has changed since I had connectivity to one of the VMs (which I seem to have lost about half an hour ago).


Edit 2: Okay, I re-disabled the network stack, re-plugged the ethernet cable to ensure a good connection, and rebooted. Now the link is briefly going up before ultimately ending up in a disabled state like before. Not sure if this is anything I did or just a roll of the dice on a reboot. (OCR errors below, but you get the picture)

root@pve:"# dmesg | grep enp
[ 1.857054] r8169 0000:01:00.0 enp1s0: renamed from etho 
[9.491096] vmbro: port 1(enp1s0) entered blocking state
[ 9.491101] vmbro: port 1 (enp1s0) entered disabled state
[ 9.491156] device enp1s0 entered promiscuous mode
[ 9.727463] r8169 0000:01:00.0 enp1s0: Link is Down
[ 9.730111] vmbro: port 1 (enp1s0) entered blocking state 
[9.730115] vmbro: port 1 (enp1s0) entered forwarding state
[ 10.497845] vmbro: port 1 (enp1s0) entered disabled state 
[13.465606] r8169 0000:01:00.0 enp1s0: Link is Up - 1Gbps/Full - flow control rx/tx
[ 13.465623] vmbro: port 1 (enp1s0) entered blocking state 
[13.465626] vmbro: port 1 (enp1s0) entered forwarding state
[ 29.027580] r8169 0000:01:00.0 enp1s0: Link is Down
[ 29.028092] vmbro: port 1(enp1s0) entered disabled state
[ 29.028271] device enp1s0 left promiscuous mode
[ 29.028276] vmbro: port 1(enp1s0) entered disabled state

Now the VM is back online, but there's nothing in my router's logs about the PVE node making a DHCP request. I also still cannot ping the gateway (as would be expected since I don't have an IP). Now to figure out why the port is going disabled and how to fix it.

 

[rprichard]

I cannot wrap my head around how it is possible that the ethernet link is down on Proxmox but one VM has network access. I am sure this is Linux networking 101 but it's beyond me. Anyone?

 

[rprichard]

Is it possibly an issue with the R8169 driver?

 

[rprichard]

Alright, I'd love to work this out and post the solution here but I'm running out of time. Reinstalling Proxmox as soon as I finish pulling backups off.

 

[rprichard]

Alright, reinstall did the trick. `/etc/network/interfaces` is configured exactly as I had it, but `enp1s0` now shows up in `ip addr`. Here's what `dmesg | grep enp` looks like on a working system:

 

[fabian]

it could very well be an issue with that particular NIC model - maybe your problematic state was using a newer kernel?

 

[rprichard]

Thanks for the response. I'm in the process of restoring the system, and the issue just resurfaced. After a bit of digging, I realized what happened - I was passing through my GPU to one of my VMs. I removed the GPU, and on reboot apparently Proxmox decided that since it couldn't find the GPU it would pass my ethernet controller through to that VM. Is this expected behavior?

 

[spirit]

Thanks for the response. I'm in the process of restoring the system, and the issue just resurfaced. After a bit of digging, I realized what happened - I was passing through my GPU to one of my VMs. I removed the GPU, and on reboot apparently Proxmox decided that since it couldn't find the GPU it would pass my ethernet controller through to that VM. Is this expected behavior?

when you passthouh a device, the whole iommu group is detached from host.

if you have a good motherboard, you'll have 1iommu for 1 device, so no problem. Some cheap motherboard can have 2 devices in 1ommu group (gpu + nic) for example, so if you passthough the gpu, it'll detach the nic too from host.

 

[Neobin]

I removed the GPU, and on reboot apparently Proxmox decided that since it couldn't find the GPU it would pass my ethernet controller through to that VM. Is this expected behavior?

In addition to what @spirit said:
If you physically removed the GPU, the PCI-addresses of other devices will most likely change and therefore the PCI-address, that was before your GPU, may now be your NIC, for example.

So, with every (physical) hardware change, you should beforehand disable autostart for all VMs that have PCIe-passthrough set up and confirm/verify every PCI-address of the PCIe-passthroughed devices first, before you start those VMs again.

 

[webmastermt]

hi guys,

i solved problem change to harddisk.

i was using sata ssd

i changed nvme disk and solved.

i think sata not enough for 10 or 1 gigabit lan

sory for my bad english