Lost Management Interface

[wdatkinson]

I've been running proxmox on a DL380G7 for a couple of years. I've slowly been migrating machines from an old ESXI server. Yesterday, while troubleshooting an unrelated network issue, I noticed that I was significantly behind on updates so I upgraded using the option in the web gui. Upon a reboot, I had zero connectivity into the box either on vmbr0 (management, single VLAN access port) or vmbr1 (trunked VM bridge).

After grabbing a crash cart of sorts, I began to poke around. I tried rebooting, using systemctl to restart the networking service, and many other options that my internet searching turned up.

Eventually, I was able to get vmbr1 up by using the 'ip link set vmbr1 up' command. After a short while, I began to see VM's with network connectivity. Using the qm command to bring up additional VM's I was able to restore most functionality, But I still had/have no management interface. Performing the same ip link command on vmbr0 appeared to bring it up, but it did not have an IP, despite what was configured in /etc/network/interfaces.

i have searched and read so many different posts, but have not found an answer. At this point, I feel like my only recourse is to re-install proxmox and import the existing VM's, but I still feel that I should be able to get the management interface back up, if for nothing else than to help ease operation while I migrate a handful of ESXI VM's over and then wipe ESXI and rebuilt it into a new proxmox server.

For now, if I need to access management functions, I have to RDP into a Win10 box and then use HP iLO to access the console, or go sit in front of the machine. Neither option is ideal

Happy to provide any additional information.

 

[aaron]

Can you provide the output of the ip a command and the contents of /etc/network/interfaces?

 

[wdatkinson]

With no management interface, I'm stuck with taking pictures of the monitor either live or in ilo.

Attached is the /etc/network/interfaces

The output of ip a is ~136 entries, so not sure how to capture that in its entirety.

 

[Ramalama]

Do you use virtual functions?

 

[aaron]

The output of ip a is ~136 entries, so not sure how to capture that in its entirety.

You could pipe it to a pager like less: ip a | less
You can scroll easily and exit it by pressing q.

Checking if the enp3s0f1 is actually present. You did not change anything on the hardware right? Like changing some PCI cards or placing them in other slots? That could cause the NIC to get another name.

 

[wdatkinson]

Ok, I can grep it out and take a pic of the interface, but it is there. And f1 is vmbr1, which is feeding the VM's, f0 is vmbr0 which is the management interface.

And no changes were made. I merely ran updates, which appeared to hang in the web interface. I came back everal hours later. So I rebooted the machine. That's it.

 

[aaron]

Hmm, so let me recap the situation. Right now you don't have any network connection. You cannot ping the server or ping something from the PVE server?

The status of the interfaces vmbr0 and enp3s0f0 from the ip a command would be very interesting.

 

[wdatkinson]

vmbr0 is toast. So no management via web, ssh, etc. Nothing I do seems to bring it back.

vmbr1 is up, so those VM's that 's either started from the physical console or via ilo are running normally. But I have no easy way to admin the box. I don't sit in front of the console and have a single Windows desktop from which to run ilo. Everything else is some flavor of linux or MacOS.

I'm heading downstairs this morning and in between meetings will grab you an 'ip a | grep enp3s0f8' and snap a pic with my phone to post here.

 

[aaron]

If you do a grep, do include the lines before and after, just to be sure: grep -B 6 -A 6 enp3s0f0. Also do it for vmbr0

 

[Wolf7]

Just had a very similar effect: after installing an additional network card, the numbering of interfaces has changed, enp3s0 no longer available, instead the first available interface is enp4s0. Once the vmbr0 got connected to enp3s0, it does not work, there is no web-gui connection.

How I solved it the dirty way:
in web-gui, without the new card, anticipate the correct interface after installation and connect it to the wanted Linux Bridge.
One may watch the process by "IP a" at the console.
Then install the netweok card, and the old bridge works with the new interface enp4s0, instead of enp3s0.

If the first interface is named eno1 (on the motherboard), then adding a network card works without problems.
My enp3s0 and enp4s0 are on the mortherboard too. Strange, all 3 boxes are identical and installed at the same day, same ISO.

I hope, there will be a more elegant way.

Yours, Wolf7

 

[wdatkinson]

I have no access to the managment interface, vmbr0, or it's associated physical interface. Vmbr1 is up (trunked) and VM's are running. If i need to do something with the VM's I either have to go to the console or ilo to the console and run qm from the shell.

Again, no hardware has changed. Nothing added, nothing removed. I merely attempted an upgrade which appeared to hang. After allowing several hours for it to complete, I gave up and rebooted and here i am.

Proxmox is installed in a USB drive and the 8 SAS drives in the DL380 are my storage. So I'm wondering if I should just re-install Proxmox from scratch and import the VM's from the storage array, however that might work.

 

[Wolf7]

Hi, well it came to my mind, that your GUI is lost, sorry. You may try to edit from console /etc/network/interfaces and "anticipate" the correct interface name. In my case, it is enp4s0, below is an excerpt of my working version, pls check vmbr0 mentioning the 2nd interface enp4s0
auto lo
iface lo inet loopback

iface enp3s0 inet manual

iface enp4s0 inet manual

iface enp5s0 inet manual

iface enp6s0 inet manual

iface enp7s0 inet manual

..... other stuff about the new network adapter

auto vmbr0
iface vmbr0 inet static
address xyz
gateway xyz
bridge-ports enp4s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094

auto vmbr1
iface vmbr1 inet static
address xyz
bridge-ports enp5s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#cluster traffic