PVE7to8 Upgrade ran without errors. PVE Server did boot and since my PVE was not reachable anymore.

[vongehlens]

Hello all,


Upgrade done using the official steps to the letter.

Upgrade ran without errors. PVE Server did boot and since my PVE server was not reachable anymore...



OK, let go to the attick, get Monitor & keyboard, goto basement ...connect...

Oha! … Maschine did reboot. good. Can login locally…



Oha! … no Ping no Pong… no LAN not any other Interface… let´s have a look:



Soso, after the Upgrade none of my VMBridges did start, systemctl status networking.service told me „Bridge port XY does not exist…

--------snip-----------

root@svg-pxmx-01:~# systemctl status networking.service

● networking.service - Network initialization

Loaded: loaded (/lib/systemd/system/networking.service; enabled; preset: enabled)

Active: active (exited) since Fri 2023-08-25 14:23:09 CEST; 6min ago

Docs: man:interfaces(5)

man:ifup(8)

man:ifdown(8)

Process: 867973 ExecStart=/usr/share/ifupdown2/sbin/start-networking start (code=exited, status=0/SUCCESS)

Main PID: 867973 (code=exited, status=0/SUCCESS)

CPU: 1.064s



Aug 25 14:23:09 svg-pxmx-01 /usr/sbin/ifup[867990]: error: vmbr1: bridge port enp4s0f1 does not exist

Aug 25 14:23:09 svg-pxmx-01 networking[867990]: warning: vmbr1: apply bridge ports settings: bridge configuration failed (missing ports)

Aug 25 14:23:09 svg-pxmx-01 /usr/sbin/ifup[867990]: warning: vmbr1: apply bridge ports settings: bridge configuration failed (missing ports)

Aug 25 14:23:09 svg-pxmx-01 networking[867990]: error: vmbr2: bridge port enp4s0f0 does not exist

Aug 25 14:23:09 svg-pxmx-01 /usr/sbin/ifup[867990]: error: vmbr2: bridge port enp4s0f0 does not exist

Aug 25 14:23:09 svg-pxmx-01 networking[867990]: warning: vmbr2: apply bridge ports settings: bridge configuration failed (missing ports)

Aug 25 14:23:09 svg-pxmx-01 /usr/sbin/ifup[867990]: warning: vmbr2: apply bridge ports settings: bridge configuration failed (missing ports)

Aug 25 14:23:09 svg-pxmx-01 networking[867990]: error: >>> Full logs available in: /var/log/ifupdown2/network_config_ifupdown2_16_Aug-25-2023_14:23:08.6>

Aug 25 14:23:09 svg-pxmx-01 /usr/sbin/ifup[867990]: >>> Full logs available in: /var/log/ifupdown2/network_config_ifupdown2_16_Aug-25-2023_14:23:08.6127>

Aug 25 14:23:09 svg-pxmx-01 systemd[1]: Finished networking.service - Network initialization.

---------------snip-------------



hmmm, which interfaces do we have now (many):

-----------------snip---------

root@svg-pxmx-01:~# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host noprefixroute

valid_lft forever preferred_lft forever

2: ens4f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

link/ether 00:10:18:64:e7:30 brd ff:ff:ff:ff:ff:ff

altname enp4s0f0

3: ens4f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

link/ether 00:10:18:64:e7:32 brd ff:ff:ff:ff:ff:ff

altname enp4s0f1

4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

link/ether 00:10:18:64:e7:34 brd ff:ff:ff:ff:ff:ff

altname enp5s0f0

5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

link/ether 00:10:18:64:e7:36 brd ff:ff:ff:ff:ff:ff

altname enp5s0f1

6: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000

link/ether ac:9e:17:b4:f6:57 brd ff:ff:ff:ff:ff:ff

altname enp0s25

….did cut the list here...

-----------------snip---------



alright, eno1 interface is the one i know i never used and it is on the that server mainboard...

ok, emergency operation:

root@svg-pxmx-01:~# nano /etc/network/interfaces

at section vmbr0 swap Interface „enp5s0f0“ against „eno1“

… do „systemctl restart networking.service“ to restart the network… ok, ping at LAN starts to work again and i can access the GUI from LAN again





Checking: well, before the upgrade i hat not that many NIC´s… and i´ve not mounted additional hardwar interfaces …





let´s check what the PCI inventory tells us:
root@svg-pxmx-01:~# lspci | grep -i ethernet

00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V (rev 05)

04:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

04:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

05:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

05:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

06:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit TN Network Connection (rev 01)

06:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit TN Network Connection (rev 01)

root@svg-pxmx-01:~#





Weired, the NICs had been mixed up with new names during the upgrade to PVE8, therefore the Bridges did not start again.



Using the Bus-IDs and the old Interfacenamee i´ve matched old vs new names.







Old vs new:

Old (when it was PVE7):	New (after upgrade to PVE8):
Enp4s0f0	Ens4f0
Enp4s0f1	Ens4f1
Enp5s0f0	Eth2
Enp5s0f1	Eth3

Now i´ve edited the Bridges throught he GUI removing the old names and putting the new names.

Start a Shell and execute again „systemctl restart networking.service“...



All VM´s using the bridges hat to be restarted to get a link again.



All services are back online now.





i´m very happy that this has happened on a machine i could quickly goto. My several other Proxmox Server und Clusters i think i must operate on PVE7 until it´s figured out why the upgrade mixed up the interface names. And that this does not happen again.


rgds
Stephan

 

[vongehlens]

little update...
one further reboot of the proxmox server has shuffled one more interface...
ens4f1 now became eth1 ... result: that vmbridge where ens4f1 was member did stop working...

 

[leesteken]

eth* names are not stable and can change every boot like /dev/sd*. Somehow the new Debian version under the hood of PVE 8 no longer renames some of your network devices. If you need stable names (and probably every Proxmox does!) you need to write your own rules for the network device names.

 

[vongehlens]

Morning all!
Update: the NIC "Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)" was the root cause in my system.
while getting desparate yesterday i took out that Broadcom BCM5709 4port card and replaced it with 2x 2port Intel 82571EB/82571GB.

the shuffeling of interface names has stopped now, i´ve even rebooted the server four times to create some issue but was not successful.
hope this finding will be helpful to some.


if time permits i´ll check that Broadcom card in lab setup.

PS: @leesteken: thanks for the reply and link to the UDEV Rule document. will try to check that out. however: each proxmox installation needs stable interface names.

 

[leesteken]

PS: @leesteken: thanks for the reply and link to the UDEV Rule document. will try to check that out. however: each proxmox installation needs stable interface names.

I agree that stable names would be great and I don't know why you don't get thsoe for that particular network controller. Note that the "stable" names are based on PCI-ID and those unfortunately change when adding PCI(e) and NVMe devices.
Using MAC addresses, like the example rules I linked, has the problem that MAC addresses are not as unique as you would expect, unfortunately. And replacing a broken device with a new one would cause the name to change because of a different MAC.
Would be nice if people were more aware of this and the Proxmox manual described how to do this, especially since PVE is designed to be managed remotely.