PVE 8.2 update changed the name of some of my interfaces

[kurdam]

Hi,

I just updated to the latest version of PVE and on restart my server was not available of all networks.

I took me a bit to figure it out but i found out that the latest update renamed some of my interfaces. The problem is limited to only one network card but it was quite troublesome.

I don't know if this behaviour is expected and it's the first time that i'm encountering something like that.

I'm linking you a screenshot.



If you need more infos, don't hesitate to ask me

 

[ahayes]

At the very end of the PVE 8.2 section of the Roadmap that was linked from the press release and the release announcement in the forums there is a Known Issues & Breaking Changes section. The last bullet point in that is:

Upgrading kernels always carries the risk of network interface names changing, which can lead to invalid network configurations after a reboot. In this case you need to update the network configuration to reflect changes in naming. See the reference documentation on how to set the interface names based on MAC Addresses.

I hit the same problem and went looking for known issues and couldn't find them. They weren't linked directly from the announcements and they weren't placed at the top or linked from the top of the "Roadmap" (which also seems like a funny name for that change log document.) Multiple forum threads document people breaking their host/cluster networking and it is clear that many were not aware of the risk highlighted in that generic warning at the very end of a long change log. There is a 7_to_8 upgrade doc. I have not seen any other generic upgrade doc for point releases. That could be a good spot to highlight steps to take and things to look out for with every upgrade.

I think the placement of the known issues could be improved. I also think the Proxmox folks could call special attention to this issue and maybe develop a way to warn or mitigate when the upstream systemd net naming scheme is known to be changing. Or pin the scheme with major releases. I have been bitten by this same net naming scheme breaking change before in Ubuntu but was surprised to see it not handled in Proxmox updates too. There is awareness and some decent documentation about the issue in the networking docs linked from that known issue for folks who got that far.

In my case, I had multiple bonded interfaces across a pair of network adapters. The interfaces on one of the adapters did not get renamed so I did not see the problem at first because my cluster remained happy with 2/6 ethernet interfaces still available. It was only when I tried to migrate VMs before upgrading my last node that I discovered that some of my Open vSwitch configuration broke after upgrading to PVE 8.2. Digging into that led me to spot the other broken interfaces. I'll post separately about the OVS issue.

 

[kurdam]

Thank you for the heads up. Yes it is a pretty big suprise. Hopefully for us not too much damages but it could have been way worse.

 

[kurdam]

Do you know if there is a way to remove the old interfaces in order to avoid the confusion ?

 

[ahayes]

Have you reloaded the web UI page in your browser? Is your interfaces file correct? If you have fixed the interfaces file with the new names and run an `ifreload -a` but the old names persist, maybe try a reboot of the node?

 

[fabian]

this is not that easy to handle, since the "predictable" part of the predictable naming scheme only means that a NIC will retain the same name if you neither change the hardware nor the software (the previous scheme was "name the devices based on the order in which the kernel saw them on boot", which is not deterministic at all ).

there are many parts here that can affect the names:
- kernel including drivers (like the current upgrade from 6.5 to 6.8)
- hardware topology (e.g., removing/plugging devices, or system firmware updates that effectively change how things are reported/enumerated)
- firmware (see above)
- systemd/udev (both policy-wise, but also due to bugs or unintended side-effects)

the only way to have actually stable names is to assign custom ones to your NICs (e.g., based on MAC addresses or similar stable identifiers). we might make this a bit more straight-forward in the future, but it's not really possible to rename NICs that are in use, so this complicates the whole network management part quite a bit.

 

[kurdam]

Have you reloaded the web UI page in your browser? Is your interfaces file correct? If you have fixed the interfaces file with the new names and run an `ifreload -a` but the old names persist, maybe try a reboot of the node?

I did several reboot during the diagnostic phase and it didn't changed a thing.
That's not really that much important but i will have to be very careful when upgrading my other nodes.

 

[kurdam]

this is not that easy to handle, since the "predictable" part of the predictable naming scheme only means that a NIC will retain the same name if you neither change the hardware nor the software (the previous scheme was "name the devices based on the order in which the kernel saw them on boot", which is not deterministic at all ).

there are many parts here that can affect the names:
- kernel including drivers (like the current upgrade from 6.5 to 6.8)
- hardware topology (e.g., removing/plugging devices, or system firmware updates that effectively change how things are reported/enumerated)
- firmware (see above)
- systemd/udev (both policy-wise, but also due to bugs or unintended side-effects)

the only way to have actually stable names is to assign custom ones to your NICs (e.g., based on MAC addresses or similar stable identifiers). we might make this a bit more straight-forward in the future, but it's not really possible to rename NICs that are in use, so this complicates the whole network management part quite a bit.

Thank you for the update, yeah i understand that is quite complicated.
Maybe the best course of action is to give an alias at the pve installation that could link the alias to the mac address giving the user the opportunity to name them in a logical sense for them (for example left to right as they are physically on the server), it would solve a lot of headache when setting up a new node especially when you have an infrastructure like me with bounds, vmbrs and vlans. It's not that much vital but it could be a nice quality of life improvement.

 

[pyrodex]

Coming from a life long ESXi personal and enterprise user it would be in the Proxmox's team best interests as they are trying to compete for the folks frustrated with Broadcom to maybe make this a little easier. Most ESXi users these days in the enterprise space have long been spoiled and forgotten the ways of the command line. Maybe talking the initiative at the upgrade or install process to follow your own documentation to pre-create those interface files would be the best option.

 

[InfinitiGuu]

Found this thread as I encountered the same issue. The NIC in my system is the Broadcom 57416 2 port 10Gb and when I boot with the 6.8.4-2-pve kernel it no longer gets any connection to the network. If I boot the previous kernel 6.5.13-5-pve it works. Is there a fix?

 

[tsprox]

Same issue here, even after overriding network device names (eno1 > envbr0, eno2 > envbr1) and modifying /etc/network/interfaces accordingly like here .
it only works if I boot 6.5.13-5-pve kernel , but no success with 6.8.4-2-pve.



More annoying is that it also affects the remote IPMI interface and I have to do all the tests directly on the console

 

[InfinitiGuu]

Same issue here, even after overriding network device names (eno1 > envbr0, eno2 > envbr1) and modifying /etc/network/interfaces accordingly like here .
it only works if I boot 6.5.13-5-pve kernel , but no success with 6.8.4-2-pve.

View attachment 66969

More annoying is that it also affects the remote IPMI interface and I have to do all the tests directly on the console

I also made the same changes as recommended but still no success with 6.8.4 kernel.

 

[fabian]

that sounds like a different issue? IPMI should not be affected at all by the kernel version you boot, it's a separate system inside your system that is indendent from your OS.. please describe in detail (including: hardware, logs, and configuration) what works and doesn't work..

 

[D4M]

I wrote a quick script that will create systemd.link files to keep the current network interface naming scheme. This will need ran prior to upgrading to PVE 8.2 or PBS 3.2.

https://github.com/D4M4EVER/Proxmox_Preserve_Network_Names

 

[tsprox]

I wrote a quick script that will create systemd.link files to keep the current network interface naming scheme. This will need ran prior to upgrading to PVE 8.2 or PBS 3.2.

https://github.com/D4M4EVER/Proxmox_Preserve_Network_Names

Nice work !
I only have one small observation though - custom link files should be named <n>-<id>.link, where n is a priority smaller than 99 - in your script is exactly 99.

 

[InfinitiGuu]

Same issue here, even after overriding network device names (eno1 > envbr0, eno2 > envbr1) and modifying /etc/network/interfaces accordingly like here .
it only works if I boot 6.5.13-5-pve kernel , but no success with 6.8.4-2-pve.

View attachment 66969

More annoying is that it also affects the remote IPMI interface and I have to do all the tests directly on the console

Looks like my issue was with the firmware level of the Broadcom 57416 10Gbase-T card. Was previously running firmware version 222.1.73.0 and when I upgraded to the latest 227.1.115.0 the networking started working. Strange that it only broke with the new 6.8 kernel.

 

[marcusone]

Proxmox working on a perminent, built in, fix for this?

 

[aempf]

I wrote a quick script that will create systemd.link files to keep the current network interface naming scheme. This will need ran prior to upgrading to PVE 8.2 or PBS 3.2.

https://github.com/D4M4EVER/Proxmox_Preserve_Network_Names

Good, It works perfect, thank you very much