[SOLVED] Latest systemd/udev breaks network device naming!!

[sigo]

After updating to latest systemd* packages:
libsystemd0:amd64 (232-25+deb9u9, 232-25+deb9u11),
udev:amd64 (232-25+deb9u9, 232-25+deb9u11),
libudev1:amd64 (232-25+deb9u9, 232-25+deb9u11),
systemd-sysv:amd64 (232-25+deb9u9, 232-25+deb9u11),
libpam-systemd:amd64 (232-25+deb9u9, 232-25+deb9u11),
systemd:amd64 (232-25+deb9u9, 232-25+deb9u11)

I have rebooted server. And lost my network interface. Instead default consistent name enp0s31f6 I got eth0, and server was unavailable from internet, so I requested a KVM to server and edit /etc/network/interfaces.

I don't use a net.ifnames=0 in my /etc/default/grub.

And this a part of kern.log:

Apr 9 20:44:39 pixie systemd-modules-load[1356]: Inserted module 'iscsi_tcp'
Apr 9 20:44:39 pixie kernel: [ 0.000000] Linux version 4.15.18-12-pve (build@pve) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)) #1 SMP PVE 4.15.18-35 (Wed, 13 Mar 2019 08:24:42 +0100) ()
Apr 9 20:44:39 pixie systemd-modules-load[1356]: Inserted module 'ib_iser'
Apr 9 20:44:39 pixie kernel: [ 0.000000] Command line: BOOT_IMAGE=/ROOT/pve-1@/boot/vmlinuz-4.15.18-12-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet
....
Apr 9 20:44:39 pixie kernel: [ 1.607905] e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) 30:9c:23:ac:5a:50
Apr 9 20:44:39 pixie kernel: [ 1.607906] e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
Apr 9 20:44:39 pixie kernel: [ 1.607980] e1000e 0000:00:1f.6 eth0: MAC: 12, PHY: 12, PBA No: FFFFFF-0FF

 

[Stoiko Ivanov]

cannot reproduce the issue here - after installing the updates my interfaces keep their name) - please
* post the complete `dmesg` (usually the rename happens a few lines below the module detecting and initializing the nic)
* post the output of `ip link`
* also take a look at the journal during boot

 

[Faris Raouf]

A user reported a similar issue via the mailing list 20/03/19 after he upgraded to PVE 4.15.18-35. I'm not sure if a reason was discovered for that or not.

 

[sigo]

I had same kernel before upgrade of systemd/udev packages and no problem with reboot the host

 

[sigo]

cannot reproduce the issue here - after installing the updates my interfaces keep their name) - please
* post the complete `dmesg` (usually the rename happens a few lines below the module detecting and initializing the nic)
* post the output of `ip link`
* also take a look at the journal during boot

I checked the old logs (kern.log and syslog) with previous bootings (correct and failed, these files are attached), and it seems that after updating the packages the kernel(systemd) stopped renaming eth0 (default detected device name) into a consistent name.

 

[Stoiko Ivanov]

* do you have any files in `/etc/udev` (`find /etc/udev`) which could be the reason for the missing rename?
* do you have any non-standard files in `/etc/systemd` ?
* `udevadm info /sys/class/net/eth0` (replace with predicatble name in the working config) - should provide some information - e.g. which file it got the name from

 

[sigo]

* do you have any files in `/etc/udev` (`find /etc/udev`) which could be the reason for the missing rename?
* do you have any non-standard files in `/etc/systemd` ?
* `udevadm info /sys/class/net/eth0` (replace with predicatble name in the working config) - should provide some information - e.g. which file it got the name from

root@pixie:~# cat /etc/udev/udev.conf
# see udev.conf(5) for details
#
# udevd is started in the initramfs, so when this file is modified the
# initramfs should be rebuilt.

#udev_log="info"
root@pixie:~# ls -alFRh /etc/udev/
/etc/udev/:
total 15K
drwxr-xr-x  4 root root   5 Apr  9 20:41 ./
drwxr-xr-x 92 root root 173 Apr 11 20:43 ../
drwxr-xr-x  2 root root   2 Dec  3  2017 hwdb.d/
drwxr-xr-x  2 root root   2 Dec  3  2017 rules.d/
-rw-r--r--  1 root root 153 Dec  3  2017 udev.conf

/etc/udev/hwdb.d:
total 1.0K
drwxr-xr-x 2 root root 2 Dec  3  2017 ./
drwxr-xr-x 4 root root 5 Apr  9 20:41 ../

/etc/udev/rules.d:
total 1.0K
drwxr-xr-x 2 root root 2 Dec  3  2017 ./
drwxr-xr-x 4 root root 5 Apr  9 20:41 ../
root@pixie:~#

It seems problem may be here:

root@pixie:~# cat /etc/systemd/network/99-default.link
# Disable warnings: Could not generate persistent MAC address
[Match]
Path=/devices/virtual/net/*

[Link]
NamePolicy=kernel database onboard slot path
MACAddressPolicy=none

I had add this file a few weeks ago to avoid warning messages in syslog "Could not generate persistent MAC address for veth****". Before latest update of systemd* all worked fine. I can't check now on production server, but may be latest systemd ignore Match section?

root@pixie:~# udevadm info /sys/class/net/eth0
P: /devices/pci0000:00/0000:00:1f.6/net/eth0
E: DEVPATH=/devices/pci0000:00/0000:00:1f.6/net/eth0
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=Ethernet Connection (2) I219-LM
E: ID_MODEL_ID=0x15b7
E: ID_NET_DRIVER=e1000e
E: ID_NET_NAME_MAC=enx309c23ac5a50
E: ID_NET_NAME_PATH=enp0s31f6
E: ID_PATH=pci-0000:00:1f.6
E: ID_PATH_TAG=pci-0000_00_1f_6
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Intel Corporation
E: ID_VENDOR_ID=0x8086
E: IFINDEX=2
E: INTERFACE=eth0
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/eth0
E: TAGS=:systemd:
E: USEC_INITIALIZED=1627100
E: net.ifnames=0

Last line (net.ifname=0) was added to resolve server booting failure.

 

[Stoiko Ivanov]

* Does the problem persist if you remove the file '/etc/systemd/network/99-default.link'
* If not - does it persist if you rename the file (e.g. 98-localdefault.link)?
* I checked the changelog and did not see any explicit mention of a change with the order of file inclusion - but it still could happen.

 

[sigo]

* Does the problem persist if you remove the file '/etc/systemd/network/99-default.link'
* If not - does it persist if you rename the file (e.g. 98-localdefault.link)?
* I checked the changelog and did not see any explicit mention of a change with the order of file inclusion - but it still could happen.

This is production server, I will can try to check your suggestions later.

 

[sigo]

* Does the problem persist if you remove the file '/etc/systemd/network/99-default.link'
* If not - does it persist if you rename the file (e.g. 98-localdefault.link)?
* I checked the changelog and did not see any explicit mention of a change with the order of file inclusion - but it still could happen.

I finally had the opportunity to check the questions - problem disappear after removing 99-default.link and reappear again if this file (99-default.link) stored in /etc/systemd/network/

Checked on latest Proxmox 6.0 with 5.0.18 kernel.

 

[Stoiko Ivanov]

Glad the issue is resolved - Thanks for reporting back!
Please mark the post as 'SOLVED' so that others know what to expect.