Network outage after "apt dist-upgrade" and reboot on proxmox 6.1-7

[pete20]

Hi,

I have a 2 node proxmox cluster on version 6.1-7 with a community Subscription.

I migrated critical VMs from node2 to node1, shutdown all other VMs on node2 and then proceeded to run an "apt update && apt dist-upgrade".

The output of this was as follows:

Reading package lists... Done
Building dependency tree      
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  pve-kernel-5.3.13-1-pve pve-kernel-5.3.13-3-pve python-argcomplete
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  pve-kernel-5.4.44-1-pve
The following packages will be upgraded:
  ifupdown2 libproxmox-backup-qemu0 libpve-common-perl linux-libc-dev lxcfs proxmox-widget-toolkit pve-container pve-edk2-firmware pve-i18n pve-kernel-5.4 pve-kernel-helper pve-manager pve-qemu-kvm qemu-server
14 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 89.7 MB of archives.
After this operation, 289 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://security.debian.org buster/updates/main amd64 linux-libc-dev amd64 4.19.118-2+deb10u1 [1,354 kB]
Get:2 https://enterprise.proxmox.com/debian/pve buster/pve-enterprise amd64 ifupdown2 all 3.0.0-1+pve2 [226 kB]
Get:3 https://enterprise.proxmox.com/debian/pve buster/pve-enterprise amd64 libpve-common-perl all 6.1-3 [98.7 kB]
Get:4 https://enterprise.proxmox.com/debian/pve buster/pve-enterprise amd64 libproxmox-backup-qemu0 amd64 0.1.6-1 [1,472 kB]
Get:5 https://enterprise.proxmox.com/debian/pve buster/pve-enterprise amd64 lxcfs amd64 4.0.3-pve3 [64.4 kB]
Get:6 https://enterprise.proxmox.com/debian/pve buster/pve-enterprise amd64 proxmox-widget-toolkit all 2.2-7 [57.5 kB]
Get:7 https://enterprise.proxmox.com/debian/pve buster/pve-enterprise amd64 pve-container all 3.1-8 [121 kB]
Get:8 https://enterprise.proxmox.com/debian/pve buster/pve-enterprise amd64 pve-edk2-firmware all 2.20200531-1 [1,710 kB]
Get:9 https://enterprise.proxmox.com/debian/pve buster/pve-enterprise amd64 pve-i18n all 2.1-3 [110 kB]
Get:10 https://enterprise.proxmox.com/debian/pve buster/pve-enterprise amd64 pve-kernel-5.4.44-1-pve amd64 5.4.44-1 [60.3 MB]
Get:11 https://enterprise.proxmox.com/debian/pve buster/pve-enterprise amd64 pve-kernel-5.4 all 6.2-3 [3,452 B]
Get:12 https://enterprise.proxmox.com/debian/pve buster/pve-enterprise amd64 pve-kernel-helper all 6.2-3 [9,368 B]
Get:13 https://enterprise.proxmox.com/debian/pve buster/pve-enterprise amd64 pve-qemu-kvm amd64 5.0.0-4 [22.0 MB]
Get:14 https://enterprise.proxmox.com/debian/pve buster/pve-enterprise amd64 qemu-server amd64 6.2-3 [219 kB]
Get:15 https://enterprise.proxmox.com/debian/pve buster/pve-enterprise amd64 pve-manager amd64 6.2-6 [2,014 kB]
Fetched 89.7 MB in 1s (73.8 MB/s)      
Reading changelogs... Done
(Reading database ... 83793 files and directories currently installed.)
Preparing to unpack .../00-ifupdown2_3.0.0-1+pve2_all.deb ...
Unpacking ifupdown2 (3.0.0-1+pve2) over (2.0.1-1+pve8) ...
Preparing to unpack .../01-libpve-common-perl_6.1-3_all.deb ...
Unpacking libpve-common-perl (6.1-3) over (6.1-2) ...
Preparing to unpack .../02-libproxmox-backup-qemu0_0.1.6-1_amd64.deb ...
Unpacking libproxmox-backup-qemu0 (0.1.6-1) over (0.1.3-1) ...
Preparing to unpack .../03-linux-libc-dev_4.19.118-2+deb10u1_amd64.deb ...
Unpacking linux-libc-dev:amd64 (4.19.118-2+deb10u1) over (4.19.118-2) ...
Preparing to unpack .../04-lxcfs_4.0.3-pve3_amd64.deb ...
Unpacking lxcfs (4.0.3-pve3) over (4.0.3-pve2) ...
Preparing to unpack .../05-proxmox-widget-toolkit_2.2-7_all.deb ...
Unpacking proxmox-widget-toolkit (2.2-7) over (2.2-1) ...
Preparing to unpack .../06-pve-container_3.1-8_all.deb ...
Unpacking pve-container (3.1-8) over (3.1-6) ...
Preparing to unpack .../07-pve-edk2-firmware_2.20200531-1_all.deb ...
Unpacking pve-edk2-firmware (2.20200531-1) over (2.20200229-1) ...
Preparing to unpack .../08-pve-i18n_2.1-3_all.deb ...
Unpacking pve-i18n (2.1-3) over (2.1-2) ...
Selecting previously unselected package pve-kernel-5.4.44-1-pve.
Preparing to unpack .../09-pve-kernel-5.4.44-1-pve_5.4.44-1_amd64.deb ...
Unpacking pve-kernel-5.4.44-1-pve (5.4.44-1) ...
Preparing to unpack .../10-pve-kernel-5.4_6.2-3_all.deb ...
Unpacking pve-kernel-5.4 (6.2-3) over (6.2-2) ...
Preparing to unpack .../11-pve-kernel-helper_6.2-3_all.deb ...
Unpacking pve-kernel-helper (6.2-3) over (6.2-2) ...
Preparing to unpack .../12-pve-qemu-kvm_5.0.0-4_amd64.deb ...
Unpacking pve-qemu-kvm (5.0.0-4) over (5.0.0-2) ...
Preparing to unpack .../13-qemu-server_6.2-3_amd64.deb ...
Unpacking qemu-server (6.2-3) over (6.2-2) ...
Preparing to unpack .../14-pve-manager_6.2-6_amd64.deb ...
Unpacking pve-manager (6.2-6) over (6.2-4) ...
Setting up proxmox-widget-toolkit (2.2-7) ...
Setting up lxcfs (4.0.3-pve3) ...
Setting up linux-libc-dev:amd64 (4.19.118-2+deb10u1) ...
Setting up pve-kernel-helper (6.2-3) ...
Setting up libpve-common-perl (6.1-3) ...
Setting up ifupdown2 (3.0.0-1+pve2) ...
Installing new version of config file /etc/default/networking ...
Installing new version of config file /etc/network/ifupdown2/addons.conf ...
Use of uninitialized value in string ne at /usr/share/perl5/PVE/INotify.pm line 1622.
iface eno1np0 - ip address can't be set on interface if bridged in vmbr2
Setting up pve-i18n (2.1-3) ...
Setting up pve-kernel-5.4.44-1-pve (5.4.44-1) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.4.44-1-pve /boot/vmlinuz-5.4.44-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.4.44-1-pve /boot/vmlinuz-5.4.44-1-pve
update-initramfs: Generating /boot/initrd.img-5.4.44-1-pve
run-parts: executing /etc/kernel/postinst.d/pve-auto-removal 5.4.44-1-pve /boot/vmlinuz-5.4.44-1-pve
run-parts: executing /etc/kernel/postinst.d/zz-pve-efiboot 5.4.44-1-pve /boot/vmlinuz-5.4.44-1-pve
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/D34A-A280
    Copying kernel and creating boot-entry for 5.3.18-2-pve
    Copying kernel and creating boot-entry for 5.3.18-3-pve
    Copying kernel and creating boot-entry for 5.4.41-1-pve
    Copying kernel and creating boot-entry for 5.4.44-1-pve
Copying and configuring kernels on /dev/disk/by-uuid/D34A-C994
    Copying kernel and creating boot-entry for 5.3.18-2-pve
    Copying kernel and creating boot-entry for 5.3.18-3-pve
    Copying kernel and creating boot-entry for 5.4.41-1-pve
    Copying kernel and creating boot-entry for 5.4.44-1-pve
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 5.4.44-1-pve /boot/vmlinuz-5.4.44-1-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.4.44-1-pve
Found initrd image: /boot/initrd.img-5.4.44-1-pve
Found linux image: /boot/vmlinuz-5.4.41-1-pve
Found initrd image: /boot/initrd.img-5.4.41-1-pve
Found linux image: /boot/vmlinuz-5.3.18-3-pve
Found initrd image: /boot/initrd.img-5.3.18-3-pve
Found linux image: /boot/vmlinuz-5.3.18-2-pve
Found initrd image: /boot/initrd.img-5.3.18-2-pve
Found linux image: /boot/vmlinuz-5.3.13-3-pve
Found initrd image: /boot/initrd.img-5.3.13-3-pve
Found linux image: /boot/vmlinuz-5.3.13-1-pve
Found initrd image: /boot/initrd.img-5.3.13-1-pve
Found linux image: /boot/vmlinuz-5.3.10-1-pve
Found initrd image: /boot/initrd.img-5.3.10-1-pve
Found memtest86+ image: /ROOT/pve-1@/boot/memtest86+.bin
Found memtest86+ multiboot image: /ROOT/pve-1@/boot/memtest86+_multiboot.bin
done
Setting up pve-edk2-firmware (2.20200531-1) ...
Setting up pve-kernel-5.4 (6.2-3) ...
Setting up libproxmox-backup-qemu0 (0.1.6-1) ...
Setting up pve-container (3.1-8) ...
system-pve\x2dcontainer.slice is a disabled or a static unit, not starting it.
Setting up pve-qemu-kvm (5.0.0-4) ...
Setting up qemu-server (6.2-3) ...
Setting up pve-manager (6.2-6) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for pve-ha-manager (3.0-9) ...
Processing triggers for mime-support (3.62) ...
Processing triggers for libc-bin (2.28-10) ...
Processing triggers for systemd (241-7~deb10u4) ...

I noticed the following lines that looked ominous:

Setting up ifupdown2 (3.0.0-1+pve2) ...
Installing new version of config file /etc/default/networking ...
Installing new version of config file /etc/network/ifupdown2/addons.conf ...
Use of uninitialized value in string ne at /usr/share/perl5/PVE/INotify.pm line 1622.

After the update completed I rebooted the system. However, to my horror, no network interfaces would come up and I had to access the system via the IPMI KVM.

If I inspect the logs for the networking.service I can see that they have the following errors:
error: netlink: vmbr0: cannot add address <ip address> dev vmbr0: 'str' object has no attribute 'packed'

As expected, this issue has meant I can make no changes to my cluster, though thankfully node1 is still operational and its VMs active.

Can anyone please advise what I can do to resolve this problem?

Please let me know if there is any additional info I can provide to help resolve this.

 

[pete20]

An update on my issue in case it helps anyone in a similar situation:
I managed to track the problem down to the new ifupdown2 package not being compatible with my existing /etc/network/interfaces file, as it had separately defined network, subnet and broadcast fields rather than using CIDR notation. I converted my /etc/network/interfaces file to use CIDR notation in the address field and removed the now extraneous network, subnet and broadcast fields. My network interfaces then sprang back in to life after doing an "ifreload -a"

 

[Faris Raouf]

I'm very glad you figured that one out. I would never have managed it.

All my systems use separately defined network subnet etc too.

But I thought ifupdown2 was only for the sdn feature, which is in beta? Something to do with rebootless network changes? Or maybe I'm mixing things up/getting confused/misremembering?

I don't have ifupdown2 on my systems (PVE 6.2, upgraded from 5.4). I only have ifupdown.

 

[pete20]

I'm very glad you figured that one out. I would never have managed it.

All my systems use separately defined network subnet etc too.

But I thought ifupdown2 was only for the sdn feature, which is in beta? Something to do with rebootless network changes? Or maybe I'm mixing things up/getting confused/misremembering?

I don't have ifupdown2 on my systems (PVE 6.2, upgraded from 5.4). I only have ifupdown.

Thank you.

I think you're right in that ifupdown2 may not be the default package currently. I have a vague memory of specifically installing it to enable rebootless network changes half a year or so ago. I wouldn't have expected a new version of it to introduce such a breaking change in the config files it accepts though and the lack of clear error messages really exacerbated the situation.

 

[spirit]

@pete20

can you send the /etc/network/interfaces content when it's not working ? (and maybe, do you have configuration before && after upgrade ?)


last proxmox version rewrite /etc/network/interfaces to use cidr when you edit from gui ,but ifupdown2 package upgrade force the rewrite too.

So, I don't think it's an ifupdown2 bug, but maybe the proxmox parser not handling correctly your configuration.