[SOLVED] No network after dist-upgrade

[stanbsky]

I performed the following dist-upgrade which included a new kernel. After reboot, I was unable to ssh into or ping the server.

Spoiler: /etc/apt/history.log

Start-Date: 2020-05-22 01:02:36
Commandline: apt-get dist-upgrade
Install: pve-kernel-5.4.41-1-pve:amd64 (5.4.41-1, automatic)
Upgrade: pve-kernel-5.4:amd64 (6.2-1, 6.2-2), libisccfg163:amd64 (1:9.11.5.P4+dfsg-5.1, 1:9.11.5.P4+dfsg-5.1+deb10u1), libpve-storage-perl:amd64 (6.1-7, 6.1-8), bind9-host:amd64 (1:9.11.5.P4+dfsg-5.1, 1:9.11.5.P4+dfsg-5.1+deb10u1), libproxmox-acme-perl:amd64 (1.0.3, 1.0.4), pve-container:amd64 (3.1-5, 3.1-6), zfs-zed:amd64 (0.8.3-pve1, 0.8.4-pve1), libisc-export1100:amd64 (1:9.11.5.P4+dfsg-5.1, 1:9.11.5.P4+dfsg-5.1+deb10u1), libisc1100:amd64 (1:9.11.5.P4+dfsg-5.1, 1:9.11.5.P4+dfsg-5.1+deb10u1), zfsutils-linux:amd64 (0.8.3-pve1, 0.8.4-pve1), libzfs2linux:amd64 (0.8.3-pve1, 0.8.4-pve1), liblwres161:amd64 (1:9.11.5.P4+dfsg-5.1, 1:9.11.5.P4+dfsg-5.1+deb10u1), libdns-export1104:amd64 (1:9.11.5.P4+dfsg-5.1, 1:9.11.5.P4+dfsg-5.1+deb10u1), pve-kernel-helper:amd64 (6.2-1, 6.2-2), libzpool2linux:amd64 (0.8.3-pve1, 0.8.4-pve1), libisccc161:amd64 (1:9.11.5.P4+dfsg-5.1, 1:9.11.5.P4+dfsg-5.1+deb10u1), libbind9-161:amd64 (1:9.11.5.P4+dfsg-5.1, 1:9.11.5.P4+dfsg-5.1+deb10u1), libnvpair1linux:amd64 (0.8.3-pve1, 0.8.4-pve1), libuutil1linux:amd64 (0.8.3-pve1, 0.8.4-pve1), libdns1104:amd64 (1:9.11.5.P4+dfsg-5.1, 1:9.11.5.P4+dfsg-5.1+deb10u1)
End-Date: 2020-05-22 01:03:42

My first instinct was that there was some sort of incompatibility with the new kernel. I chrooted into the system from systemrescuecd and tried booting the previous functioning kernel 5.4.34-1 as well as 4.19.0-9 (only two other kernels I have installed), but this didn't help.

My next thought was that the naming scheme for the network interfaces changed, thereby breaking static ip setup. I created the following file to pin interface name by MAC address:

Spoiler: /etc/systemd/network/10-enp0s8.link

[Match]
MACAddress=00:25:90:11:df:ea

[Link]
Name=enp0s8

Looking at the logs, this was probably never an issue because of:

Spoiler: /var/log/syslog

May 22 01:23:39 comet kernel: [ 7.317892] forcedeth 0000:00:08.0 enp0s8: renamed from eth0
May 22 01:23:39 comet kernel: [ 7.332806] forcedeth 0000:00:09.0 enp0s9: renamed from eth1

I presume this means the naming scheme hasn't changed.

I'm pretty much stumped. I can't see any packages from the log above that would seem likely to mess up networking. Maybe somebody on this forum can?

If no one has any ideas about this, I guess I'll ask my host to fit KVM over IP and reinstall PVE from scratch. That would be a pity because then I'll be none the wiser and even more worried before the next time I have to upgrade...

 

[Moayad]

Hi,

Could you please post the following commands:

~$ cat /etc/network/interfaces

~$ ip a

~$ pveversion -v

 

[stanbsky]

Thank you for your reply.

The output of ip a would suggest the interface names are not being changed to their predictive names. Although I would presume this is a result of me being chrooted into the system.

root@sysresccd:/# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface enp0s8 inet static
        address xxx.xx.xxx.82/30
        gateway xxx.xx.xxx.81
        dns-nameservers 1.1.1.1 8.8.8.8 8.8.4.4
        dns-search mydomain.com
# dns-* options are implemented by the resolvconf package, if installed

iface enp0s9 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.10.10.1
        netmask 255.255.255.0
        bridge-ports none
        bridge-stp off
        bridge-fd 0

        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o enp0s8 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o enp0s8 -j MASQUERADE
root@sysresccd:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:25:90:11:df:ea brd ff:ff:ff:ff:ff:ff
    inet xxx.xx.xxx.82/30 brd xxx.xx.xxx.83 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::225:90ff:fe11:dfea/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 00:25:90:11:df:eb brd ff:ff:ff:ff:ff:ff
root@sysresccd:/# pveversion -v
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
proxmox-ve: 6.2-1 (running kernel: 4.9.47-std510-amd64)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-2
pve-kernel-helper: 6.2-2
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 2.0.1-1+pve8
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-6
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

 

[Moayad]

The output of ip a would suggest the interface names are not being changed to their predictive names. Although I would presume this is a result of me being chrooted into the system.

Try to edit file /etc/network/interfaces and just change name interfaces that changed at startup

 

[stanbsky]

I've just tried your suggestion. There was no change.
Not surprising, given the syslog reporting

May 26 18:12:49 comet kernel: [    7.332797] forcedeth 0000:00:08.0 enp0s8: renamed from eth0

 

[spirit]

It's missing

"auto enp0s8"

 

[stanbsky]

Omg... You're absolutely right.
I doubt the upgrade had anything to do with it then. More likely, I've not rebooted the server since I've recently installed it and so never had the chance to run into trouble.
Thank you very much for spotting this @spirit!