Network Problems after Upgrade from 8.0.4 to 8.1.3

[BRC]

After the upgrade, the only interface holding an IP address is the loopback device. All others are lost. This happened on my test, standalone and also my integration environment (being clustered with ceph and using VLANs).

I start with the standalone scenario (should be easiest to debug)

- Single Server being updated from 8.0.4 to 8.1.3
- Network changes:

--- /etc/network/interfaces 2017-06-17 09:21:48.348000000 +0200
+++ /etc/network/interfaces.new 2024-01-02 13:17:37.674468356 +0100
@@ -1,11 +1,26 @@
+# network interface settings; autogenerated
+# Please do NOT modify this file directly, unless you know what
+# you're doing.
+#
+# If you want to manage parts of the network configuration manually,
+# please utilize the 'source' or 'source-directory' directives to do
+# so.
+# PVE will preserve these directives, but will NOT read its network
+# configuration from sourced files, so do not attempt to move any of
+# the PVE managed interfaces into external files!
+
auto lo
iface lo inet loopback

+iface eth0 inet manual
+
+iface eth1 inet manual
+
auto vmbr0
iface vmbr0 inet static
- address 192.168.32.171
- netmask 255.255.255.0
+ address 192.168.32.171/24
gateway 192.168.32.234
- bridge_ports eth0
- bridge_stp off
- bridge_fd 0
+ bridge-ports eth0
+ bridge-stp off
+ bridge-fd 0
+

After the upgrade, there is no vmbr0 and no ip address besides 127.0.0.1 on the loopback device.

The status in my more complicated environments ist pretty the same.

Even after searching some threads here, googling on the Internet, I have no idea what might have happened.

Any ideas, THANK YOU?

 

[shanreich]

Is it possible that the names of the interfaces changed with the upgrade? That happens quite commonly. Check via the following command: ip a - additionally you can check the networking service for any clues (systemctl status networking)

 

[BRC]

Thank You for the suggestions:
- Interfaces were not renamed
- ohoh, networking service was disabled (I re-enabled it and it looks promising...)

But why is the networking service disabled by the upgrade?
And btw. the disabled networking service is also the cause for "ifreload -a" to fail with: "Another instance is running" (also not very obvious).

I'll try to fix my environments and send an update to this thread.

Thank You

 

[BRC]

As promised, here is my follow-up:
My local cluster is completely fine with this /etc/network/interfaces:
--
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

iface eth2 inet manual

iface eth3 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.32.207/24
gateway 192.168.32.234
bridge-ports eth0
bridge-stp off
bridge-fd 0
#smu-trust

auto vmbr1
iface vmbr1 inet static
address 192.168.46.3/24
bridge-ports eth1
bridge-stp off
bridge-fd 0
#smu-clu

auto vmbr2
iface vmbr2 inet static
address 192.168.45.3/24
bridge-ports eth2
bridge-stp off
bridge-fd 0
#smu-san

auto vmbr3
iface vmbr3 inet manual
bridge-ports eth3
bridge-stp off
bridge-fd 0

source /etc/network/interfaces.d/*
--

But my remote (unfortunately) cluster cannot start networking.service and therefore does not come up.
So far I have not found the reason for this behaviour. Here is the /etc/network/interfaces of a remote node:
--
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface lo inet6 loopback

auto eno1
iface eno1 inet static
address 135.181.163.247/26
gateway 135.181.163.193
up route add -net 135.181.163.192 netmask 255.255.255.192 gw 135.181.163.193 dev eno1
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
#EXT

iface eno1 inet6 static
address 2a01:4f9:3a:1b48::2/64
gateway fe80::1

auto eno1.4000
iface eno1.4000 inet static
address 192.168.240.10/24
mtu 1400
#clu-link2

auto eno1.4001
iface eno1.4001 inet static
address 192.168.241.10/24
mtu 1400
#clu-link1

auto eno1.4002
iface eno1.4002 inet static
address 192.168.242.10/24
mtu 1400
#clu-link0

auto vmbr0
iface vmbr0 inet static
address 192.168.250.254/24
bridge-ports none
bridge-stp off
bridge-fd 0
#INT0

auto vmbr1
iface vmbr1 inet static
address 192.168.251.254/24
bridge-ports none
bridge-stp off
bridge-fd 0
#INT1

auto vmbr2
iface vmbr2 inet static
address 192.168.252.254/24
bridge-ports none
bridge-stp off
bridge-fd 0
#INT2

auto vmbr3
iface vmbr3 inet static
address 192.168.253.10/24
bridge-ports eno1.4004
bridge-stp off
bridge-fd 0
#MGMT

source /etc/network/interfaces.d/*
--

This might be the reason why networking.service was disabled during the first reboot. But I cannot see the problem now..

Any help would be appreciated.

THANK YOU

 

[basil]

1. check if ifupdown2 is installed

dpkg -l | grep ifupdown2

2. vlan 4000 and mtu 1400 - your server on hetzner?
you must remember the limit of 32 mac addresses in one vlan

 

[BRC]

1) yes ifupdown2 is installed and working, "ifreload" -a and "apply changes" works on every node
2) yes it is and I remember the limit, no problem

My problem is:
The networking.service (systemd service unit) is not starting up (runs forever and does not finish) during normal system boot.
When I disable the service ("systemctl disable networking.service"), reboot the system and do a manual "systemctl start networking.service" the networking service starts (no other changes in configurations). As soon as I enable the service again and the system reboots, it does not come up.

I read other threads, analyzed the logs but cannot find the problem till now... I think it is a dependency related to the networking service, but it is just a guess...
In adddition (not related to this issue), my nodes reboot randomly (I could live with this alone, but also have to face it later) which makes this problem even more urgent.

Well, I'm going to read my logs and the man pages (again), but any help would be great..

Thanks to all who try.

 

[basil]

1. all the problems that I had with the network after the updates were corrected by replacing ifupdown with ifupdown2
2. after reboot ifconfig show interfaces from config?
3. I don't see any problems with the network configuration. I would reinstall proxmox
symple way
# crontab -e
@reboot sleep 60; systemctl start networking.service

 

[BRC]

As far as I can see, the update to ifupdown2 also worked on my machines. All test I made were successfull.
All important interfaces show up by an ifconfig and communication is fine. Although the whole interface listing is a mess and not identical betweeen all cluster nodes.
A new installation is not an option ;-), I have to fix the problem.
But Your idea to use crontab as a workaround will ease my pain (I have not tried it yet).

The main question remains: What is the difference between starting the networking service at boot time by enabling it and starting the networking service interactively later...?

I have no idea. THANK YOU so much so far.

 

[shanreich]

Are the logs giving any indicator as to why it fails?

journalctl -b -u networking

 

[basil]

The main question remains: What is the difference between starting the networking service at boot time by enabling it and starting the networking service interactively later...?

i need logs - I'm not a magician

 

[BRC]

I analyzed the logs myself so far but I could not find the cause of the problem...
Here is the result of a rebooted node with the crontab workaround, all looks fine:

root@herz ~ # uptime
13:11:40 up 3 min, 2 users, load average: 0.59, 0.47, 0.20

root@herz ~ # journalctl -b -u networking
Jan 08 13:09:46 herz systemd[1]: Starting networking.service - Network initialization...
Jan 08 13:09:46 herz networking[2097]: networking: Configuring network interfaces
Jan 08 13:09:53 herz ntpdate[2600]: 2024-01-08 13:09:52.32802 (+0100) +0.160012 +/- 0.045780 1.debian.pool.ntp.org 131.234.220.232 s1 no-leap
Jan 08 13:09:53 herz systemd[1]: Finished networking.service - Network initialization.

The node came up and all is working (due to basil's idea). I can life with it but it is not the final solution.
I try to deliver logs for the case of the hanging networking service. The problem is: This only occurs when the networking service is enabled whch itself prevents a successfull system start and I cannot get a command prompt...

That is the reason why I assked this stupid question: What is the difference between an enabled service during boot and the same service started manually? Normally I would say: If systemctl start XXX works, this service should come up automatically when enabled.

 

[spirit]

Hi, you have ifupdown2 (including start log), in /var/log/ifupdown2/network_config_ifupdown2_*

 

[fweber]

Hi, can you check whether both ntpsec and ntpsec-ntpdate are installed? If yes, this could cause networking to hang during boot, see bug #5009 for more details [1]. This should be fixed in ifupdown2 >= 3.2.0-1+pmx8 which is available on pve-no-subscription as of yesterday. Can you check whether installing this version fixes the hang?

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=5009

 

[BRC]

Thank You to all the supporters. fweber's hint helped. During my analysis of the logs I could see some problems related to NTP but I didn't see the relation to my networking problem.
I updated a node, removed the cron entry, re-enabled networking service and als removed the ntpsec-ntpdate package.
The node came back up again. I will also change the config on all others nodes....

THANK YOU

 

[joem]

I am struggling with a similar issue.
After upgrading from 7.4 to 8.1 I lost network connectivity.
I am running proxmox on a Supermicro x9dri-ln4f+, with i350 NICs
ntpsec is not installed
See below the error I am getting.

 

[basil]

I am struggling with a similar issue.
After upgrading from 7.4 to 8.1 I lost network connectivity.
I am running proxmox on a Supermicro x9dri-ln4f+, with i350 NICs
ntpsec is not installed
See below the error I am getting.

your sysytem dont see network interface

check iface names
ip link show

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP mode DEFAULT group default qlen 1000
link/ether 0c:9d:92:64:d9:94 brd ff:ff:ff:ff:ff:ff
altname enp0s31f6

fix eno4 to correct name
/etc/network/interfaces