No network after reboot - dependency issue

[thj]

Hello all. I needed to reboot my Proxmox server to get into BIOS and change fan speed settings. After boot I could not longer reach it on static IP. There were no vmbr0 interface available. Network port on my router was also not up. Managed to solve it by restarting networking.service. After restart:

root@proxmox:~# journalctl -u networking.service -f
-- Journal begins at Mon 2021-12-27 16:29:55 CET. --
Jan 15 16:03:10 proxmox systemd[1]: Stopped Network initialization.
-- Boot 2d2296f40e6d473498cb972d99b12992 --
Jan 15 16:09:40 proxmox systemd[1]: Dependency failed for Network initialization.
Jan 15 16:09:40 proxmox systemd[1]: networking.service: Job networking.service/start failed with result 'dependency'.
-- Boot d9514ddd5ee342e484c91b63d8c78757 --
Jan 15 16:21:31 proxmox systemd[1]: Dependency failed for Network initialization.
Jan 15 16:21:31 proxmox systemd[1]: networking.service: Job networking.service/start failed with result 'dependency'.
-- Boot df08ad1fb8724131b8f6471a6182af6f --
Jan 15 16:28:16 proxmox systemd[1]: Dependency failed for Network initialization.
Jan 15 16:28:16 proxmox systemd[1]: networking.service: Job networking.service/start failed with result 'dependency'.
Jan 15 16:36:18 proxmox systemd[1]: Starting Network initialization...
Jan 15 16:36:18 proxmox networking[12591]: networking: Configuring network interfaces
Jan 15 16:36:18 proxmox systemd[1]: Finished Network initialization.

Some output of my settings:

# lspci | grep 'Ethernet'
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.28.70/24
        gateway 192.168.28.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

iface wlp3s0 inet manual

After I restarted the service link went up and I could ping/reach the server again. Proxmox webui was working but all the VM and LXC were stopped so I had to start them all manually.

I kindly ask for advise what could the problem be with network during the restart that caused it not to properly start.

Thank you.

 

[Moayad]

Hello,

Are you using ifupdown2 or ifupdown in your PVE host?

 

[thj]

Looks like I have ifupdown2 package installed. Is this an issue? Proxmox was installed using official Proxmox ISO:

# apt list --installed | grep ifup

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

ifupdown2/stable,now 3.1.0-1+pmx3 all [installed]

 

[Moayad]

Hi,

Looks like I have ifupdown2 package installed. Is this an issue?

No, there is no issue.

did you notice anything in the Syslog during restart time instead of the above journalctl output?

also please provide us with the output of pveversion -v

 

[thj]

Nothing special inside syslog during restart that would stand out regarding network. Just output from networking.service.

# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-8 (running version: 7.1-8/5b267f33)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.11: 7.0-10
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.13.19-1-pve: 5.13.19-3
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 15.2.14-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3

 

[philsson]

Did you solve this? I ran into the same problem after upgrading.

 

[pedroamador]

I had the same issue. Some of the systemd network configuration was vanished, I don't know the root cause.

To solve the issue in my case:

apt-get install --reinstall ifupdown2

This do the trick and configure systemd network related scripts, so the network will start right in the next reboot

 

[philsson]

I had the same issue. Some of the systemd network configuration was vanished, I don't know the root cause.

To solve the issue in my case:

apt-get install --reinstall ifupdown2

This do the trick and configure systemd network related scripts, so the network will start right in the next reboot

I've done this now and removed my ipv6 configuration. At least one reboot I've tried so far I had working network with my interface up. Before en*** showed as UP but vmbr0 was DOWN.
I removed ifupdown to install ifupdown2, but now I don't have the tools `ifup` and `ifdown`. As I understand it ifupdown2 is a direct replacement and should include both these tools. Any clue?

 

[pedroamador]

I understand that you had resolved the issue...

The "ifup" and "idown" comands are part of the "ifupdown2" package

# dpkg -L ifupdown2
/.
/etc
/etc/default
/etc/default/networking
/etc/network
/etc/network/ifupdown2
/etc/network/ifupdown2/addons.conf
/etc/network/ifupdown2/ifupdown2.conf
/etc/network/ifupdown2/policy.d
/etc/network/interfaces.d
/lib
/lib/systemd
/lib/systemd/system
/lib/systemd/system/ifup@.service
/lib/systemd/system/ifupdown2-pre.service
/lib/systemd/system/networking.service
/sbin
/usr
/usr/share
/usr/share/doc
/usr/share/doc/ifupdown2
/usr/share/doc/ifupdown2/README.rst
/usr/share/doc/ifupdown2/changelog.Debian.gz
/usr/share/doc/ifupdown2/copyright
/usr/share/doc/ifupdown2/examples
/usr/share/doc/ifupdown2/examples/batman_adv
/usr/share/doc/ifupdown2/examples/batman_adv/configure_batman_adv.sh
/usr/share/doc/ifupdown2/examples/batman_adv/interfaces_batman
/usr/share/doc/ifupdown2/examples/generate_interfaces.py
/usr/share/doc/ifupdown2/examples/interfaces
/usr/share/doc/ifupdown2/examples/interfaces_bridge_igmp_mstp
/usr/share/doc/ifupdown2/examples/interfaces_bridge_template_func
/usr/share/doc/ifupdown2/examples/interfaces_bridge_vlan_bond_vxlan_vrf
/usr/share/doc/ifupdown2/examples/interfaces_with_template
/usr/share/doc/ifupdown2/examples/mgmt-vrf
/usr/share/doc/ifupdown2/examples/vlan_aware_bridges
/usr/share/doc/ifupdown2/examples/vlan_aware_bridges/interfaces.basic
/usr/share/doc/ifupdown2/examples/vlan_aware_bridges/interfaces.vlan_prune_and_access_ports
/usr/share/doc/ifupdown2/examples/vlan_aware_bridges/interfaces.with_bonds
/usr/share/doc/ifupdown2/examples/vlan_aware_bridges/interfaces.with_clag
/usr/share/ifupdown2
/usr/share/ifupdown2/__init__.py
/usr/share/ifupdown2/__main__.py
/usr/share/ifupdown2/addons
/usr/share/ifupdown2/addons/__init__.py
/usr/share/ifupdown2/addons/address.py
/usr/share/ifupdown2/addons/addressvirtual.py
/usr/share/ifupdown2/addons/batman_adv.py
/usr/share/ifupdown2/addons/bond.py
/usr/share/ifupdown2/addons/bridge.py
/usr/share/ifupdown2/addons/bridgevlan.py
/usr/share/ifupdown2/addons/bridgevlanport.py
/usr/share/ifupdown2/addons/dhcp.py
/usr/share/ifupdown2/addons/ethtool.py
/usr/share/ifupdown2/addons/link.py
/usr/share/ifupdown2/addons/mstpctl.py
/usr/share/ifupdown2/addons/mtu.py
/usr/share/ifupdown2/addons/openvswitch.py
/usr/share/ifupdown2/addons/openvswitch_port.py
/usr/share/ifupdown2/addons/ppp.py
/usr/share/ifupdown2/addons/tunnel.py
/usr/share/ifupdown2/addons/usercmds.py
/usr/share/ifupdown2/addons/vlan.py
/usr/share/ifupdown2/addons/vrf.py
/usr/share/ifupdown2/addons/vrrpd.py
/usr/share/ifupdown2/addons/vxlan.py
/usr/share/ifupdown2/addons/xfrm.py
/usr/share/ifupdown2/ifupdown
/usr/share/ifupdown2/ifupdown/__init__.py
/usr/share/ifupdown2/ifupdown/argv.py
/usr/share/ifupdown2/ifupdown/client.py
/usr/share/ifupdown2/ifupdown/config.py
/usr/share/ifupdown2/ifupdown/exceptions.py
/usr/share/ifupdown2/ifupdown/graph.py
/usr/share/ifupdown2/ifupdown/iface.py
/usr/share/ifupdown2/ifupdown/ifupdownconfig.py
/usr/share/ifupdown2/ifupdown/ifupdownflags.py
/usr/share/ifupdown2/ifupdown/ifupdownmain.py
/usr/share/ifupdown2/ifupdown/main.py
/usr/share/ifupdown2/ifupdown/networkinterfaces.py
/usr/share/ifupdown2/ifupdown/policymanager.py
/usr/share/ifupdown2/ifupdown/scheduler.py
/usr/share/ifupdown2/ifupdown/statemanager.py
/usr/share/ifupdown2/ifupdown/template.py
/usr/share/ifupdown2/ifupdown/utils.py
/usr/share/ifupdown2/ifupdownaddons
/usr/share/ifupdown2/ifupdownaddons/__init__.py
/usr/share/ifupdown2/ifupdownaddons/cache.py
/usr/share/ifupdown2/ifupdownaddons/dhclient.py
/usr/share/ifupdown2/ifupdownaddons/modulebase.py
/usr/share/ifupdown2/ifupdownaddons/mstpctlutil.py
/usr/share/ifupdown2/ifupdownaddons/systemutils.py
/usr/share/ifupdown2/ifupdownaddons/utilsbase.py
/usr/share/ifupdown2/lib
/usr/share/ifupdown2/lib/__init__.py
/usr/share/ifupdown2/lib/addon.py
/usr/share/ifupdown2/lib/base_objects.py
/usr/share/ifupdown2/lib/dry_run.py
/usr/share/ifupdown2/lib/exceptions.py
/usr/share/ifupdown2/lib/io.py
/usr/share/ifupdown2/lib/iproute2.py
/usr/share/ifupdown2/lib/log.py
/usr/share/ifupdown2/lib/nlcache.py
/usr/share/ifupdown2/lib/status.py
/usr/share/ifupdown2/lib/sysfs.py
/usr/share/ifupdown2/nlmanager
/usr/share/ifupdown2/nlmanager/__init__.py
/usr/share/ifupdown2/nlmanager/ipnetwork.py
/usr/share/ifupdown2/nlmanager/nllistener.py
/usr/share/ifupdown2/nlmanager/nlmanager.py
/usr/share/ifupdown2/nlmanager/nlpacket.py
/usr/share/ifupdown2/sbin
/usr/share/ifupdown2/sbin/start-networking
/usr/share/ifupdown2-3.0.0.egg-info
/usr/share/ifupdown2-3.0.0.egg-info/PKG-INFO
/usr/share/ifupdown2-3.0.0.egg-info/dependency_links.txt
/usr/share/ifupdown2-3.0.0.egg-info/top_level.txt
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/ifupdown2
/usr/share/man
/usr/share/man/man5
/usr/share/man/man5/ifupdown-addons-interfaces.5.gz
/usr/share/man/man5/interfaces.5.gz
/usr/share/man/man8
/usr/share/man/man8/ifdown.8.gz
/usr/share/man/man8/ifquery.8.gz
/usr/share/man/man8/ifreload.8.gz
/usr/share/man/man8/ifup.8.gz
/usr/share/python3
/usr/share/python3/runtime.d
/usr/share/python3/runtime.d/ifupdown2.rtupdate
/var
/var/lib
/var/lib/ifupdown2
/var/lib/ifupdown2/hooks
/var/lib/ifupdown2/policy.d
/var/lib/ifupdown2/policy.d/proxmox-bridge-mac-from-port.json
/sbin/ifdown
/sbin/ifquery
/sbin/ifreload
/sbin/ifup
/usr/share/ifupdown2/ifupdown2

 

[philsson]

I understand that you had resolved the issue...

The "ifup" and "idown" comands are part of the "ifupdown2" package

# dpkg -L ifupdown2
/.
/etc
/etc/default
/etc/default/networking
/etc/network
/etc/network/ifupdown2
/etc/network/ifupdown2/addons.conf
/etc/network/ifupdown2/ifupdown2.conf
/etc/network/ifupdown2/policy.d
/etc/network/interfaces.d
/lib
/lib/systemd
/lib/systemd/system
/lib/systemd/system/ifup@.service
/lib/systemd/system/ifupdown2-pre.service
/lib/systemd/system/networking.service
/sbin
/usr
/usr/share
/usr/share/doc
/usr/share/doc/ifupdown2
/usr/share/doc/ifupdown2/README.rst
/usr/share/doc/ifupdown2/changelog.Debian.gz
/usr/share/doc/ifupdown2/copyright
/usr/share/doc/ifupdown2/examples
/usr/share/doc/ifupdown2/examples/batman_adv
/usr/share/doc/ifupdown2/examples/batman_adv/configure_batman_adv.sh
/usr/share/doc/ifupdown2/examples/batman_adv/interfaces_batman
/usr/share/doc/ifupdown2/examples/generate_interfaces.py
/usr/share/doc/ifupdown2/examples/interfaces
/usr/share/doc/ifupdown2/examples/interfaces_bridge_igmp_mstp
/usr/share/doc/ifupdown2/examples/interfaces_bridge_template_func
/usr/share/doc/ifupdown2/examples/interfaces_bridge_vlan_bond_vxlan_vrf
/usr/share/doc/ifupdown2/examples/interfaces_with_template
/usr/share/doc/ifupdown2/examples/mgmt-vrf
/usr/share/doc/ifupdown2/examples/vlan_aware_bridges
/usr/share/doc/ifupdown2/examples/vlan_aware_bridges/interfaces.basic
/usr/share/doc/ifupdown2/examples/vlan_aware_bridges/interfaces.vlan_prune_and_access_ports
/usr/share/doc/ifupdown2/examples/vlan_aware_bridges/interfaces.with_bonds
/usr/share/doc/ifupdown2/examples/vlan_aware_bridges/interfaces.with_clag
/usr/share/ifupdown2
/usr/share/ifupdown2/__init__.py
/usr/share/ifupdown2/__main__.py
/usr/share/ifupdown2/addons
/usr/share/ifupdown2/addons/__init__.py
/usr/share/ifupdown2/addons/address.py
/usr/share/ifupdown2/addons/addressvirtual.py
/usr/share/ifupdown2/addons/batman_adv.py
/usr/share/ifupdown2/addons/bond.py
/usr/share/ifupdown2/addons/bridge.py
/usr/share/ifupdown2/addons/bridgevlan.py
/usr/share/ifupdown2/addons/bridgevlanport.py
/usr/share/ifupdown2/addons/dhcp.py
/usr/share/ifupdown2/addons/ethtool.py
/usr/share/ifupdown2/addons/link.py
/usr/share/ifupdown2/addons/mstpctl.py
/usr/share/ifupdown2/addons/mtu.py
/usr/share/ifupdown2/addons/openvswitch.py
/usr/share/ifupdown2/addons/openvswitch_port.py
/usr/share/ifupdown2/addons/ppp.py
/usr/share/ifupdown2/addons/tunnel.py
/usr/share/ifupdown2/addons/usercmds.py
/usr/share/ifupdown2/addons/vlan.py
/usr/share/ifupdown2/addons/vrf.py
/usr/share/ifupdown2/addons/vrrpd.py
/usr/share/ifupdown2/addons/vxlan.py
/usr/share/ifupdown2/addons/xfrm.py
/usr/share/ifupdown2/ifupdown
/usr/share/ifupdown2/ifupdown/__init__.py
/usr/share/ifupdown2/ifupdown/argv.py
/usr/share/ifupdown2/ifupdown/client.py
/usr/share/ifupdown2/ifupdown/config.py
/usr/share/ifupdown2/ifupdown/exceptions.py
/usr/share/ifupdown2/ifupdown/graph.py
/usr/share/ifupdown2/ifupdown/iface.py
/usr/share/ifupdown2/ifupdown/ifupdownconfig.py
/usr/share/ifupdown2/ifupdown/ifupdownflags.py
/usr/share/ifupdown2/ifupdown/ifupdownmain.py
/usr/share/ifupdown2/ifupdown/main.py
/usr/share/ifupdown2/ifupdown/networkinterfaces.py
/usr/share/ifupdown2/ifupdown/policymanager.py
/usr/share/ifupdown2/ifupdown/scheduler.py
/usr/share/ifupdown2/ifupdown/statemanager.py
/usr/share/ifupdown2/ifupdown/template.py
/usr/share/ifupdown2/ifupdown/utils.py
/usr/share/ifupdown2/ifupdownaddons
/usr/share/ifupdown2/ifupdownaddons/__init__.py
/usr/share/ifupdown2/ifupdownaddons/cache.py
/usr/share/ifupdown2/ifupdownaddons/dhclient.py
/usr/share/ifupdown2/ifupdownaddons/modulebase.py
/usr/share/ifupdown2/ifupdownaddons/mstpctlutil.py
/usr/share/ifupdown2/ifupdownaddons/systemutils.py
/usr/share/ifupdown2/ifupdownaddons/utilsbase.py
/usr/share/ifupdown2/lib
/usr/share/ifupdown2/lib/__init__.py
/usr/share/ifupdown2/lib/addon.py
/usr/share/ifupdown2/lib/base_objects.py
/usr/share/ifupdown2/lib/dry_run.py
/usr/share/ifupdown2/lib/exceptions.py
/usr/share/ifupdown2/lib/io.py
/usr/share/ifupdown2/lib/iproute2.py
/usr/share/ifupdown2/lib/log.py
/usr/share/ifupdown2/lib/nlcache.py
/usr/share/ifupdown2/lib/status.py
/usr/share/ifupdown2/lib/sysfs.py
/usr/share/ifupdown2/nlmanager
/usr/share/ifupdown2/nlmanager/__init__.py
/usr/share/ifupdown2/nlmanager/ipnetwork.py
/usr/share/ifupdown2/nlmanager/nllistener.py
/usr/share/ifupdown2/nlmanager/nlmanager.py
/usr/share/ifupdown2/nlmanager/nlpacket.py
/usr/share/ifupdown2/sbin
/usr/share/ifupdown2/sbin/start-networking
/usr/share/ifupdown2-3.0.0.egg-info
/usr/share/ifupdown2-3.0.0.egg-info/PKG-INFO
/usr/share/ifupdown2-3.0.0.egg-info/dependency_links.txt
/usr/share/ifupdown2-3.0.0.egg-info/top_level.txt
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/ifupdown2
/usr/share/man
/usr/share/man/man5
/usr/share/man/man5/ifupdown-addons-interfaces.5.gz
/usr/share/man/man5/interfaces.5.gz
/usr/share/man/man8
/usr/share/man/man8/ifdown.8.gz
/usr/share/man/man8/ifquery.8.gz
/usr/share/man/man8/ifreload.8.gz
/usr/share/man/man8/ifup.8.gz
/usr/share/python3
/usr/share/python3/runtime.d
/usr/share/python3/runtime.d/ifupdown2.rtupdate
/var
/var/lib
/var/lib/ifupdown2
/var/lib/ifupdown2/hooks
/var/lib/ifupdown2/policy.d
/var/lib/ifupdown2/policy.d/proxmox-bridge-mac-from-port.json
/sbin/ifdown
/sbin/ifquery
/sbin/ifreload
/sbin/ifup
/usr/share/ifupdown2/ifupdown2

It seemed so but I lost network again. The only way for now to get it working was to configure the "en...." Interface and remove vmbr0.
My /sbin
directory does have the ifup and ifdown tools so I wonder why they are not recognized. I've reinstalled it with "apt install --reinstall" but it remains the same.

Edit: "/sbin" is missing from my PATH. Wonder why. Could maybe be the cause of my problems

 

[trent--]

Hi,
Have you switched packages between ifupdown and ifupdown2 ?
I had similar problems with the ifupdown2 package postrm scripts altering /etc/network/interface file and disabling the networking service. Make sure you don't run into the same issue.

 

[philsson]

Hi,
Have you switched packages between ifupdown and ifupdown2 ?
I had similar problems with the ifupdown2 package postrm scripts altering /etc/network/interface file and disabling the networking service. Make sure you don't run into the same issue.

Yes I have. I did this manually to try to get the broken network after upgrade working. My interfaces file was not touched by the upgrade.

 

[proxale]

I had the same issue. Some of the systemd network configuration was vanished, I don't know the root cause.

To solve the issue in my case:

apt-get install --reinstall ifupdown2

This do the trick and configure systemd network related scripts, so the network will start right in the next reboot

I encounter the same problem after a node crash, networking for VMs on the node are not working until migrated to another node.
However, I am unable to reinstall ifupdown2 when I tried. Did you not get the following error?

 apt-get install --reinstall ifupdown2
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Reinstallation of ifupdown2 is not possible, it cannot be downloaded.
0 upgraded, 0 newly installed, 0 to remove and 99 not upgraded.

Trying to remove then reinstall cannot be done because removing ifupdown2 will attempt to remove proxmox-ve too.

 

[Aman Kumar]

In my Proxmox server, the network is working, but I am still facing issues with dependency failures,
I am encountering dependency failures for the following files: media-vz.mount, local-fs.target, and systemd-fsck@dev-proxvg-proxvg.service. I am unable to connect to my server using SSH. Please help me resolve these issues.