Boot hangs after Upgrade to Proxmox 8

[ubu]

After the Upgrade booting hangs permanently at starting networking.service.

I have disabled networking, system bootsup, login via RMC works, starting networking by hand with systemctl start networking works.

If i reenable networking system hangs again

 

[ubu]

Boot with networking disabled, started later by hand

 

[niteshadow]

Reboot (with network enabled) and select the previous kernel.
If it works maybe you have an interface rename issue.

 

[ubu]

Already tried that, does not help. Since the network can be started after the boot, Interfaces and config seem to be ok

 

[fweber]

Hi, could it be the case that ntpsec is installed and you are running into the issue described in [1] [2]?

[1] https://pve.proxmox.com/wiki/Upgrade_from_7_to_8#Network_Fails_on_Boot_Due_to_NTPsec_Hook
[2] https://forum.proxmox.com/threads/proxmox-ve-8-0-released.129320/page-6#post-567264

 

[ubu]

DANKE !!! Hat funktioniert (und mich 2 Tage gekostet

 

[ubu]

Bitte Hinweis in pve7to8 einbauen

 

[fweber]

Good to hear that you could fix the issue!

So far the root cause of the hang was not clear. I took another look now, and might have a theory how/why the hang happens. It would be great if you could run some checks (see below) to find out whether this theory matches your experience.

In general, it seems like networking hangs at boot if both ntpsec and ntpsec-ntpdate are installed -- the reason seems to be a deadlock during network setup which only triggers if both these packages are present.

However, this seems to be the case for PVE 7.4 already (can be reproduced by installing both packages). But according to you and others [3], the hangs did not happen before the upgrade to PVE 8. One explanation would be:

• before the upgrade to PVE 8, at least one of ntpsec and ntpsec-ntpdate was not installed, so the node boots fine
• after the upgrade to PVE 8, both ntpsec and ntpsec-ntpdate are installed, so the node now hangs at boot

If this is the case, the question is why the upgrade installs those packages. I suspect this is due to changes in ntp-related packages between Debian Bullseye and Bookworm. For example, if I install ntp and ntpsec-ntpdate on PVE 7.4 and upgrade to 8, the upgrade also installs ntpsec (because ntp depends on ntpsec since Bookworm [4]), and the node hangs after reboot.

To find out what happened in your case, could you maybe post the following?

• the output of dpkg -l | grep ntp
• the entry of /var/log/apt/history.log corresponding to the upgrade to PVE 8. The entry should look as follows:
  

  Start-Date: 2023-07-...
  Commandline: apt dist-upgrade
  Install: <long list of packages>
  Upgrade: <long list of packages>
  Remove: <list of packages>
  End-Date: 2023-07-...

  This would allow us to find out if/why ntpsec or ntpsec-ntpdate got installed during upgrade.

Bitte Hinweis in pve7to8 einbauen

Note that there is a note in the upgrade guide already [5]. But yeah, if we can find a reliable way to detect this situation before the upgrade, this check could potentially be added to pve7to8.

[1] https://pve.proxmox.com/wiki/Upgrade_from_7_to_8#Network_Fails_on_Boot_Due_to_NTPsec_Hook
[2] https://salsa.debian.org/debian/ntpsec/-/blob/debian/unstable/debian/ntpsec-ntpdate.if-up#L40
[3] https://forum.proxmox.com/threads/proxmox-ve-8-0-released.129320/page-6#post-567264
[4] https://packages.debian.org/bookworm/ntp
[5] https://pve.proxmox.com/wiki/Upgrade_from_7_to_8#Network_Fails_on_Boot_Due_to_NTPsec_Hook