Proxmox crashes/reboot after upgrading packages

[Kurtulus]

Last week i did package upgrades of my nodes

apt upgrade

and one of the nodes wasn't responsive after i hit enter OK to restart a bunch of services, this node started rebooting.

Today the same happened, this time to another node. I have never experienced this before.

I am using Proxmox 8.0.4 with enterprise subscription.

Does this mean that Proxmox can't handle too many service restarts anymore and therefore crashes?

pveversion
pve-manager/8.0.4/d258a813cfa6b390 (running kernel: 6.2.16-12-pve)

 

[fiona]

Hi,

Last week i did package upgrades of my nodes

apt upgrade

you should always use apt dist-upgrade which will also remove packages if necessary to upgrade the system as a whole. With apt upgrade you can run into problematic situations.

and one of the nodes wasn't responsive after i hit enter OK to restart a bunch of services, this node started rebooting.

Today the same happened, this time to another node. I have never experienced this before.

I am using Proxmox 8.0.4 with enterprise subscription.

Does this mean that Proxmox can't handle too many service restarts anymore and therefore crashes?

pveversion
pve-manager/8.0.4/d258a813cfa6b390 (running kernel: 6.2.16-12-pve)

Please check the system logs/journal from around the time the issue happened.

 

[Kurtulus]

Hi Fiona, sorry for my late reply, but today i have with the upgrade to Proxmox 8.1.3 again this problem.
First of all, apt dist-upgrade is not the issue. I do the upgrade via the Proxmox management GUI.

Another issue with the node 1 (pve01) is this, I can upgrade 3 other nodes and reboot them while node 1 is still ongoing upgrading... I don't understand this as well. It has plenty of resources and is per core even faster then the other nodes. If you monitor via top/htop you don't see any resource intensive services in the background running. This happens now for a while. The upgrade mechanism on this node is extremely slow.

I have attached the syslogs and the services Proxmox is restarting after an upgrade and doing so crashes.

 

[fabian]

you use some sort of extra restarting after upgrade script, and that does unexpected things. there are some services which are part of PVE which must not be restarted (but either left running, or reloaded gracefully). our packaging takes care of that, and you should probably exclude all PVE specific services from your tooling.

 

[Kurtulus]

I use a package called needrestart, which checks if there are services to be restarted after an update or upgrade.

||/ Name           Version      Architecture Description
+++-==============-============-============-===============================================================
ii  needrestart    3.6-4        all          check which daemons need to be restarted after library upgrades

 

[fabian]

yes, I suspected that but there is more than one solution for that problem. please make needrestart ignore the PVE services, they all handle graceful reloads on upgrades themselves!

 

[UdoB]

please make needrestart ignore the PVE services, they all handle graceful reloads on upgrades themselves!

That is interesting!

Needrestart reported a long list of "pve-" on several occasions on my systems. Usually I do follow that recommendation. And restarting "pve-*" one by one seems to work well, without hickup. (There was one problem last year when needrestart made me restart watchdog-mux which led to a surprising reboot.)

When needrestart -rl lists pve-* and I do restart those services the next "needrestart -rl" check does not list them anymore. Why is this when pve-* services manage their restart by themselve?

Just trying to understand waht's happening...

 

[fabian]

it depends on the service. but for example, for pveproxy and pvedaemon (and the corresponding PMG/PBS services as well), the graceful reload might still leave the old process running for a while to finish up running tasks (like shell sessions, backups, migrations, ..). a restart would kill all those tasks - so maybe you were just lucky that no tasks were running, or you didn't notice that you killed them . some other services are just never supposed to be stopped after they've been started, or require special handling to restart (lxcfs, watchdog-mux, .. fall into this category).

 

[UdoB]

but for example, for pveproxy and pvedaemon (and the corresponding PMG/PBS services as well), the graceful reload might still leave the old process running for a while to finish up running tasks (like shell sessions, backups, migrations, ..). a restart would kill all those tasks - so maybe you were just lucky that no tasks were running, or you didn't notice that you killed them

Okay.

Just to be explicit - and basically repeating your statement from above: after "full-upgrade"/"dist-upgrade" it is NOT recommended to follow the output of "needrestart".

Instead of that one should leave pve-* untouched.

While I basically just need to internalize this I am a little bit uneasy with it. In my opinion this is a quirk in your middleware - or in needrestart: both tools should really be "compatible".

The same is true for the well known advice to never run "apt upgrade". This surprises new users - which is not a surprise ;-)


Whatsoever: I am really happy to have Proxmox VE!

 

[fabian]

While I basically just need to internalize this I am a little bit uneasy with it. In my opinion this is a quirk in your middleware - or in needrestart: both tools should really be "compatible".

not really - needrestart already comes with a list of "special" units/.. preloaded, it just doesn't know about PVE/PBS/PMG services this is also the case for any other custom service you run that has special requirements, out of the box needrestart only works (properly/safely) for very "vanilla" (stock Debian) systems.

The same is true for the well known advice to never run "apt upgrade". This surprises new users - which is not a surprise ;-)

yes. there is a big difference between how stock Debian works, and how we do things. Debian basically only has "breaking" package relationship changes on major releases (~every 2 years), since anything that goes into stable after release (via -security or point releases) is small fixes that only change a single source package's contents, but nothing about how it interacts with the rest of the distro. for PVE/PBS/PMG, this is different - when we add new features, we might need to move code from one package to another, introduce new ones, break some internal compatibility guarantee, .. and some of those changes will require encoding in packaging relationships (by depending on a certain version, or declaring a certain older version incompatible). APT by default (for example, when used via "apt upgrade") is very conservative about handling such changes, and does not upgrade all packages in all circumstances. hence the need for "apt dist-upgrade" for regular upgrades on Proxmox systems. in most cases, the worst thing that happens with "apt upgrade" is being left behind on older versions of some packages (which means less test coverage of course, since you'll end up on a version combo that barely anyone else is running), but unless we missed encoding some requirement, nothing should break.

 

[UdoB]

Thanks for the explanation, Fabian.

All in all I am a happy user (Despite being bitten by 6.5.11 problems yesterday...)

Best regards
Udo