On package updates, how do you know when a reboot is required?

A

akulbe

Member
Jan 1, 2023
55
4
13
Portland, OR
Just like the title says. How can you tell?

I know that when a kernel gets updated, of course, but are there other situations where you should reboot, as well?

The context is this: I have a nested Proxmox cluster that I run the test version on, and there were some "pve*" packages that just got updated.
 
Last edited:
Run needrestart in a root shell on the proxmox node. If you don't want to do this manually: Setup a nagios compatible monitoring software like Icinga2, needrestart can be called with the parameter -p to act as nagios check plugin
 
  • Like
Reactions: Johannes S
UdoB said:
Actually I do restart most of the services - slowly, one by one, with checking the results. My personal understanding is unchanged: after installing updates, the packages are uptodate but the running processes use the old implementation - until they are restarted.
Click to expand...
If the package is done correctly, the services they provide are already restarted after the install is done. I assume you're talking about the VMs, which need to be restarted (or live migrated?) in order to load the new qemu implementation.
 
  • Like
Reactions: Johannes S
LnxBil said:
If the package is done correctly, the services they provide are already restarted after the install is done. I assume you're talking about the VMs,
Click to expand...
After some updates on a node I saw a lot of pve-* services listed by needrestart. In my observation they do not restart themselfes, at least not all of them. The linked thread suggests (surprisingly) that's fine.
 
UdoB said:
After some updates on a node I saw a lot of pve-* services listed by needrestart. In my observation they do not restart themselfes, at least not all of them. The linked thread suggests (surprisingly) that's fine.
Click to expand...
Maybe not all of them, yes. Last time I checked, I was surprised how many actually restartet, yet I haven't check every single service.
 
LnxBil said:
If the package is done correctly, the services they provide are already restarted after the install is done.
Click to expand...
Okay, let's verify my observation. I have a Test-Cluster using the pve-no-subscription repos. The software is uptodate.

1) Force a DOWNGRADE of a base library to be able to upgrade in step 3 - do NOT do this just-for-fun:

Code: 
~# apt install libc6=2.36-9+deb12u7

2) Reboot

This gets me a clean environment, except for that libc. "needrestart" does not list anything.

3) Upgrade
Code: 
~# apt full-upgrade 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  libc6
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,754 kB of archives.

Code: 
root@pm0:~# needrestart  -rl -b  | grep -c service ; needrestart -rl -b | grep pve.*service 
49
NEEDRESTART-SVC: pve-cluster.service
NEEDRESTART-SVC: pve-firewall.service
NEEDRESTART-SVC: pve-ha-crm.service
NEEDRESTART-SVC: pve-ha-lrm.service
NEEDRESTART-SVC: pve-lxc-syscalld.service
NEEDRESTART-SVC: pvedaemon.service
NEEDRESTART-SVC: pvefw-logger.service
NEEDRESTART-SVC: pveproxy.service
NEEDRESTART-SVC: pvescheduler.service
NEEDRESTART-SVC: pvestatd.service

How long should I wait until any ofthese 10 services gets restarted without manual intervention? Usually I do not wait... :-)
 
UdoB said:
How long should I wait until any ofthese 10 services gets restarted without manual intervention? Usually I do not wait...
Click to expand...
Just for the sake of completeness: 20 h later (and after cron.daily did its job) I see that two of those 10 services got restarted:
Code: 
root@pm0:~# needrestart  -rl -b  | grep -c service ; needrestart -rl -b | grep pve.*service
46
NEEDRESTART-SVC: pve-cluster.service
NEEDRESTART-SVC: pve-firewall.service
NEEDRESTART-SVC: pve-ha-crm.service
NEEDRESTART-SVC: pve-ha-lrm.service
NEEDRESTART-SVC: pve-lxc-syscalld.service
NEEDRESTART-SVC: pvedaemon.service
NEEDRESTART-SVC: pvescheduler.service
NEEDRESTART-SVC: pvestatd.service
"pvefw-logger" and "pveproxy" is gone.
 
you are mixing up two things:
- if the PVE packages are upgraded, services are restarted/reloaded/left-alone as needed by the package maintainer scripts that are executed before/during/after upgrades, including things like a pve-container upgrade triggering a reload of pveproxy/pvedaemon even though the latter is shipped by pve-manager
- if random library X is upgraded which some PVE service(s) might use, they are not restarted/reloaded automatically

for the latter part, you need to setup needrestart (or reboot the whole host), but you need to mimic what the packages themselves do on upgrades else things will break:
- for some services, the right choice is restarting the service (because they are safe to stop and start)
- for others, the right choice is reloading the service (else, running tasks/.. would get interrupted!)
- for another set, the right choice is to leave them as they are, as there is no reload support and restarting them would cause guests or other important parts to be killed (e.g., networking.service, the services representing individual guest instances, dbus things, ..)

note that some services will periodically reload themselves anyway (e.g., when rotating logfiles, or after a certain time, or after a certain memory usage threshold)
 
  • Like
Reactions: Johannes S and UdoB
fabian said:
you are mixing up two things:
Click to expand...
Thanks for clarifying/rephrasing this once more!
Obviously my personal understanding of your answers in the other thread were wrong - and that's good! :cool:
 
LnxBil said:
Check if the file /run/reboot-required is there. It will get created by e.g. kernel updates, which require a reboot to activate. Everything else can be restarted while the OS is running.
Click to expand...
In my PVE 3 node cluster ( no subscription) even after kernel update( dist upgrade message says reboot is required) i couldn't see this file. Would there be any reason why this file wasn't created?
 
  • Like
Reactions: AlexHK
Try watching /var/run/reboot-required , not "/run" ;-)
 
I don't think we set that flag file currently, but some stock Debian packages do. but anyhow - every kernel upgrade requires a reboot ;)
 
  • Like
Reactions: Johannes S and thulle
If you look at /usr/bin/pveupgrade (PVE's upgrade wrapper script) you will find that it checks the list of pending packages from before running apt-get dist-upgrade and outputs the "consider rebooting message" on kernel package that is no longer pending afterwards.

Since PVE does not write the reboot-required flag, I simply compare the output of the following two commands (via Ansible in my case) to know if a reboot is required:

Code: 
# running kernel
uname -r

# latest installed kernel
ls -1 /boot/vmlinuz-* | sed 's|.*/vmlinuz-||' | sort -V | tail -n1

If they are not the same, I start my reboot procedure.
 
AlexHK said:
Since PVE does not write the reboot-required flag
Click to expand...
Does it?

non-subscription in this very moment, after full-upgrade:
Code: 
~# ls -Al /var/run/reboot-required*
-rw-r--r-- 1 root root  0 Sep  9 16:32 /var/run/reboot-required
-rw-r--r-- 1 root root 36 Sep  9 16:32 /var/run/reboot-required.pkgs

~# cat /var/run/reboot-required.pkgs
proxmox-kernel-7.0.14-16-pve-signed

Which mechanism had created these if not plain apt/dpkg?

----
Sidenote: if you expected to find them below /run talk to those people who thought it would be a good idea to trash FHS (partially) after decades of stable use... --> https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
 
Last edited:
  • Like
Reactions: Johannes S
UdoB said:
Which mechanism had created these if not plain apt/dpkg?
Click to expand...
something custom you have installed on your system? I'd check the /etc/kernel/*.d and /lib/kernel/*.d directories for hooks that might be responsible..
 
  • Like
Reactions: Johannes S
First I must admit that I really didn't know which script is responsible. It just didn't matter to me - as it just did work. Always.

fabian said:
something custom you have installed on your system? I'd check the /etc/kernel/*.d and /lib/kernel/*.d directories for hooks that might be responsible..
Click to expand...
And now I had to check :-)

This one is it (probably):
Code: 
~# grep -R  reboot-required /etc/kernel/   /lib/kernel/
/etc/kernel/postinst.d/unattended-upgrades:    touch /var/run/reboot-required

Additionally I've found the (here irrelevant):
Code: 
~# grep -R reboot-required /var/lib/dpkg
/var/lib/dpkg/info/dbus.postinst:        touch /var/run/reboot-required || true

So unattended-upgrades seems to be responsible in my case. Probably it does not belong to a default PVE installation, but my personal "post-install" does install some additional packages...
 
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom
Back