Node reboots by itself with current updates?!

[prahn]

Hi!

This morning I received the hint, that there are updates available:

libc-bin	2.36-9+deb12u4	2.36-9+deb12u6
libc-l10n	2.36-9+deb12u4	2.36-9+deb12u6
libc6	2.36-9+deb12u4	2.36-9+deb12u6
locales	2.36-9+deb12u4	2.36-9+deb12u6

Like always I simply updated the first node this morning during production time.
After installing them, needrestart asked me to update the depending services, which were a lot...

Suddenly the host was marked offline in the cluster... and after some moment I realized, that the node was restarting.
I did the same with another node... and also this one was restarting?!

Why does that happen?
Should I better not restart services after doing updated??

Regards,
Ingo

 

[fiona]

Hi,
please see: https://forum.proxmox.com/threads/proxmox-crashes-reboot-after-upgrading-packages.134670/post-608885

 

[prahn]

Hi Fiona!

Thank you for quick reply. I now installed updates on the third node, but I did not restart the pve-services.
I deselected them in the needrestart-interface. However, this host also went down! :-(
Here is the output:

Restarting services...
 systemctl restart ceph-crash.service ceph-mgr@pve2.service ceph-mon@pve2.service ceph-osd@3.service ceph-osd@4.service ceph-osd@5.service check-mk-agent-async.service chrony.service cron.service ksmtuned.service lxc-monitord.service postfix@-.service rpcbind.service rrdcached.service smartmontools.service spiceproxy.service ssh.service systemd-journald.service systemd-udevd.service watchdog-mux.service zfs-zed.service
Service restarts being deferred:
 systemctl restart corosync.service
 /etc/needrestart/restart.d/dbus.service
 systemctl restart getty@tty1.service
 systemctl restart lxcfs.service
 systemctl restart pve-cluster.service
 systemctl restart pve-container@108.service
 systemctl restart pve-firewall.service
 systemctl restart pve-ha-crm.service
 systemctl restart pve-ha-lrm.service
 systemctl restart pve-lxc-syscalld.service
 systemctl restart pvedaemon.service
 systemctl restart pvefw-logger.service
 systemctl restart pveproxy.service
 systemctl restart pvescheduler.service
 systemctl restart pvestatd.service
 systemctl restart qmeventd.service
 systemctl restart systemd-logind.service

Why that?

 

[fiona]

I have never used needrestart, but maybe the new config was not yet correctly loaded by the tool?

 

[prahn]

Should I simply uninstall it and everything is safe?
I thought it would be safer to have it installed...

 

[Kingneutron]

> Like always I simply updated the first node this morning during production time

Why are you doing updates in the morning without a scheduled downtime window (and/or a full backup before changing the system)?? Do you not have a Test environment to vet these changes before it affects your Prod cluster?

 

[fiona]

Should I simply uninstall it and everything is safe?
I thought it would be safer to have it installed...

It's your choice. But if deselecting the Proxmox VE services doesn't work for some reason, it's a problem of course.

 

[fabian]

all our services already properly hot-reload or restart or keep running on upgrades, depending on what is the right choice for that particular service.

if you don't exclude them from needrestart, then it will unconditionally restart them, which is the wrong choice for many of our services, with potentially fatal consquences for your guests or the whole node. so yes, either uninstall/disable needrestart, or configure it to ignore all PVE related services..