service pve-manager restart effect restart of all VMs

VollStrecker

Renowned Member
Dec 29, 2014
4
1
68
Hello,

in our company we are using Proxmox 4.2 and we found out, that always when we restart the pve-manager service, all VMs on the same node are stopping and restarting.
Independend of any kernel update, this action happens also while installing proxmox package updates.
To prevent restarting all VMs, we have to hold back all Proxmox updates to install important linux security updates on our productive system.
In a commercial, productive environment it is not acceptable to restart all VMs to install software updates.
On a big node (with dualsocket cpus e.g. E5-2690v2 or E5-2640v4) with 40-80 VMs the system needs about 1 hour until all client VMs are booted completly, KSM Sharing is working and the load normalized.

Is there a way to disable the automatic VM restart while installing package updates?

Thank in advance!
 
  • Like
Reactions: chrone
The way I see it, when we manually restarting the pve-manager.service, it will stop all VMs and Containers first and then start all VMs and Containers again. I wish they didn't restart running VMs though. :D

Update:
As pointed out by Tom, updating pve-manager package does not restart VMs/CTs.
 
Last edited:
  • Like
Reactions: VollStrecker
Is there a way to disable the automatic VM restart while installing package updates?

Thank in advance!

Installing updates do NEVER stop or restart VM or containers.

So if you see this, there is another reason for the restart, never applying updates. Do a deeper analysis on your system.
 
  • Like
Reactions: chrone
The purpose of the pve-manager.service is to start/stop all VM and Containers at startup/shutdown. It is not a real service, and it is never restarted on upgrades. So updates do NEVER stop or restart VM or containers.
 
  • Like
Reactions: chrone
Hi Ditmar,

did you verified this?

I have the same problem here. (PVE 4.3 with ZFS RAID1)

"service pve-manager restart" also starts/stops all lxc containers and kvm vms.
Verified by GUI (Uptime off all guests are resetted) and ping to network interfaces.
"uptime" in container is the same as in GUI.

Also the last "apt upgrade" on the host asks for "pve-manager" service restart (ncurses gui).

Another strange behaviour I realized: The restart request also wants to restart services that are inside the containers (e.g. mysql) and not on the host.

/var/log/apt/history
Start-Date: 2016-12-07 15:44:34
Commandline: apt upgrade
Install: pve-kernel-4.4.35-1-pve:amd64 (4.4.35-75, automatic)
Upgrade: libpve-storage-perl:amd64 (4.0-68, 4.0-70), pve-manager:amd64 (4.3-12, 4.3-14), qemu-server:amd64 (4.0-96, 4.0-101), pve-firewall:amd64 (2.0-31, 2.0-33), pve-cluster:amd64 (4.0-47, 4.0-48), pve-qemu-kvm:amd64 (2.7.0-8, 2.7.0-9), pve-container:amd64 (1.0-85, 1.0-87), lxc-pve:amd64 (2.0.6-1, 2.0.6-2), proxmox-ve:amd64 (4.3-72, 4.3-75), pve-docs:amd64 (4.3-17, 4.3-19)
End-Date: 2016-12-07 15:45:10


/var/log/daemon.log
Hi Ditmar,

did you verified this?

I have the same problem here. (PVE 4.3 with ZFS RAID1)

"service pve-manager restart" also starts/stops all lxc containers and kvm vms.
Verified by GUI (Uptime off all guests are resetted) and ping to network interfaces.
"uptime" in container is the same as in GUI.

Also the last "apt upgrade" on the host asks for "pve-manager" service restart (ncurses gui).

Another strange behaviour I realized: The restart request also wants to restart services that are inside the containers (e.g. mysql) and not on the host.

Greetz,
Chris

/var/log/apt/history
Start-Date: 2016-12-07 15:44:34
Commandline: apt upgrade
Install: pve-kernel-4.4.35-1-pve:amd64 (4.4.35-75, automatic)
Upgrade: libpve-storage-perl:amd64 (4.0-68, 4.0-70), pve-manager:amd64 (4.3-12, 4.3-14), qemu-server:amd64 (4.0-96, 4.0-101), pve-firewall:amd64 (2.0-31, 2.0-33), pve-cluster:amd64 (4.0-47, 4.0-48), pve-qemu-kvm:amd64 (2.7.0-8, 2.7.0-9), pve-container:amd64 (1.0-85, 1.0-87), lxc-pve:amd64 (2.0.6-1, 2.0.6-2), proxmox-ve:amd64 (4.3-72, 4.3-75), pve-docs:amd64 (4.3-17, 4.3-19)
End-Date: 2016-12-07 15:45:10


# pveversion --verbose
proxmox-ve: 4.3-75 (running kernel: 4.4.24-1-pve)
pve-manager: 4.3-14 (running version: 4.3-14/3a8c61c7)
pve-kernel-4.4.35-1-pve: 4.4.35-75
pve-kernel-4.4.21-1-pve: 4.4.21-71
pve-kernel-4.4.24-1-pve: 4.4.24-72
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-48
qemu-server: 4.0-101
pve-firmware: 1.1-10
libpve-common-perl: 4.0-83
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-70
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.3-19
pve-qemu-kvm: 2.7.0-9
pve-container: 1.0-87
pve-firewall: 2.0-33
pve-ha-manager: 1.0-38
ksm-control-daemon: 1.2-1
glusterfs-client: 3.9.0-1
lxc-pve: 2.0.6-2
lxcfs: 2.0.5-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.8-pve13~bpo80
 

Attachments

  • daemon.log.txt
    4.7 KB · Views: 1
Hi Dietmar,

thanks for your answer.

So the the purpose of the pve-manager.service is to start/stop all VM and Containers but NOT ONLY at server startup/shutdown.

However the Debian interactive post upgrade routine asks for service restarts (pveproxy, pvedaemon,etc.) and also includes pve-manager by default. Now I know that it should not be restartet but it is activated by default. It would be nice when it is not activated by default.

Best regards,
Chris
 
So the the purpose of the pve-manager.service is to start/stop all VM and Containers but NOT ONLY at server startup/shutdown.

We do that only at startup and shutdown.

However the Debian interactive post upgrade routine asks for service restarts (pveproxy, pvedaemon,etc.) and also includes pve-manager by default.

This is not true. The 'postinst' script does not stop or start service pve-manager.
 
Hi Dietmar,

thanks for your answer.

So the the purpose of the pve-manager.service is to start/stop all VM and Containers but NOT ONLY at server startup/shutdown.

However the Debian interactive post upgrade routine asks for service restarts (pveproxy, pvedaemon,etc.) and also includes pve-manager by default. Now I know that it should not be restartet but it is activated by default. It would be nice when it is not activated by default.

Best regards,
Chris

you probably have installed some tool like "needrestart" or "checkrestart" that hooks into apt and prompts for service restart after upgrades. you should probably blacklist the PVE services there (at least pve-manager!) if you don't want to deselect them on every upgrade
 
  • Like
Reactions: CeeWee
you probably have installed some tool like "needrestart" or "checkrestart" that hooks into apt and prompts for service restart after upgrades. you should probably blacklist the PVE services there (at least pve-manager!) if you don't want to deselect them on every upgrade

When I enabled HA on VM across 3 Proxmox nodes, updating pve on one of the node triggered the node to reboot due to it's been out of corosync timeout tolerance. I decided to not using Proxmox HA anymore for more node's uptime.
 
you probably have installed some tool like "needrestart" or "checkrestart" that hooks into apt and prompts for service restart after upgrades. you should probably blacklist the PVE services there (at least pve-manager!) if you don't want to deselect them on every upgrade

Hi Fabian,

you are right. "needrestart" was installed and caused that behavior. So the restart behaviour was not caused by Promox default settings.
First I blacklisted all pve-* services by adding:

/etc/needrestart/conf.d/pve.conf

Code:
$nrconf{override_rc} = {
  # PVE Services
  qr(^pve) => 0,
};

But when I started "needrestart" manually it asks me to restart services that are inside the containers (screenshot). So finally I removed the package "needrestart".

Many thanks for your hints.

Best regards,
Chris
 

Attachments

  • needrestart_on_host.png
    needrestart_on_host.png
    24.1 KB · Views: 21
Last edited:
i just updated lxc-pve, and needrestart asked to restart these services

Code:
lxc-monitord.service
pve-container@101.service
pve-container@102.service
pve-container@10x.service
[...]
pvedaemon.service

i understand that I should remove needrestart OR ignore when it asks to restart anything pve related

is this right, anything pve related? no pve related updates will ever require restarting anything (aside obviously from a kernel update, for which I am waiting for kernel care support)
 
i just updated lxc-pve, and needrestart asked to restart these services

Code:
lxc-monitord.service
pve-container@101.service
pve-container@102.service
pve-container@10x.service
[...]
pvedaemon.service

i understand that I should remove needrestart OR ignore when it asks to restart anything pve related

is this right, anything pve related? no pve related updates will ever require restarting anything (aside obviously from a kernel update, for which I am waiting for kernel care support)

the packages will restart their own services, and anything providing an API endpoint will make sure that the API services are restarted. but an upgrade of a library used by a service still requires a manual restart (or an automatic one triggered by needrestart). that being said, needrestart will definitely have some false positives that you need to put into a "never restart" list - e.g., you probably don't want to restart any guests automatically ;)
 
the packages will restart their own services, and anything providing an API endpoint will make sure that the API services are restarted. but an upgrade of a library used by a service still requires a manual restart (or an automatic one triggered by needrestart). that being said, needrestart will definitely have some false positives that you need to put into a "never restart" list - e.g., you probably don't want to restart any guests automatically ;)

Thanks Fabian. I put in an issue to see if it is possible to exclude containers and VMs from being suggested. What systemd services should I also exclude (since they auto restart own their own)?
 
like I said - all of our own services reload or restart when their packages get upgraded, and the API ones additionally restart when any of the API endpoint modules are upgraded.
 
  • Like
Reactions: Proxygen

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!