VMs hang on shutdown during backup job

[sar]

@fiona

VM134

 

[fiona]

Again, replacing XYZ with the VM ID of a stuck VM, can you check what the following show?

systemctl status XYZ.scope
cat /proc/$(cat /var/run/qemu-server/XYZ.pid)/status | grep PPid

and then using the result of the last command

cat /proc/PPID/cmdline

Are there any systemd settings you modified?

 

[sar]

Again, replacing XYZ with the VM ID of a stuck VM, can you check what the following show?

systemctl status XYZ.scope
cat /proc/$(cat /var/run/qemu-server/XYZ.pid)/status | grep PPid

and then using the result of the last command

cat /proc/PPID/cmdline

Are there any systemd settings you modified?

i see this:

root@pve:~# systemctl status 134.scope
Unit 134.scope could not be found.
root@pve:~# cat /proc/$(cat /var/run/qemu-server/134.pid)/status | grep PPid
PPid: 1
root@pve:~# cat /proc/1/cmdline
/sbin/initroot@pve:~#


No any systemd settings dont modified

 

[sar]

i see this:

root@pve:~# systemctl status 134.scope
Unit 134.scope could not be found.
root@pve:~# cat /proc/$(cat /var/run/qemu-server/134.pid)/status | grep PPid
PPid: 1
root@pve:~# cat /proc/1/cmdline
/sbin/initroot@pve:~#


No any systemd settings dont modified

@fiona

Do you have any ideas?

 

[fiona]

I do have a patch in the works that would go back to getting the VM ID from the process's commandline rather than the cgroup file, which would be a workaround.

But the real question is why does the cgroup file look like it does in your case. We always run the QEMU command in the qemu.slice/ID.scope systemd scope: https://git.proxmox.com/?p=qemu-ser...88629e1fa1b96f500ab902ccbaffb77;hb=HEAD#l5861
So there has to be a bug either in our code for setting this up or in systemd itself. But I wasn't able to reproduce the issue yet and am still investigating.

 

[sar]

I do have a patch in the works that would go back to getting the VM ID from the process's commandline rather than the cgroup file, which would be a workaround.

But the real question is why does the cgroup file look like it does in your case. We always run the QEMU command in the qemu.slice/ID.scope systemd scope: https://git.proxmox.com/?p=qemu-ser...88629e1fa1b96f500ab902ccbaffb77;hb=HEAD#l5861
So there has to be a bug either in our code for setting this up or in systemd itself. But I wasn't able to reproduce the issue yet and am still investigating.

@fiona

Can I install proxmox ve7.4?
Maybe it will helps with this bugs?

 

[fiona]

You can't downgrade Proxmox VE installation (or Debian) across major versions. As a workaround, you can use snapshot mode backup. You should install and enable the guest agent for VMs that don't have it yet, so filesystem consistency is not an issue.

 

[sar]

You can't downgrade Proxmox VE installation (or Debian) across major versions. As a workaround, you can use snapshot mode backup. You should install and enable the guest agent for VMs that don't have it yet, so filesystem consistency is not an issue.

@fiona
But I can make backup my VMs on backup server and after reinstall proxmox version 7.4 and restore my VMs

 

[fiona]

@fiona
But I can make backup my VMs on backup server and after reinstall proxmox version 7.4 and restore my VMs

Yes, if you think that is worth the effort. Is there a special reason you need to use stop mode backup?
Patch to improve qmeventd: https://lists.proxmox.com/pipermail/pve-devel/2024-June/064210.html

 

[sar]

Yes, if you think that is worth the effort. Is there a special reason you need to use stop mode backup?
Patch to improve qmeventd: https://lists.proxmox.com/pipermail/pve-devel/2024-June/064210.html

Can you explain me how to use this patch?

maybe I should use special commands, or edit some configs ?

 

[sar]

Can you explain me how to use this patch?

mayme I should use special commands, or edit some configs ?

@fiona

I use Stop mode backup cuz this is full backup and it’s safely for me

 

[fiona]

Can you explain me how to use this patch?

maybe I should use special commands, or edit some configs ?

It's intended to be reviewed by other developers and if they deem it acceptable, it will be applied and rolled out in a future version. You could apply and build it yourself, but do so at your own risk.

@fiona

I use Stop mode backup cuz this is full backup and it’s safely for me

snapshot mode backup is also a full backup and as long as the guest agent is installed and enabled, the filesystem status will be consistent too. Of course, there can be special applications that do require even more than that, e.g. databases which can be handled with a hook script.