Unable to shut down or restart a VM

[siggib]

I keep getting timeout when trying to shut down or restart a VM, both soft reboot, hard reset and stop. Here are the errors

TASK ERROR: VM quit/powerdown failed - got timeout
and

trying to acquire lock... TASK ERROR: can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout

I already tried qm unlock 100 and I've tried to delete /run/lock/qemu-server/lock-100.conf. While the commands all appeared to work, it did not make any difference.

I also tried restarting pve-cluster (single node) with systemctl. Any ideas what's going on?

 

[Moayad]

Hi,

In the system report you've provided I don't see the VM 100 config. Could you please post the VM config using `qm config 100` command?

And also the syslog when you try to restart or shutdown the VM.

 

[siggib]

Here are the configs of the three VM's I have built so far. All are giving the same challanges

siggib@pve:~$ sudo qm config 100
boot: order=ide0;net0
cores: 1
cpu: x86-64-v2-AES
ide0: local-lvm:vm-100-disk-0,size=32G
machine: pc-i440fx-8.1
memory: 4096
meta: creation-qemu=8.1.2,ctime=1706734202
name: Windows7
net0: e1000=BC:24:11:38:A1:2E,bridge=vmbr0,firewall=1
numa: 0
ostype: win7
scsihw: virtio-scsi-single
smbios1: uuid=5f6ed057-b642-470b-9afe-5815ff68f30e
sockets: 1
vmgenid: 49838a78-6e3c-4bb7-adcf-cda81831a3cb
siggib@pve:~$ sudo qm config 101
agent: 1
boot: order=ide0;ide2;net0
cores: 1
cpu: x86-64-v2-AES
ide0: local-lvm:vm-101-disk-0,size=32G
ide2: iso:iso/Windows10.iso,media=cdrom,size=3583616K
machine: pc-i440fx-8.1
memory: 4096
meta: creation-qemu=8.1.2,ctime=1706738352
name: Windows10
net0: e1000=BC:24:11:0E:03:45,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-single
smbios1: uuid=f8325498-c0d4-4b50-a180-ae1f42a5f4f0
sockets: 1
tpmstate0: local-lvm:vm-101-disk-1,size=4M,version=v2.0
vmgenid: 001abc38-0dc9-4101-9f19-3d977cbab74a
siggib@pve:~$ sudo qm config 102
agent: 1
bios: ovmf
boot: order=ide0;ide2;net0
cores: 1
cpu: x86-64-v2-AES
efidisk0: local-lvm:vm-102-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
ide0: local-lvm:vm-102-disk-1,size=32G
ide2: iso:iso/Windows11.iso,media=cdrom,size=5529856K
machine: pc-q35-8.1
memory: 4096
meta: creation-qemu=8.1.2,ctime=1706739008
name: Win11
net0: e1000=BC:24:11:C5:C7:00,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsihw: virtio-scsi-single
smbios1: uuid=1de4def3-83e2-4e5c-93d4-886eb859a559
sockets: 1
tpmstate0: local-lvm:vm-102-disk-2,size=4M,version=v2.0
vmgenid: 8ae514c3-cdb5-4745-8f1a-71acb1ad1c66
siggib@pve:~$

Here are the syslogs entries I was able to find

siggib@pve:~$ sudo tail /var/log/pve/tasks/active
UPID:pve:0014A84D:1BCC1C37:65BAC2AF:qmcreate:101:siggib@pam: 1 65BAC2B0 OK
UPID:pve:0014575E:1BC89729:65BAB9AD:qmshutdown:100:siggib@pam: 1 65BAB9E9 VM quit/powerdown failed - got timeout
UPID:pve:00145159:1BC855A7:65BAB905:qmshutdown:100:siggib@pam: 1 65BAB941 VM quit/powerdown failed - got timeout
UPID:pve:00144C8F:1BC81BB0:65BAB871:qmshutdown:100:siggib@pam: 1 65BAB8AD VM quit/powerdown failed - got timeout
UPID:pve:001428E3:1BC6F8BC:65BAB588:qmstop:100:siggib@pam: 1 65BAB592 can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout
UPID:pve:001428B2:1BC6F322:65BAB57A:qmshutdown:100:siggib@pam: 1 65BAB5B6 VM quit/powerdown failed - got timeout
UPID:pve:00142640:1BC6D93C:65BAB537:qmstop:100:siggib@pam: 1 65BAB541 can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout
UPID:pve:00142606:1BC6D0BC:65BAB521:qmshutdown:100:siggib@pam: 1 65BAB55D VM quit/powerdown failed - got timeout
UPID:pve:00142267:1BC6B182:65BAB4D2:qmreset:100:siggib@pam: 1 65BAB4D2 OK
UPID:pve:00142237:1BC6AC42:65BAB4C4:vncproxy:100:siggib@pam: 1 65BAB52B OK
siggib@pve:~$ sudo tail /var/log/pve/tasks/index
UPID:pve:0014CB5D:1BCD33C7:65BAC57B:vncproxy:102:siggib@pam: 65BAC5A5 OK
UPID:pve:0014CDDD:1BCD4085:65BAC59C:qmreset:102:siggib@pam: 65BAC5A6 can't lock file '/var/lock/qemu-server/lock-102.conf' - got timeout
UPID:pve:0014CDF5:1BCD4412:65BAC5A5:vncproxy:101:siggib@pam: 65BAC724 OK
UPID:pve:0014ED2E:1BCE21BC:65BAC7DC:qmshutdown:102:siggib@pam: 65BAC7E6 can't lock file '/var/lock/qemu-server/lock-102.conf' - got timeout
UPID:pve:0014CDB9:1BCD3B45:65BAC58E:qmreboot:102:siggib@pam: 65BAC7E6 VM quit/powerdown failed
UPID:pve:0014E60A:1BCDDC5F:65BAC72B:vncproxy:102:siggib@pam: 65BAC7F7 OK
UPID:pve:0014EF5F:1BCE31FC:65BAC806:qmshutdown:100:siggib@pam: 65BAC842 VM quit/powerdown failed - got timeout
UPID:pve:0014EF47:1BCE2DBA:65BAC7FB:qmshutdown:102:siggib@pam: 65BACA53 VM quit/powerdown failed

 

[sb-jw]

You should install and activate the Guest Agent in the VM.

 

[siggib]

Is that required for stop and reset to work? How does that help if the VM doesn't boot?

 

[sb-jw]

Your Windows probably doesn't accept commands from the host, the agent allows just that. The VM then accepts the command from the hypervisor and tells the VM to shut down now.

This doesn't help at first, but it might save you the boot problems because it could also be a subsequent error.

 

[siggib]

Your Windows probably doesn't accept commands from the host, the agent allows just that. The VM then accepts the command from the hypervisor and tells the VM to shut down now.

This doesn't help at first, but it might save you the boot problems because it could also be a subsequent error.

I though stop and reset where the same as the hard reset button on the PC or pulling the power plug? If not what exactly is the difference between shutdown and stop?
BTW the "what if the VM doesn't boot" was a hypothetical question. In that hypothetical, exactly how do I install the agent if I never get the VM to boot? If the VM froze on boot, how do I reboot it? Or worse it froze during initial install?
For clarification couple of the VMs have no problem booting up and shutting down as long as I do it from within Windows. However, if I try a hard reset or a hard stop, I just get that timeout error.
Also, for clarification, I'm not arguing against installing the agent, I just don't see how that relates to my issue of not being able to hard shutdown (i.e. hard power-off) a VM.
While I am new to PVE, I'm far from new to Virtualization and what you are suggestion makes zero sense. I've never heard of any virtualization platform needing an agent for hard power off or hard reset.

 

[sb-jw]

qm stop <vmid> [OPTIONS]

Stop virtual machine. The qemu process will exit immediately. Thisis akin to pulling the power plug of a running computer and may damage the VM data

qm shutdown <vmid> [OPTIONS]

Shutdown virtual machine. This is similar to pressing the power button on a physical machine.This will send an ACPI event for the guest OS, which should then proceed to a clean shutdown.

Source: https://pve.proxmox.com/pve-docs/qm.1.html

 

[siggib]

Yeah that is exactly what I am saying. Yesterday these were just getting timeout. Today everything seems to be working as expected.

 

[esi_y]

Your Windows probably doesn't accept commands from the host, the agent allows just that. The VM then accepts the command from the hypervisor and tells the VM to shut down now.

I don't want to hijack the thread, but it is not Windows-specific.

This doesn't help at first, but it might save you the boot problems because it could also be a subsequent error.

I also do not understand this remark - about "boot problems"?



BTW Soft reboot is the same as graceful shutdown and new start, so it's really not reacting to the shutdown command. But if it's not reacting to the stop, that's qm problem really.

 

[sb-jw]

I don't want to hijack the thread, but it is not Windows-specific.

That's right, but since the TO has Windows VMs, I made my statement more specific.

I also do not understand this remark - about "boot problems"?

How does that help if the VM doesn't boot?

 

[esi_y]

While I am new to PVE, I'm far from new to Virtualization and what you are suggestion makes zero sense. I've never heard of any virtualization platform needing an agent for hard power off or hard reset.

I think he just misread your question / been in a haste. I have seen @sb-jw dispense some good advice here before. I also cannot put my finger on this one, but I can confirm it is something I had occasionally experienced before myself with VMs on PVE.

Tried to check what's happening there [1] and unless the log is a red herring, it's happening right at the beginning and then through [2] and [3].

If you wonder what the heck am I trying to accomplish, not sure - either it helps @Moayad or I get back to it myself some time later. I am sure he will reply again tomorrow.

BTW Something crossed my mind, is this in a cluster or standalone node?

EDIT: I see you noted "single node" above, but unsure if you meant you restarted it on single node only or you have a single node only (no cluster)?

[1] https://github.com/proxmox/qemu-ser...039b3b75a9f7a2f6d4163/PVE/QemuServer.pm#L6270
[2] https://github.com/proxmox/pve-gues...172296c38058d0/src/PVE/AbstractConfig.pm#L291
[3] https://github.com/proxmox/qemu-ser...d4039b3b75a9f7a2f6d4163/PVE/QemuConfig.pm#L55

 

[sb-jw]

I think he just misread your question / been in a haste.

Funny, I didn't miss anything, the post was subsequently edited and significantly expanded. I looked at the emails and that was the only thing that was initially in there:

I though stop and reset where the same as the hard reset button on the PC or pulling the power plug? If not what exactly is the difference between shutdown and stop?

@tempacc346235 Thanks for making the subsequent change visible, otherwise I wouldn't have noticed it.

 

[esi_y]

Funny, I didn't miss anything, the post was subsequently edited and significantly expanded. I looked at the emails and that was the only thing that was initially in there:

@tempacc346235 Thanks for making the subsequent change visible, otherwise I wouldn't have noticed it.

I just wish the forum was doing it by itself. I also often edit my posts, but even if I highlight it, it's not sending out any new notifications. I was only reading the later / expanded version then. Either way, it's now set as "solved", but I wonder what was the resolution. Because I really had the same happen to me before on non-Win VMs and it was not any of the stuck lock from previous migration, etc. cases.

 

[siggib]

This problem has resurfaced:

siggib@pve:~$ sudo qm reset 104
trying to acquire lock...
can't lock file '/var/lock/qemu-server/lock-104.conf' - got timeout
siggib@pve:~$ sudo qm config 104
agent: 1
boot: order=scsi0;ide2;net0
cores: 1
cpu: x86-64-v2-AES
ide2: iso:iso/ubuntu-23.10.1-desktop-amd64.iso,media=cdrom,size=5052730K
memory: 2048
meta: creation-qemu=8.1.2,ctime=1706902457
name: ubuntu23
net0: virtio=BC:24:11:31:78:05,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-104-disk-0,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=a624f94a-12a2-475b-8f71-b14e942ae056
sockets: 1
tpmstate0: local-lvm:vm-104-disk-1,size=4M,version=v2.0
vmgenid: f6db3439-cacb-473b-bbea-3662e9d10290
siggib@pve:~$ 
siggib@pve:~$ sudo qm stop 104
trying to acquire lock...
can't lock file '/var/lock/qemu-server/lock-104.conf' - got timeout
siggib@pve:~$ sudo qm unlock 104
trying to acquire lock...
 OK
siggib@pve:~$ sudo qm reset 104
siggib@pve:~$

What could cause qme to freeze up like that?

 

[esi_y]

This problem has resurfaced:

siggib@pve:~$ sudo qm reset 104
trying to acquire lock...
can't lock file '/var/lock/qemu-server/lock-104.conf' - got timeout
siggib@pve:~$ sudo qm config 104
agent: 1
boot: order=scsi0;ide2;net0
cores: 1
cpu: x86-64-v2-AES
ide2: iso:iso/ubuntu-23.10.1-desktop-amd64.iso,media=cdrom,size=5052730K
memory: 2048
meta: creation-qemu=8.1.2,ctime=1706902457
name: ubuntu23
net0: virtio=BC:24:11:31:78:05,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-104-disk-0,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=a624f94a-12a2-475b-8f71-b14e942ae056
sockets: 1
tpmstate0: local-lvm:vm-104-disk-1,size=4M,version=v2.0
vmgenid: f6db3439-cacb-473b-bbea-3662e9d10290
siggib@pve:~$
siggib@pve:~$ sudo qm stop 104
trying to acquire lock...
can't lock file '/var/lock/qemu-server/lock-104.conf' - got timeout
siggib@pve:~$ sudo qm unlock 104
trying to acquire lock...
 OK
siggib@pve:~$ sudo qm reset 104
siggib@pve:~$

What could cause qme to freeze up like that?

Siggi, you would need to provide more information to troubleshoot this with others:

And also the syslog when you try to restart or shutdown the VM.

BTW Something crossed my mind, is this in a cluster or standalone node?

EDIT: I see you noted "single node" above, but unsure if you meant you restarted it on single node only or you have a single node only (no cluster)?

 

[iliadz]

Interesting enough, the problem surfaced on my system, but without an operating system actually installed. In the process of installing WIndows 10, I realized I need to change a few hardware items, and shutdown the sytem. No OS has been installed, but it is in the process of "Install Now". Meaning, ready to install. There are no visible HDs to install to, as that's the change I need to make.

Just mentioning it, so there should for sure be no reason to have the guest agent installed as there simply isn't an OS running or installed yet.

Error can't lick file '/var/lock/qemu-server/lock-300.conf" - got timeout.

I ended up just using qm to destroy the image as it wasn't mission critical and no real need to troubleshoot on my end.