Stop VM impossible

[HikariNoKitsune]

Just uh... throwing this out there.... since it is a somewhat older post...

"cancel the shutdown task at the bottom" is NOT the solution. and should never be 'the solution'. Specially since as I've just found out, THAT DOESN'T WORK.... I have a VM that is stuck trying to boot. Its not even getting past the post. I try to shut it down, but nothing happens. So what do I do? Press STOP like any other sane person would do. What happens when I click it? NOTHING... I search the net to see what's going on about it.. and see people saying that I have to reboot the host... great... welp! I guess I'll press that 'reboot' button for the host...

Oh... Crap... That's not doing anything! Craaap!

More searching....

Find people stating that 'oh you can't do that if you have a shutdown queued'....... what? why?! Why can I not reboot the host if another thing has a shutdown task queued? that should immediately attempt a shut down of EVERYTHING... and then if nothing does shut down, KILL THEM SO IT CAN.....

But nope... that isn't what happens... not even in the slightest....

More searching...

Find more of these posts.. finally one that says "well just kill the task below"... oh, ok.. well lets give that a shot...

NOPE! No stop button. no way to kill the task. the option is grayed out. Guess I'm just shit outa luck. Time to possibly corrupt everything I've been working on and just yank the freaking plug on a physical server with physical hardware... cause yup... that's a looooovely solution to this ass backwards bug that's apparently 'not a bug'. I sure keep finding these "Bugs That Are Actually Features" this week.... I'll never understand the logic behind some decisions...

 

[Dunuin]

Also keep in mind that when "QEMU guest agent" is enabled in the VMs options tab, it requires the guest agent to be running in order to shutdown a VM. If it is stuck at boot, so the guest agent never starts, PVE won't shutdown down that VM as it can't communicate with the VM.
With the "QEMU guest agent" disabled it will instead use ACPI signals to call a shutdown.

 

[HikariNoKitsune]

That is good to know. I had wondered what GEMU Guest Agent was. I figured it was something like VMTools. I've primarily used esxi in the work environment. However, this machine did not have that checked. I did find however that if secureboot is on, it causes a plethora of problems, and once that was disabled, all my issues vanished. The machine will even insta stop when clicking 'stop' when in the middle of looking for the boot drive now

Also I should apologize for my rant/venting above. I've been working all week trying to rebuild/migrate 50tb worth of VMs, Dockers, and data from an unrecoverable hardware failure... after getting new equipment to replace the damage, I figured I'd give Proxmox a go since it was built more toward what my previous servers ended up becoming. And I've been struggling to get things up and running for a week. But thankfully finding that issue with SecureBoot seems to have solved every issue I've been having.

Now if only Proxmox would support Docker Containers directly lol.

Back to the thing with QEMU however:
What is the benefit of QEMU Guest Agent being installed verses not? Is it really just the equivalent of VMTools?

 

[Dunuin]

Back to the thing with QEMU however:
What is the benefit of QEMU Guest Agent being installed verses not? Is it really just the equivalent of VMTools?

Its the only way how the PVE host can communicate with the guests. So yes, it is similar to the VMTools. It's especially important when doing snapshot mode backups, as the guest agent is needed so the PVE host can tell the guest OS to do a fsfreeze. This will for example tell the guest OS to flush the write cache before doing the backup, so data integrity can be ensured:
https://pve.proxmox.com/wiki/Qemu-guest-agent

 

[NOiSEA]

when the "Stop" job doesn't kill the VM it's probably because there's another pending "Reboot" or "Shutdown" job going on, which you can cancel in the task log at the bottom if you double click it and select "Stop" inside the task (i guess that's a bit confusing and should be rather called "Cancel" or so though...)


as mentioned, you can kill the kvm process if need be. just try:

ps aux | grep <VMID> # will show you the kvm process ID running your VMID
kill -9 <PID> # you can now kill it with the process ID

beware that this is equivalent to pulling the plug on a physical machine though

to kill the process wouldn't you need to be able to get into the proxmox shell? From my (admittedly limited) experience with proxmox, if the VM needs to be stopped, the host's shell will also be unavailable. I am trying this method right now but the shell just returns that frowny paper face thing that proxmox does when something won't load.

 

[Neobin]

Where we are on the subject:
Why is the "Bulk Stop" named so, while it actually does (fortunately) not stop the VMs/LXCs, but rather shuts them down?
"Bulk Shutdown" with the appropriate icon would be correct.
Currently the naming is inconsistent and basically wrong or do I miss something?

Ah, okay. Thank you for the explanation.

Maybe I am the only one who thinks so, but I had to google first what it actually does, because I was scared that it does directly (force) stop the VMs/LXCs, like the per machine "Stop"-button does.

At least, I am not the only one :
https://bugzilla.proxmox.com/show_bug.cgi?id=4194

What we'd like to change in the shutdown/stop procedure though is that a later stop trumps an earlier, still running, shutdown, basically overruling it. That makes sense to have for convenience.

I opened a ticket for better tracking:
https://bugzilla.proxmox.com/show_bug.cgi?id=4474

 

[Vorl]

One thing mentioned here was the QEMU guest agent was a checkbox. Why is it a checkbox instead of being auto detected? It would also be nice if it somehow checked the installed version so we knew when we needed to update things.

 

[t.lamprecht]

One thing mentioned here was the QEMU guest agent was a checkbox. Why is it a checkbox instead of being auto detected?

Because one needs to add extra (not hot un-pluggable) virtual hardware that takes up resources, is an extra communication channel between guest and host (so also has some security implications) and might confuse the guest OS in general.

 

[Vorl]

Because one needs to add extra (not hot un-pluggable) virtual hardware that takes up resources, is an extra communication channel between guest and host (so also has some security implications) and might confuse the guest OS in general.

Thats the only way to do it? Seems like you could just poll the OS every so often to query if the agent is there. I don't understand the under the hood stuff to know why it requires "not hot un-pluggable" virtual hardware, but that has to be when the guest is running the tools, so it would be or not be extra resources based on if the guest is running the guest agent software. So extra checking would be a minimal additional resources server side.

As for extra communication channels between the host and guest, that's only when the agent is installed as well, and that's the choice we as admins make by installing the guest stuff (which everyone should be doing anyway, if for no other reason than the balloning memory driver) and the other stuff like the snapshotting and drivers it includes.

Are you saying it's not recommended that we install the qemu guest agent/drivers?

 

[t.lamprecht]

Thats the only way to do it? Seems like you could just poll the OS every so often to query if the agent is there.

How do you "poll the OS" if you have not added the HW that adds the communication channel to "poll the OS"?
And even if you'd add the HW you couldn't tell for sure just from a few requests, it's more robust if the admin controls that. If, I could imagine adding a way to adapt the default for that (there are some ideas floating around for default management).

So extra checking would be a minimal additional resources server side.

And you determined that from what basis? Did you implement it and test it or is this just pure speculation?

Are you saying it's not recommended that we install the qemu guest agent/drivers?

Note that the QEMU Guest Agent and the VirtIO drivers are two very different things; the latter is like any other driver, in general unproblematic as longs its kept up to date to get the latest (security) bug fixes. The former, QGA, is a bit more intrusive as it's a full-blown daemon running in the guest OS, and some trust (especially from guest towards the host its running on) should be there when using it. I know nothing about your setup and if you need assistance w.r.t. such decisions I'd recommend asking our enterprise support, if you got an eligible enterprise subscription for the host(s) in question, if not see:
https://www.proxmox.com/proxmox-ve/pricing
https://my.proxmox.com/

 

[dwoodroofe]

hi,

you should be able to force stop the VMs by hitting "Stop" on the power menu.

if that is not working, then it might be because you have another "Shutdown" or "Reboot" pending... you can cancel that in the task log (at the bottom).

if none of that works for some reason, the most forceful way of killing the VM without rebooting the host would be to just kill the kvm process for that VM

This feature has not worked for several years.

 

[wolframio]

Agreed!!! we need a "FORCE KILL" button ASAP! this is a must be proxmox feature.

 

[b.miller]

Stop works fine for our production cluster. If a shutdown/reboot is hung, cancel that task and hit Stop. Only very rarely has this failed to work on a problematic guest and I had to do any CLI intervention.

 

[ivenae]

Just to give my 2 cents: I know this 'bug' not being able to stop a virtual machine since i use proxmox and i still make a lot use of a 'kill -9', because the stop never works for me in exactly those special conditions, where you normally use a 'force stop'.

Mostly the 'stop' does not work if i forget to set a correct boot device. The VM is then in a boot-loop, searching for a boot device and stopping the machine is impossible via GUI.

Other conditions i need a 'force stop' and the 'stop' won't work is if a machine hangs. Ping is not possible, 'console' shows no output and the machine does not react to any 'shutdown' command. Then i use the 'stop' and this does not work as expected. I use a 'kill -9' then.

It won't work so often that i normally do not even try to stop a machine via GUI, i always use 'kill -9'.

It would be great if we'll get a always working stop command.

Don't get me wrong: I love Proxmox VE, i can help myself with a 'kill -9', but the non working 'stop' command is a 'meme' in proxmox.
"Hey, I'm new to proxmox, can you explain me why the stop command does not kill my hanging machine?"
- "Oh, stop command never works, you have to kill it with 'kill -9'. Bug since ever, won't fix!"

 

[Dunuin]

Ping is not possible, 'console' shows no output and the machine does not react to any 'shutdown' command. Then i use the 'stop' and this does not work as expected. I use a 'kill -9' then.

Again: Stop that stuck shutdown task (double-click its task at the bottom of the webUI and hit the tasks "stop" button). then you can start a new stop task that kills the VM. So no need for "kill -9". People usually don't get that this shutdown task is still running and the stop task is waiting for the shutdown task to finish before doing something.

 

[BobhWasatch]

On the Internet nobody reads the manual but everyone is a developer

 

[abdo75]

From what I observed, shutting a VM down is bugging/not working. However, if you try to click on Stop instead, it seems to be working. I have been doing this now instead of a bulk shut down. I have no clue what Poweroff is really supposed to do now but Stop does the job

 

[Dunuin]

I have no clue what Poweroff is really supposed to do now but Stop does the job

Stop will kill it, instead of a graceful shutdown. So it's like pulling a virtual power plug while the VM is running and you will lose some data.

 

[abdo75]

Stop will kill it, instead of a graceful shutdown. So it's like pulling a virtual power plug while the VM is running and you will lose some data.

Thank you very much for the quick reply

 

[ivenae]

From what I observed, shutting a VM down is bugging/not working. However, if you try to click on Stop instead, it seems to be working. I have been doing this now instead of a bulk shut down. I have no clue what Poweroff is really supposed to do now but Stop does the job

Shutdown will only work, if you have installed and activated the Guest Agent. The Agent is normally installed in any Linux distribution (but you have to activate is in VM options), but you have to install it by yourself on a windows machine.