[SOLVED] Found out hard way you can't do a snapshot while VM is running, bricked it. Is there a way to recover from this?

[Red Squirrel]

I have a backup that's a few days old so my fall back plan is to just restore it, but I am wondering if this is something that I can recover from in case it ever happens to a VM with no backup or a much older backup.

I did a live snapshot, thinking it would work, or at least tell me I can't do it if it's not doable, but it started anyway but it got stuck and never finished. I shutdown the VM thinking it would make it finish but it just bricked the VM.

This is the error that I get:

Requesting HA start for VM 115
service 'vm:115' in error state, must be disabled and fixed first
TASK ERROR: command 'ha-manager set vm:115 --state started' failed: exit code 255

I tried to set HA to ignore but it says the VM is locked so can't do that either. I forgot to do it at first so maybe that was part of my issue, it probably tried to start back up after I shut it down.

Also ran

 qm cleanup 115 0 0

As well as

rm /var/lock/qemu-server/lock-115.conf

Anything else I can try to recover this? I also tried to delete the lock but it just comes back. Basically I can't do anything with the VM in it's current state.



Edit: SOLVED!

Ended up using Grok... I really try not to rely on AI for these things but sometimes, it just works!

So here's more or less the steps I had to do in case anyone ever runs into this:

Some of these may be redundant and repeated, just following exactly what my console looked like that lead to it working.

qm unlock 115
ha-manager set vm:115 --state disabled
rm /var/lock/qemu-server/lock-115.conf
ha-manager set vm:115 --state disabled
qm delsnapshot 115 b4update --force
ha-manager set vm:115 --state disabled
ha-manager set vm:115 --state ignored

I was then able to start the VM. Not sure what state it's in, if it's after the snapshot, or before. I assume it's before.

So yeah, live from my fail... turn OFF vm before taking snapshot! Could have sworn I had done live snapshots before though... Maybe it only works on quiet VMs and not ones writing lot of data such as CCTV.

 

[LnxBil]

I did a live snapshot, thinking it would work, or at least tell me I can't do it if it's not doable, but it started anyway but it got stuck and never finished. I shutdown the VM thinking it would make it finish but it just bricked the VM.

There has to be something else at fault here. Normally, there is not problem doing a live snapshot. I don't do it because I don't like crash consistency, but at least nothing normally breaks if you do so.

The unlock step was correct, but in the end, it should not have gone there in the first place. Can you try to recreate the problem? Can you explain more about your setup and underlying storage structure?

 

[Onslow]

Of course, live snapshots are supported.

https://pve.proxmox.com/wiki/Live_Snapshots

"By using Proxmox VE live snapshots you can preserve the KVM virtual machine state. A snapshot includes the contents of the virtual machine memory, virtual machine settings, and the state of all the virtual disks. When you rollback to a snapshot, you restore the memory, virtual disks and all settings of the virtual machine to the state they were in when you took the snapshot.
[...]
In order to use Proxmox VE live snapshots all your virtual machine disk images must be stored as qcow2 image or be in a storage that supports live snapshots and is managed by Proxmox VE. For a list of storage with snapshot capabilities, see Storage"

 

[Red Squirrel]

There has to be something else at fault here. Normally, there is not problem doing a live snapshot. I don't do it because I don't like crash consistency, but at least nothing normally breaks if you do so.

The unlock step was correct, but in the end, it should not have gone there in the first place. Can you try to recreate the problem? Can you explain more about your setup and underlying storage structure?

I have a 3 node cluster, storage is a NAS (just normal Linux with mdraid) running NFS. The disks are qcow2. Although I don't know about the live snapshot support on that setup so maybe that's my issue. I was able to shutdown the VM to do the snapshot though.

Now I'm trying to delete it live and it looks like I might run into the same issue... Maybe i will stick to just doing a backup instead of a snapshot before I do anything risky like a major update within the guest.

Going to let it go to see what happens but it's not looking good, whole VM is locked right up.


Edit: So upon further investigation it seems snapshots will snap shot *ALL* drives even ones that are not marked for backup.... well there's my problem. One of the drives is 13TB. So yeah I think I will need to just stick to doing a manual backup instead of a snapshot when I want to have a rollback point.

it's still deleting right now. No progress or any indication of what's happening, I think it's stuck. I should have shut down the VM before doing this... but I figured simply deleting a snapshot would be non invasive.