Removing "hibernate" state of vm? Proxmox 5.4

[holr]

Hello,
I went through the process of hibernating all VMs on a 5.4 proxmox server recently (qm suspend --vmid <id> --todisk 1) before a server reboot.

After rebooting, I tried resuming all the vms (qm resume <vmid>) but not all of them woke back up, many of them not thawing but timing out en masse.

Is it possible to delete the hibernate state and have the VM just start up, without the restoration of RAM etc.? My snapshots are from some time ago and I'd like to try just booting, without the restoring!

 

[dcsapak]

first it would be interesting why they are not starting, because this should not happen...

otherwise, you can try to boot when you manually edit the config (be sure to know what you are doing, this circumvents all checks from us) to
remove the 'suspend' lock and the 'vmstate' (you can then delete the file/lv etc. of the state via pvesm (see man pvesm))

then it should start 'normally' but behaves like the power was cut

 

[holr]

first it would be interesting why they are not starting, because this should not happen...

otherwise, you can try to boot when you manually edit the config (be sure to know what you are doing, this circumvents all checks from us) to
remove the 'suspend' lock and the 'vmstate' (you can then delete the file/lv etc. of the state via pvesm (see man pvesm))

then it should start 'normally' but behaves like the power was cut

Thank you for the suggestion! I will try this, in the meantime here is the output I get after attempting a resume on several of the vms in question:

qm resume 534
Resuming suspended VM
start failed: command '/usr/bin/kvm -id 534 -name <vm-name> -chardev 'socket,id=qmp,path=/var/run/qemu-server/534.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/534.pid -daemonize -smbios 'type=1,uuid=658c52c7-b345-4eff-a9fa-9a43207f0996' -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/534.vnc,x509,password -no-hpet -cpu 'kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,enforce' -m 16384 -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'vmgenid,guid=c8e06ffe-b515-43f0-9495-f98d14149941' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'VGA,id=vga,bus=pci.0,addr=0x2' -chardev 'socket,path=/var/run/qemu-server/534.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:c59960939514' -drive 'file=rbd:ceph_vm/vm-534-disk-0:conf=/etc/pve/ceph.conf,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap534i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=02:1D:5D:FC:80:FC,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'type=pc-i440fx-3.0' -global 'kvm-pit.lost_tick_policy=discard' -loadstate 'rbd:ceph_vm/vm-534-state-suspend-2019-07-09:conf=/etc/pve/ceph.conf'' failed: got timeout

 

[dcsapak]

the default timeout for the start is 30 seconds normally and 300 second if hugepages are on or we resume from a suspended state, but it seems it takes more than 5 minutes to load the 32GB state from ceph in your case...
maybe we should make the timeout configurable...

 

[holr]

Thank you for the suggestion! Is there anywhere I can manually change the timeout, just to try it? In a config file or something on the system?

update - I was able to use "pvesm free ceph_vm:vm-534-state-suspend-2019-07-09" (which I got from vmstate in the 534.conf file) to erase the hibernation file as you suggested. After removing the lock state and the vmstate entries from the corresponding conf file, the button changed from resume to start, and the VM started up ok - thanks for the tip!

 

[masgo]

the default timeout for the start is 30 seconds normally and 300 second if hugepages are on or we resume from a suspended state, but it seems it takes more than 5 minutes to load the 32GB state from ceph in your case...
maybe we should make the timeout configurable...

Is there already a bug report for this which I could follow?

I plan to set up the server in a way that if the UPS looses power the server waits for 5min for the power to come back and then it suspends all VMs and shuts down. Gracefully shutting down all VMs might take way to long, especially since some are Windows VMs.

Obviously, this is only a good plan if I can resume the VMs after the power has returned.

 

[voarsh]

the default timeout for the start is 30 seconds normally and 300 second if hugepages are on or we resume from a suspended state, but it seems it takes more than 5 minutes to load the 32GB state from ceph in your case...

I often get timeouts for 6-9 GB's of RAM when resuming VM's - a bit frustrating. It's "spinning rust", even using striped HDD's which is faster than a single HDD.
Any news about making the resume timeout configurable?

 

[Vasily Poterjyko]

the default timeout for the start is 30 seconds normally and 300 second if hugepages are on or we resume from a suspended state, but it seems it takes more than 5 minutes to load the 32GB state from ceph in your case...
maybe we should make the timeout configurable...

I ran into exactly the same problem. Restoring a heavy machine with >270Gb RAM. The state of the machine is on SSD disks, reading the entire volume of this file takes less than 300 seconds, but I still get the timeout of the machine restore operation from hibernate.
How to increase VM recovery timeout?
I also want to note that saving the state of the virtual machine to disk took more than 11 minutes.

https://paste.ofcode.org/38KgvzZYjvexytd6JZLiAtY

Spoiler: pve-version

proxmox-ve: 7.1-1 (running kernel: 5.13.19-4-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-12
pve-kernel-5.13: 7.1-7
pve-kernel-5.11: 7.0-10
pve-kernel-5.13.19-4-pve: 5.13.19-9
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.13.19-1-pve: 5.13.19-3
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 15.2.14-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-3
libpve-guest-common-perl: 4.1-1
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-2
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-6
pve-cluster: 7.1-3
pve-container: 4.1-4
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-5
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-2
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1