Does Proxmox manage hibernation ?

[wireless]

Ok, putting the resume in the grub... the machine now hibernates and can be woken up but it wake and then restart! Sob

 

[jimklimov]

Thanks to the pointers in this discussion, I got my ProxMox VM (demoing it to colleagues, so PVE ISO was installed into a Hyper-V VM of all things) with kernel 5.15.74-1-pve hibernating at last. The ride was bumpy, and I'm sure I had tried the combo which works for me now some time along the way and it did not


So, one thing first: use a dedicated swap partition. A zfs volume seems to have been read upon resume, but logs were lost (many messages about un-aligned IO to the HyperV-backed disk flooded the kernel buffer, I suppose) so I can't quickly say why it did not work. Perhaps that can also be done.

I went with least resistance and added another vHDD (sdb below), sized a healthy bit larger than the VM RAM. (The alternative would be figuring out ZFS volume support, or resize the rpool so you have a new RAM+-sized partition on the only disk you have... maybe you can extend the disk "hardware" if it is provisioned outside the sandbox like on a NAS in a bare-metal setup.)


Some caveats about earlier posts in this thread:

* resume=UUID=... must use upper-cased "UUID" (some examples went resume=uuid=... which failed to be recognized).

* Allegedly other ways to specify the block device underlying your swap can also work, like resume=/dev/mapper/VolGroup00-lvolswap or resume=/dev/disk/by-uuid/... or plain resume=/dev/sdb1, but I did not try those.

* The /etc/initramfs-tools/conf.d/pve-initramfs.conf should not remove/comment-away the RESUME=none line (as I first understood that post), but rather replace it with RESUME=UUID=...

* The update-* programs take a minute to do their magic, but do not add (or actually do remove any earlier manually added) resume=... kernel boot option in the EFI loader pre-set. I would take this as a PVE bug, although probably one they would not accept as they went some lengths to prohibit hibernation in the first place. Maybe there are some further integration options to set somewhere (posts from other distros suggested `mkinitcpio.conf` should have a "resume" hook mentioned, for example).


Saving my notes here, hopefully as a reproducible how-to


* Partition the swap disk, if you too attach a new one:

:; fdisk /dev/sdb
# g (for new GPT label)
# c (create partition) => accept all for full disk size
# t (type) => swap
# w (write new table and exit)

For the purposes of this example, you should have a /dev/sdb1 now (or adapt the copy-pasta accordingly).

* Format the new partition as swap; remember the UUID (or set mine with the -U parameter to mkswap), e.g.:

:; mkswap -L hiberswap -U f7113877-affb-4f45-bf6b-d9a998f09d2b /dev/sdb1
Setting up swapspace version 1, size = 6 GiB (6441377792 bytes)
LABEL=hiberswap, UUID=f7113877-affb-4f45-bf6b-d9a998f09d2b

* Edit /etc/fstab to auto-mount this swap (must be mounted for hibernation to save data into it), e.g. add this line:
UUID=f7113877-affb-4f45-bf6b-d9a998f09d2b swap swap defaults 0 0

* NOTE: I have no clear idea if editing these files is needed or helpful, as long as they botch the EFI pre-set for kernel boot options.

** In /etc/initramfs-tools/conf.d/pve-initramfs.conf comment away the RESUME=none line, and add this line instead:
RESUME=UUID=f7113877-affb-4f45-bf6b-d9a998f09d2b

** In /etc/default/grub find the lines GRUB_CMDLINE_LINUX_DEFAULT and GRUB_CMDLINE_LINUX, and into their quoted contents append:
resume=UUID=f7113877-affb-4f45-bf6b-d9a998f09d2b


** Run the tools to apply these edited files in whatever way they see fit:

:; update-initramfs -k all -u && update-grub

* Check where your EFI loader is (if used), e.g. here in lsblk output it is the only FAT partition, as sda2 (note also the ZFS rpool/swap volume from the other half of this experiment, seen as zd0 block device):

:; lsblk --fs
NAME   FSTYPE     FSVER LABEL  UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
├─sda1
├─sda2 vfat       FAT32        FC19-C9FD
└─sda3 zfs_member 5000  rpool  17394304370527845472
sdb
└─sdb1 swap       1     swap6g f7113877-affb-4f45-bf6b-d9a998f09d2b                [SWAP]
sr0
zd0    swap       1            f85df210-d5c3-4f63-8b80-dfe727c1a137

* Check that EFI loader pre-set did get generated with the resume=UUID=... part in the options line, fix (add it) if not.
NOTE: it seems that the boot=zfs option must be last on the line.
Assuming /dev/sda2 is it (per above):

:; mount /dev/sda2 /boot/efi && \
    vi /boot/efi/loader/entries/*.conf && \
    umount /boot/efi

### options  root=ZFS=rpool/ROOT/pve-1 resume=UUID=f7113877-affb-4f45-bf6b-d9a998f09d2b boot=zfs

** NOTE: You would probably have to repeat this bit whenever you update PVE kernel or otherwise re-generate initrd/grub parts.

* Finally, reboot so the kernel knows all these settings.

* Then in the new up-time try systemctl hibernate. It should write the swapping progress for a few seconds on the console, and power off PVE box (VM).

* Upon a power-on it would mention "suspend/resume" and maybe "Running /scripts/local-block" a few times. This may actually mean it struggles to find the info, or it is inconsistent (later dmesg or journal -xn can expose why, e.g. I saw "PM: hibernation: Image mismatch: memory size"/"PM: Error -1 resuming"/"PM: hibernation: Failed to load image, recovering." a few times). NOTE: If you are using a PVE as a VM and seeing these particular errors, check that its RAM size is fixed on the hypervisor and not "dynamic".

Ideally it would quickly restore the saved data during boot. In my test, a remote SSH session did not notice the server was gone, although it did disconnect a few seconds after seeing it again (may be due to SSH client/server keepalive - sessions just survived a hibernation at other times and remained usable, or disconnected during VM boot). YMMV

 

[effgee]

Yet to try all your steps here, but I have too struggled with this issue and really happy to find someone who managed to figure it out.
When I do get the courage (I already spent... many many hours) trying to get it working.

In my case, I actually only wanted suspend to ram working but the same suspend, then wake up.... it would wake and reboot I kept encountering.

If I manage it I will update. Thanks @jimklimov

Thanks to the pointers in this discussion, I got my ProxMox VM (demoing it to colleagues, so PVE ISO was installed into a Hyper-V VM of all things) with kernel 5.15.74-1-pve hibernating at last. The ride was bumpy, and I'm sure I had tried the combo which works for me now some time along the way and it did not


So, one thing first: use a dedicated swap partition. A zfs volume seems to have been read upon resume, but logs were lost (many messages about un-aligned IO to the HyperV-backed disk flooded the kernel buffer, I suppose) so I can't quickly say why it did not work. Perhaps that can also be done.

I went with least resistance and added another vHDD (sdb below), sized a healthy bit larger than the VM RAM. (The alternative would be figuring out ZFS volume support, or resize the rpool so you have a new RAM+-sized partition on the only disk you have... maybe you can extend the disk "hardware" if it is provisioned outside the sandbox like on a NAS in a bare-metal setup.)


Some caveats about earlier posts in this thread:

* resume=UUID=... must use upper-cased "UUID" (some examples went resume=uuid=... which failed to be recognized).

* Allegedly other ways to specify the block device underlying your swap can also work, like resume=/dev/mapper/VolGroup00-lvolswap or resume=/dev/disk/by-uuid/... or plain resume=/dev/sdb1, but I did not try those.

* The /etc/initramfs-tools/conf.d/pve-initramfs.conf should not remove/comment-away the RESUME=none line (as I first understood that post), but rather replace it with RESUME=UUID=...

* The update-* programs take a minute to do their magic, but do not add (or actually do remove any earlier manually added) resume=... kernel boot option in the EFI loader pre-set. I would take this as a PVE bug, although probably one they would not accept as they went some lengths to prohibit hibernation in the first place. Maybe there are some further integration options to set somewhere (posts from other distros suggested `mkinitcpio.conf` should have a "resume" hook mentioned, for example).


Saving my notes here, hopefully as a reproducible how-to


* Partition the swap disk, if you too attach a new one:

:; fdisk /dev/sdb
# g (for new GPT label)
# c (create partition) => accept all for full disk size
# t (type) => swap
# w (write new table and exit)

For the purposes of this example, you should have a /dev/sdb1 now (or adapt the copy-pasta accordingly).

* Format the new partition as swap; remember the UUID (or set mine with the -U parameter to mkswap), e.g.:

:; mkswap -L hiberswap -U f7113877-affb-4f45-bf6b-d9a998f09d2b /dev/sdb1
Setting up swapspace version 1, size = 6 GiB (6441377792 bytes)
LABEL=hiberswap, UUID=f7113877-affb-4f45-bf6b-d9a998f09d2b

* Edit /etc/fstab to auto-mount this swap (must be mounted for hibernation to save data into it), e.g. add this line:
UUID=f7113877-affb-4f45-bf6b-d9a998f09d2b swap swap defaults 0 0

* NOTE: I have no clear idea if editing these files is needed or helpful, as long as they botch the EFI pre-set for kernel boot options.

** In /etc/initramfs-tools/conf.d/pve-initramfs.conf comment away the RESUME=none line, and add this line instead:
RESUME=UUID=f7113877-affb-4f45-bf6b-d9a998f09d2b

** In /etc/default/grub find the lines GRUB_CMDLINE_LINUX_DEFAULT and GRUB_CMDLINE_LINUX, and into their quoted contents append:
resume=UUID=f7113877-affb-4f45-bf6b-d9a998f09d2b


** Run the tools to apply these edited files in whatever way they see fit:

:; update-initramfs -k all -u && update-grub

* Check where your EFI loader is (if used), e.g. here in lsblk output it is the only FAT partition, as sda2 (note also the ZFS rpool/swap volume from the other half of this experiment, seen as zd0 block device):

:; lsblk --fs
NAME   FSTYPE     FSVER LABEL  UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
├─sda1
├─sda2 vfat       FAT32        FC19-C9FD
└─sda3 zfs_member 5000  rpool  17394304370527845472
sdb
└─sdb1 swap       1     swap6g f7113877-affb-4f45-bf6b-d9a998f09d2b                [SWAP]
sr0
zd0    swap       1            f85df210-d5c3-4f63-8b80-dfe727c1a137

* Check that EFI loader pre-set did get generated with the resume=UUID=... part in the options line, fix (add it) if not.
NOTE: it seems that the boot=zfs option must be last on the line.
Assuming /dev/sda2 is it (per above):

:; mount /dev/sda2 /boot/efi && \
    vi /boot/efi/loader/entries/*.conf && \
    umount /boot/efi

### options  root=ZFS=rpool/ROOT/pve-1 resume=UUID=f7113877-affb-4f45-bf6b-d9a998f09d2b boot=zfs

** NOTE: You would probably have to repeat this bit whenever you update PVE kernel or otherwise re-generate initrd/grub parts.

* Finally, reboot so the kernel knows all these settings.

* Then in the new up-time try systemctl hibernate. It should write the swapping progress for a few seconds on the console, and power off PVE box (VM).

* Upon a power-on it would mention "suspend/resume" and maybe "Running /scripts/local-block" a few times. This may actually mean it struggles to find the info, or it is inconsistent (later dmesg or journal -xn can expose why, e.g. I saw "PM: hibernation: Image mismatch: memory size"/"PM: Error -1 resuming"/"PM: hibernation: Failed to load image, recovering." a few times). NOTE: If you are using a PVE as a VM and seeing these particular errors, check that its RAM size is fixed on the hypervisor and not "dynamic".

Ideally it would quickly restore the saved data during boot. In my test, a remote SSH session did not notice the server was gone, although it did disconnect a few seconds after seeing it again (may be due to SSH client/server keepalive - sessions just survived a hibernation at other times and remained usable, or disconnected during VM boot). YMMV