Oh no! "Error: checksum verification failed. Entering rescue mode ..."

[jbannow]

New to ProxMox, only installed it a few weeks ago. Last night, it locked up and I had to do a hard reboot. Upon rebooting, it comes up to:

"Error: checksum verification failed.
Entering rescue mode ..."

Tried rebooting and obviously it comes up to the same prompt.

ProxMox should be running on the latest version (ran updates within the last week.)

I'm pretty new to ProxMox and Linux systems in general. Any pointers on how to recover from this would be really appreciated.

 

[t.lamprecht]

Hi,

were you using ZFS? You could try to boot the installer ISO, select "Debug" mode, skip the first shell prompt, and on the second one try:

zpool import

and post the results.
Maybe you had a failed disk.

 

[jbannow]

 

[jbannow]

Seems like the RAID arrays are OK?

Thanks for the assistance BTW!

 

[jbannow]

Still not booting. Suggestions on what to check to see what the error is?

 

[t.lamprecht]

Seems like the RAID arrays are OK?

Yes, it really seems that way, which is good news.

So, the next thing I suspect is a broken grub/bootloader, whyever that happened. I mean grub's support for ZFS is somewhat shaky, and using (U)EFI boot mode for new installations is recommended (just as pointer for other reading this, or if you setup a new PVE sometime in the future).

Do you see the initial GRUB bootmenu?

You may want to check out:
https://forum.proxmox.com/threads/grub-rescue-checksum-verification-failed.39143/#post-193794
https://forum.proxmox.com/threads/grub-rescue-error-checksum-verification-failed.52730/#post-248073

Maybe @fabian has an idea, he spend quite a bit of time with ZFS+GRUB

 

[fabian]

if this is a PVE 6.x install, I highly recommend using UEFI boot instead of legacy/BIOS.

it is likely (yet) another occurence of "Grub's ZFS code hits some on-disk edge-case that is improperly handled", which caused us to deprecate Grub+BIOS in favor of systemd-boot+UEFI in 6.0. you might (temporarily) get out of this instance of the issue by booting a live-CD/rescue environment , importing + mounting + chrooting your pool/rootfs and re-installing the kernel you are attempting to boot (it re-writes the kernel image and initrd), but there are no guarantees that it will help, nor that it won't return at some random point in the future.

 

[jbannow]

Thanks @fabian This system was built up new on v6.

Is there a way to safely reinstall as UEFI from where it is at now? Or would I need to fix the system, find somewhere to evacuate the data to, and reinstall?

 

[t.lamprecht]

Is there a way to safely reinstall as UEFI from where it is at now?

If the server/motherboard and it's firmware support UEFI there'd be a way.

The installer should already have added a ~512M big partiton on the disks used by the ZFS rpool in preparation for hosting an EFI partition. First, confirm that those partitions are already there and have the EFI type:

lsblk -o +PARTTYPE | grep -i C12A7328-F81F-11D2-BA4B-00A0C93EC93B

(above weird string is the UUID for EFI partitions).

If that lists a bunch of partitions (from your screenshot I'd guess four), all 512M big, you can just do

pve-efiboot-tool init /dev/<PARTITION>

Repeat that for all EFI paretitions, then you should be ready to reboot and enable EFI mode in the BIOS/UEFI settings (may need to disable secure boot, or set the boot mode to "Linux" (depends on the specific firmware))

If you are unsure just post the output from lsblk and what you'd do, I can take a look if those seem reasonable.

Also, you can always go back and disable EFI in bios if something goes wrong here, GRUB won't get removed automatically, so you can only win - so to say

 

[jbannow]

Well, bad news (for me.) My Supermicro H8DGU-F motherboard doesn't support EFI / UEFI.

Any suggestions on where to go from this point? Will I need to convert the boot disk into ext4 somehow?

 

[t.lamprecht]

Will I need to convert the boot disk into ext4 somehow?

You could add a small ext4 boot disk, for only the base system..
It would be even enough if it only holds /boot, I'd guess..

Effectively, you could mkfs.ext4 the 512MB partition at the start, the one what is now reserved for EFI - in your case it has not much other use anyway..
In the debug mode of the installer, you could mount that new ext4 partition to a mountpoint, then import the rpool ZFS pool at another mountpoint, copy over the /boot to this new ext4 mounpoint and move it to /boot.bak in the ZFS pool to avoid issues of cannot use non-empty directories for mountpoints at the next boot.
The grub.cfg from the new boot partition may need some changes, but it then should be bootable from ext4..
The kernel commandline should be left intact, as ZFS as Linux root is still valid, mostly the set root='...' needs to be changed.. I'd guess to something like hd0,msdos1 or the like..

I can try to play this through and give you more clear pointers next week if you're interested into that.

 

[jbannow]

I can try to play this through and give you more clear pointers next week if you're interested into that.

If you're able to do that would be very helpful. Thank you!

 

[jbannow]

@t.lamprecht If you get a chance to help that would be very appreciated.

 

[rholighaus]

Hi jnannow,

check the thread in the article below, in particular tge comments. I have used the procedure to move /boot to both USB sticks and SD cards on various PVE servers and they run very well, both with PVE 5.x and 6.x

https://www.reddit.com/r/homelab/comments/ap9usf/proxmoxzfs_installed_on_hp_dl360p/