How to fix: zstd on your ZFS rpool not booting

Michael Herf

Member
May 31, 2019
11
2
23
49
Had a bad morning, because I read the ZFS 2.0 release notes wrong: the rule is that ANY use of zstd on your rpool will make grub fail to boot. I thought this would apply only to the root dataset or something like that, but it's really a big deal if you use zstd anywhere in the rpool, so don't do it.

There are other bootloaders and complicated fixes, but the easiest option (which I did) is to destroy all the datasets using zstd, and then everything will boot again.

Steps:
1. Boot from a proxmox 6.4 ISO or later, choose Debug install, "exit" at the first prompt so zfs loads.
2. zpool import -f -R /mnt rpool
3. zfs get compression | grep zstd - write these all down because they will need to be destroyed
4. For each affected dataset, you can replay them to use lz4 this way:

zfs set compression=lz4 DATASET zfs create DATASET-replay zfs snapshot -r DATASET@re zfs send -R DATASET@re | zfs recv -vF DATASET-replay ... zfs destroy -rf DATASET zfs rename DATASET-replay DATASET zpool export rpool

Reboot and all is well.
 
Last edited:
Hi,

first: Thanks for sharing you're approach.

People reading this should remember that depending on the amount of datasets affected this could take a longer (down) time and cause potential some wear on the underlying storage, may be negligible if only a small amount of data and/or fast enterprise SSDs, but still good to have in mind IMO.

The switch over to a simple vfat boot partition is not that hard and provides an actual long-term solution for avoiding issues with the ZFS support in GRUB in the future too, see:
https://pve.proxmox.com/wiki/ZFS:_Switch_Legacy-Boot_to_Proxmox_Boot_Tool

It may be good to know that it can also be done when already affected, see the section here:
https://pve.proxmox.com/wiki/ZFS:_S...iring_a_System_Stuck_in_the_GRUB_Rescue_Shell

Just mentioning so that people finding this thread know the full range of option, your solution is def. workable and surely useful to know for some!
 
Last edited:
  • Like
Reactions: janssensm
Thanks for this!

Yes, I had found proxmox-boot-tool, but only one of my machines (installed recently) had a 512M volume - the ones installed with older Proxmox installers were smaller. So I was not ready to resize partitions in a boot recovery shell. But perhaps in the future...
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!