How to fix: zstd on your ZFS rpool not booting

Michael Herf

Active Member
May 31, 2019
12
2
43
49
Had a bad morning, because I read the ZFS 2.0 release notes wrong: the rule is that ANY use of zstd on your rpool will make grub fail to boot. I thought this would apply only to the root dataset or something like that, but it's really a big deal if you use zstd anywhere in the rpool, so don't do it.

There are other bootloaders and complicated fixes, but the easiest option (which I did) is to destroy all the datasets using zstd, and then everything will boot again.

Steps:
1. Boot from a proxmox 6.4 ISO or later, choose Debug install, "exit" at the first prompt so zfs loads.
2. zpool import -f -R /mnt rpool
3. zfs get compression | grep zstd - write these all down because they will need to be destroyed
4. For each affected dataset, you can replay them to use lz4 this way:

zfs set compression=lz4 DATASET zfs create DATASET-replay zfs snapshot -r DATASET@re zfs send -R DATASET@re | zfs recv -vF DATASET-replay ... zfs destroy -rf DATASET zfs rename DATASET-replay DATASET zpool export rpool

Reboot and all is well.
 
Last edited:
Hi,

first: Thanks for sharing you're approach.

People reading this should remember that depending on the amount of datasets affected this could take a longer (down) time and cause potential some wear on the underlying storage, may be negligible if only a small amount of data and/or fast enterprise SSDs, but still good to have in mind IMO.

The switch over to a simple vfat boot partition is not that hard and provides an actual long-term solution for avoiding issues with the ZFS support in GRUB in the future too, see:
https://pve.proxmox.com/wiki/ZFS:_Switch_Legacy-Boot_to_Proxmox_Boot_Tool

It may be good to know that it can also be done when already affected, see the section here:
https://pve.proxmox.com/wiki/ZFS:_S...iring_a_System_Stuck_in_the_GRUB_Rescue_Shell

Just mentioning so that people finding this thread know the full range of option, your solution is def. workable and surely useful to know for some!
 
Last edited:
  • Like
Reactions: janssensm
Thanks for this!

Yes, I had found proxmox-boot-tool, but only one of my machines (installed recently) had a 512M volume - the ones installed with older Proxmox installers were smaller. So I was not ready to resize partitions in a boot recovery shell. But perhaps in the future...