Stuck in grub rescue during boot with zfs

[DeinFreund]

I recently rebooted a proxmox 4.5 machine after having it running a rather long time and it is now stuck in grub rescue. It uses root on zfs.

error: no such device: c747d64b46956a22
error: unknown filesystem.
Entering rescue mode...
grub rescue> ls
(hd0) (hd0, gpt3) ... [all disks and partitions are listed]
grub rescue> insmod zfs
grub rescue> ls (hd0)
(hd0): Filesystem is unknown.
grub rescue> ls (hd0,gpt1)
(hd0, gpt1): Filesystem is unknown.
grub rescue> ls (hd0,gpt2)
(hd0, gpt2): Filesystem is unknown.
grub rescue> ls (hd0,gpt3)
(hd0, gpt3): Filesystem is unknown.
grub rescue> ls (hd0)
(hd0): Filesystem is unknown.
grub rescue> insmod normal
error: unknown filesystem.
grub rescue> set
cmdpath=(hd0)
prefix=(hd0)/ROOT/pve-1@/boot/grub
root=hd0
grub rescue> ls (hd0)/ROOT/pve-1@/boot/grub
error: unknown filesystem
grub rescue>

The most promising approach in my eyes has been the following:
1. Boot using a Ubuntu live cd
2. Import the zfs pool
3. Mount proc, dev, sys and chroot into the pool
4. Run apt dist-upgrade
5. update-initramfs && update-grub
6. grub-install to all disks
7. exit chroot and export the zfs pool
8. reboot

Unfortunately that hasn't changed anything though. I believe some part of the configuration or kernel must've been corrupted. How do I verify/replace those files? What other steps would be necessary to properly reinstall grub?

The zfs pool itself seems to be fine. There have been no issues accessing it from Ubuntu and it has recently performed a scrub without errors.

 

[DeinFreund]

I've tried inserting another disk and installing grub there, but it behaves exactly the same (so at least the grub installation is working, it's just installing a broken config). Copying over the images in /boot from a fresh proxmox install also didn't change anything.

As a workaround, I've now moved /boot to a separate ext4 partition on the non-zfs disk and installed grub there. This loses me the whole redundancy advantage of having zfs though.

 

[Kaijia]

Are you using HP server with HP Smart Array? It seems other people had the same issue too all HP server, e.g.

https://forum.proxmox.com/threads/grub-errors-hp-servers.62009/
https://forum.proxmox.com/threads/p...ar-and-sas-drives-final-recommendation.44694/

I got the exact same issue today just after a routine restart, tried update-grub and pve-efiboot-tool, cannot make it work.