ProxMox 5.1 ZFS Fresh Installation unable to Boot due to grub

Jan 29, 2018
15
3
3
33
We are running into issues with getting a zfs rpool to boot after the PVE installation completes.

Recently we have been using ZFS more and more and are working on setting a new server to use ZFS instead of a hardware raid card.

The server we are installing onto has the following specs.

Code:
CPU: 2 x Intel Xeon Gold 6140, 2.3GHz (18-Core, HT, 2666 MT/s, 140W) 14nm
RAM: 192GB (12 x 16GB DDR4-2666 ECC Registered 1R 1.2V RDIMMs) Operating at 2666 MT/s Max
I/O Controller: Dual-Port Intel X540 10GbE Controller (RJ45)
LP PCIe 3.0 x8 internal: Supermicro 12Gb/s SAS HBA (LSI 3008), 8-Port Internal, RAID 0,1,10 - up to 63 Devices
Drive Set 1: 4 x Intel 480GB DC S4600 Series 3D TLC (6Gb/s, 3 DWPD) 2.5" SATA SSD
Motherboard: SuperMicro X11DPU

The LSI 3008 HBA board is flashed into IT Mode. The BIOS is set to LEGACY boot only.

When following the PVE installation and setting up the target disks to be ZFS Raid Z2 and selecting our 4 disks and rebooting we get to the grub rescue prompt in which an error appears in which it is unable to find a device.

Since the install fails to boot, we got into the debug shell through the installer and we are able mount the zpool and look at the partitions the installer created. Looking at the partitions we see the /dev/sda1 and /dev/sdb1... partitions it created as type EF02 or the bios boot partitions as expected. But it appears those are unable to actually boot the OS. Seems to me that the BIOS boot manager is not able to boot the ZFS partitions since the /boot appears to be inside of ZFS much like how ProxMox 3.4 has the /boot mount inside of the LVM setup on EXT4.

I attached screenshots that show the grub rescue screen, the zpool setup and the block device ID's and partition setup and occurred during a regular install.

Any help on this would be appreciated.
 

Attachments

  • MVIMG_20180130_115331.jpg
    MVIMG_20180130_115331.jpg
    671.6 KB · Views: 141
  • MVIMG_20180130_114513.jpg
    MVIMG_20180130_114513.jpg
    808.2 KB · Views: 133
  • MVIMG_20180130_111803.jpg
    MVIMG_20180130_111803.jpg
    878.3 KB · Views: 127
  • IMG_20180130_112334.jpg
    IMG_20180130_112334.jpg
    906.1 KB · Views: 120
  • MVIMG_20180130_114541.jpg
    MVIMG_20180130_114541.jpg
    688.6 KB · Views: 115
Last edited:
  • Like
Reactions: Andymeows
Does the emergency boot from the install cd bring it up normaly?

Does pointing the legacy boot to one of the other three disks make a difference? (grub-install on all 4 again for fun too)

Does turning on uEFI along with Legacy boot help your MB bios find the Legacy boot partition?
 
your SCSI controller/BIOS/... only presents the first disk as bootable (as can be seen on the screenshot where you did 'ls' in the grub rescue shell). with just that one disk, grub is unable to read the data. you might be able to fix this by playing around with BIOS/controller settings, otherwise you need to work around it by using a different disk as boot disk and put /boot there (losing redundancy for booting).
 
  • Like
Reactions: G0ldmember
your SCSI controller/BIOS/... only presents the first disk as bootable (as can be seen on the screenshot where you did 'ls' in the grub rescue shell). with just that one disk, grub is unable to read the data. you might be able to fix this by playing around with BIOS/controller settings, otherwise you need to work around it by using a different disk as boot disk and put /boot there (losing redundancy for booting).

Hey Fabian,

You were right! I had to go into the LSI HBA board configuration and update the number of int 13 devices. It was defaulted to 1, once I updated it to 4 to match the 4 disks in the server, ProxMox installed and booted without issues.

Thanks, for all the help!
 
  • Like
Reactions: G0ldmember
Wow fantastic. This saved my day! :)
We also had a ZFS pool spanned over 2 disks and did a kernel update after which the PVE system refused to boot.
Increasing the maximum number of int 13 devices worked like a charm!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!