[SOLVED] Installing Proxmox on ZFS Raid 10

[Onigami]

Hey guys and girls it's been a few day I try to install Proxmox on a ZFS Raid 10 and i can't get it to boot. I read on the forums that Debian and Proxmox can't boot a ZFS root under UEFI so I reverted my server to BIOS but it's the same big mess.

My setup is a Dell R710 with a LSI HBA 9211 in IT mode ( bios disabled ) with 4x 256GB SSD and 2x1tb HDD. During the Proxmox installation I set the ZFS RAID 10 to use the 4 SSD ( /dev/sdc , /dev/sdd , /dev/sde , /dev/sdf ) installation went fine but after reboot it can't find any bootable devices.

The only thing I can get to change is if I set the first SSD as a bootable device in the HBA configuration it will load grub but with an error and put me in rescue mode because of normal.mod not found.

 

[udo]

Hi,
you can use the install-usb as rescue-boot. If this work, you can install grub again (update-grub2) with this ssd as bios-boot-disk?

Udo

 

[mram]

See this: https://forum.proxmox.com/threads/grub-rescue.33696/

I had lots of trouble booting ZFS lately and rootdelay=20 (even 5) was long enough to fix most of my troubles. Hope this helps.

 

[Onigami]

Hi,
you can use the install-usb as rescue-boot. If this work, you can install grub again (update-grub2) with this ssd as bios-boot-disk?

Udo

See this: https://forum.proxmox.com/threads/grub-rescue.33696/

I had lots of trouble booting ZFS lately and rootdelay=20 (even 5) was long enough to fix most of my troubles. Hope this helps.

The main issue is that I can't get grub to load, it happened one time but I don't know why

But i'm able to fully boot and use Proxmox using "rescue boot" with the install media but only on UEFI, on BIOS it doesn't work at all

 

[mram]

If you have an ODD to 2.5" SATA HDD adapter swap your DVD drive and try booting from the onboard SATA port instead. I have an R710 but I used M1015 and H310 cards in IT mode without problems with ZFS and the PERC6 without ZFS.

https://www.newegg.com/Product/Product.aspx?Item=N82E16817997048

 

[fabian]

what does "set" and "ls" output in the rescue shell?

 

[gsupp]

I'm running into the same issue on a Dell R710 with a PERC 6/iR flashed to IT mode. Installed Proxmox on 6x 1TB drives in ZFS RAID10 configuration. After the install finishes and the server reboots, I get "no boot device found". The BIOS is set to BIOS(legacy) boot mode (not EFI) and the HBA is set as a boot device. If I boot off the Proxmox install CD and select "rescue mode", the server boots fine. I've tried running 'update-grub2' after booting rescue mode from the CD, as well as 'install-grub /dev/sda' and 'install-grub /dev/sdb' with no effect.

@Onigami I see this thread is marked as [SOLVED] but I don't see any resolution posted?

 

[Onigami]

@gsupp yeah finally I just bought a 32gb usb key and installed proxmox on it with UEFI + ZFS

Works well

 

[gsupp]

@gsupp yeah finally I just bought a 32gb usb key and installed proxmox on it with UEFI + ZFS

@Onigami Thank for you the super quick response!

I'm really confused as I have a nearly identical R710 with the same HBA that I installed Proxmox on in the same RAID10 configuration that works. The only difference I can tell is the working server has (8) 300GB drives and the non-working server has (6) 1TB drives.

what does "set" and "ls" output in the rescue shell?

@fabian, here's the output from the "set" command after I boot in rescue mode off the install CD and login to the console as root:
https://pastebin.com/raw/9sQ43PZU
The "ls" command didn't show anything, as it was just listing files in the /root directory (which there are none).

 

[rpotter28]

@Onigami Thank for you the super quick response!

I'm really confused as I have a nearly identical R710 with the same HBA that I installed Proxmox on in the same RAID10 configuration that works. The only difference I can tell is the working server has (8) 300GB drives and the non-working server has (6) 1TB drives.

Do you really have the same HBA? I would check how you flashed the Perc 6/IR and to what.

 

[gsupp]

Do you really have the same HBA? I would check how you flashed the Perc 6/IR and to what.

Well, what I meant was I have two Dell R710's and both of them have the same HBA that was flashed with the same firmware where one boots and the other does not. I don't necessarily have the same HBA as OP. The PERC 6/iR's were flashed to LSI SAS3081e per https://www.mrbill.net/perc/

My setup is a Dell R710 with a LSI HBA 9211 in IT mode ( bios disabled )

@Onigami I know you've already found a work around, but if you are referring to the LSI 9211 having its BIOS disabled (i.e. MPTSAS2.ROM was not flashed), it is my understanding that you cannot boot off any drives attached to the card.

 

[rpotter28]

Well, what I meant was I have two Dell R710's and both of them have the same HBA that was flashed with the same firmware where one boots and the other does not. I don't necessarily have the same HBA as OP. The PERC 6/iR's were flashed to LSI SAS3081e per https://www.mrbill.net/perc/

OK, was just an idea. I have lots of experience with the R710's, between working and I also own 3 at home. But, I only have used the H700 or H200 HBA's.

 

[gsupp]

@gsupp yeah finally I just bought a 32gb usb key and installed proxmox on it with UEFI + ZFS

So after fighting with this for a couple of days, I ended up doing something similar. I plugged a USB flash drive into the R710's internal USB port, ran 'install-grub /dev/sdX' on the flash drive, ran 'update-grub' (not sure if this was necessary) and then set the USB flash drive as the first boot device in the BIOS. Works like a charm.