[SOLVED] Installing Proxmox on ZFS Raid 10

Onigami

Member
Jan 14, 2017
6
1
8
31
Geneva, Switzerland
onigami.ch
Hey guys and girls it's been a few day I try to install Proxmox on a ZFS Raid 10 and i can't get it to boot. I read on the forums that Debian and Proxmox can't boot a ZFS root under UEFI so I reverted my server to BIOS but it's the same big mess.

My setup is a Dell R710 with a LSI HBA 9211 in IT mode ( bios disabled ) with 4x 256GB SSD and 2x1tb HDD. During the Proxmox installation I set the ZFS RAID 10 to use the 4 SSD ( /dev/sdc , /dev/sdd , /dev/sde , /dev/sdf ) installation went fine but after reboot it can't find any bootable devices.

The only thing I can get to change is if I set the first SSD as a bootable device in the HBA configuration it will load grub but with an error and put me in rescue mode because of normal.mod not found.
 

Onigami

Member
Jan 14, 2017
6
1
8
31
Geneva, Switzerland
onigami.ch
Hi,
you can use the install-usb as rescue-boot. If this work, you can install grub again (update-grub2) with this ssd as bios-boot-disk?

Udo

See this: https://forum.proxmox.com/threads/grub-rescue.33696/

I had lots of trouble booting ZFS lately and rootdelay=20 (even 5) was long enough to fix most of my troubles. Hope this helps.

The main issue is that I can't get grub to load, it happened one time but I don't know why

But i'm able to fully boot and use Proxmox using "rescue boot" with the install media but only on UEFI, on BIOS it doesn't work at all
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,870
1,507
164
what does "set" and "ls" output in the rescue shell?
 

gsupp

Active Member
Jun 27, 2017
38
17
28
TX, USA
I'm running into the same issue on a Dell R710 with a PERC 6/iR flashed to IT mode. Installed Proxmox on 6x 1TB drives in ZFS RAID10 configuration. After the install finishes and the server reboots, I get "no boot device found". The BIOS is set to BIOS(legacy) boot mode (not EFI) and the HBA is set as a boot device. If I boot off the Proxmox install CD and select "rescue mode", the server boots fine. I've tried running 'update-grub2' after booting rescue mode from the CD, as well as 'install-grub /dev/sda' and 'install-grub /dev/sdb' with no effect.

@Onigami I see this thread is marked as [SOLVED] but I don't see any resolution posted?
 

gsupp

Active Member
Jun 27, 2017
38
17
28
TX, USA
@gsupp yeah finally I just bought a 32gb usb key and installed proxmox on it with UEFI + ZFS
@Onigami Thank for you the super quick response!

I'm really confused as I have a nearly identical R710 with the same HBA that I installed Proxmox on in the same RAID10 configuration that works. The only difference I can tell is the working server has (8) 300GB drives and the non-working server has (6) 1TB drives.

what does "set" and "ls" output in the rescue shell?
@fabian, here's the output from the "set" command after I boot in rescue mode off the install CD and login to the console as root:
https://pastebin.com/raw/9sQ43PZU
The "ls" command didn't show anything, as it was just listing files in the /root directory (which there are none).
 

rpotter28

New Member
Feb 29, 2012
23
2
3
@Onigami Thank for you the super quick response!

I'm really confused as I have a nearly identical R710 with the same HBA that I installed Proxmox on in the same RAID10 configuration that works. The only difference I can tell is the working server has (8) 300GB drives and the non-working server has (6) 1TB drives.

Do you really have the same HBA? I would check how you flashed the Perc 6/IR and to what.
 

gsupp

Active Member
Jun 27, 2017
38
17
28
TX, USA
Do you really have the same HBA? I would check how you flashed the Perc 6/IR and to what.
Well, what I meant was I have two Dell R710's and both of them have the same HBA that was flashed with the same firmware where one boots and the other does not. I don't necessarily have the same HBA as OP. The PERC 6/iR's were flashed to LSI SAS3081e per https://www.mrbill.net/perc/

My setup is a Dell R710 with a LSI HBA 9211 in IT mode ( bios disabled )
@Onigami I know you've already found a work around, but if you are referring to the LSI 9211 having its BIOS disabled (i.e. MPTSAS2.ROM was not flashed), it is my understanding that you cannot boot off any drives attached to the card.
 

rpotter28

New Member
Feb 29, 2012
23
2
3
Well, what I meant was I have two Dell R710's and both of them have the same HBA that was flashed with the same firmware where one boots and the other does not. I don't necessarily have the same HBA as OP. The PERC 6/iR's were flashed to LSI SAS3081e per https://www.mrbill.net/perc/

OK, was just an idea. I have lots of experience with the R710's, between working and I also own 3 at home. But, I only have used the H700 or H200 HBA's.
 

gsupp

Active Member
Jun 27, 2017
38
17
28
TX, USA
@gsupp yeah finally I just bought a 32gb usb key and installed proxmox on it with UEFI + ZFS

So after fighting with this for a couple of days, I ended up doing something similar. I plugged a USB flash drive into the R710's internal USB port, ran 'install-grub /dev/sdX' on the flash drive, ran 'update-grub' (not sure if this was necessary) and then set the USB flash drive as the first boot device in the BIOS. Works like a charm.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!