Proxmox 3.4 won't boot on ZFS and 440AR in HBA mode (HP GL380 Gen9 server)

J

jinjer

Renowned Member
Oct 4, 2010
204
7
83
Hi,

I've been trying to get proxmox 3.4 to boot from a native ZFS root, to no avail. I will boot from ext3 partition, but not ZFS. I don't see the grub prompt at all. Just a cursor and the server resets itself and starts the boot procedure over and over.

This is an HP380 Gen9 using an HP 440ar raid controller in HBA mode. In HBA the disks are exposed as SATA Ahci ports. There is no controller bios shown during boot.

I tried both UEFI and Standard BIOS. The result is the same. I can, however, boot in UEFI mode when using ext4 as the boot disk. This seems to be zfs related.

Any hints will be appreciated.

EDIT: It seems that the install is missing the EFI partition. This is really strange:

Code: 
# gdisk -l /dev/sda
GPT fdisk (gdisk) version 0.8.5


Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present


Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 1953525168 sectors, 931.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): FDBF364B-01C3-4962-B202-AE88DEA2DDB2
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1953525134
Partitions will be aligned on 8-sector boundaries
Total free space is 0 sectors (0 bytes)


Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02
   2            2048      1953508749   931.5 GiB   BF01  zfs
   9      1953508750      1953525134   8.0 MiB     BF07
 
Thank you for replying. The point is, the server will not boot from ZFS root. It will do fine with Ext4. I am not sure if efi is needed or not, but if I boot to debug and try to update-grub and grub-install on the ZFS disks, I get an error that EFI partition cannot be found.

I did try the 4.0 beta 1 and the results are the same. No boot on ZFS.
 
jinjer said:
Thank you for replying. The point is, the server will not boot from ZFS root. It will do fine with Ext4. I am not sure if efi is needed or not, but if I boot to debug and try to update-grub and grub-install on the ZFS disks, I get an error that EFI partition cannot be found.

I did try the 4.0 beta 1 and the results are the same. No boot on ZFS.
Click to expand...

One more hint: Perhaps, the lack of EFI partition on a disk will prevent the server bios from actually trying to boot in UEFI ?
 
jinjer said:
One more hint: Perhaps, the lack of EFI partition on a disk will prevent the server bios from actually trying to boot in UEFI ?
Click to expand...
This is confirmed. With UEFI Bios, the efi partition is required anyway. I did a manual repartitioning and created the EFI partition. Then I installed grub and voila: The server booted properly.

I am not sure if this was/is a requirement on this server or generic EFI requirement, but I would bet the later is true.
 
A few days passed and no reaction from the developers. I think that this is a possible bug to be addressed in a future proxmox update. Shall I open a bug report or is it enough that I raised the issue here?
 
jinjer said:
A few days passed and no reaction from the developers. I think that this is a possible bug to be addressed in a future proxmox update. Shall I open a bug report or is it enough that I raised the issue here?
Click to expand...

pls open a bug on https://bugzilla.proxmox.com
 
"Me too."
I am trying to avoid the other dominant virtualization solutions and stick with Proxmox VE 3.4.

Trying to install it on a Lenovo x3500 M5 (basically a x3550 in a tower case instead of rack-mounted) with the built-in ServerRAID M5210 and ZFS.
In fact, the installer runs in UEFI mode (that is, not-Legacy mode) just fine, it finds my disks (each disk is a one-disk RAID0 - lacking an HBA mode or disk pass-through) and sets up a ZFS raid1 installation as expected.
Yet, after the reboot, it hangs around for some seconds and the (UEFI) BIOS outputs "Boot failed" with the boot entry discription next to it.
Legacy mode works and boots the installation. I would really prefer to go UEFI - though there does not seem to be an obvious downside to legacy mode.

After reading this post (and learning an awful lot about EFI, ZFS on Linux), I can confirm that the lack of the EFI System Partition (ESP) is the root cause here.
The ESP is a 100-260MB FAT32-partition with gdisk-type EF00 within the first 2TB of the boot device(s). GPT instead of MBR is advised.
It contains the /boot/efi-directory with a .efi-file that is loaded by the UEFI firmware (BIOS).
AFAICT, the boot order is UEFI board firmware -> ESP-efi-file -> BIOS boot partition -> GRUB -> kernel/ramdisk from ZFS-RAID1.

These are the steps I did to confirm that:

  • Boot the Proxmox VE 3.4 CD/ISO, choosing "Debug mode", then exiting the console with Ctrl+D, clicking "Abort" in the installer. This drops you to a shell again. Info from /threads/21306-Grub2-recovery-on-ZFS-Proxmox-VE-3-4. (Forum doesn't let me post links yet)
  • Use cgdisk /dev/sda and delete the ZFS partition (partition number 2). That does not destroy data as ZFS is in pseudo-RAID1-mode. Pool is now degraded.
  • Create a new (second) partition of size 260 MB, type EF00, label it ESP.
  • Create a new (third) partition in the remaining space, type BF01, label it ZFS
  • Write that partition layout and exit cgdisk.
  • Verify that you see sda1, sda2 and sda3. In my case I saw a strange (=no idea what it is) Part# 9 "Solaris Reserved 1" that I did not touch.
  • The next steps create a FAT32(!) FS on sda2, replace the missing ZFS disk/partition and reinstall grub and add a new UEFI Boot entry. Not sure if the last step was needed.
    Additional information came from here: http: superuser.com/a/376471/59518

Code: 
$ modprobe efivars

$ zpool import rpool -R /mnt

$ zpool replace rpool /dev/sda2 /dev/sda3
# watch resilvering finish before you continue!

$ mkfs.vfat -F32 /dev/sda2
$ mount /dev/sda2 /mnt/boot/efi
        
$ mount -t proc /proc /mnt/proc
$ mount --rbind /dev /mnt/dev
$ mount --rbind /sys /mnt/sys

$ chroot /mnt /bin/bash
(chroot)$ source /etc/profile
(chroot)$ grub-install /dev/sda
(chroot)$ update-grub2
(chroot)$ update-initramfs -u
(chroot)$ efibootmgr -c --disk /dev/sda --part 2
# verify a new record called Linux is there
(chroot)$ efibootmgr -v 
Ctrl-D
 
$ umount /mnt/sys
$ umount /mnt/dev
$ umount /mnt/proc
$ umount /mnt/boot/efi
$ zfs unmount rpool/ROOT/pve-1
Ctrl-D
Ctrl-D


After a reboot, Proxmox VE starts as usual in UEFI boot mode.
The second disk should get a copy of that boot partition to allow booting after failure of the first one.

Obviously it is not a good idea to let new user (like me) hang in the air after a clean install, so:
Could you please fix the installer?

-- Philipp
 
Hello,

Sorry for bringing back the thread. I am having the same issue, HBA with P440ar on DL380 Gen9. Proxmox can see the drives and installation completed, but cannot boot. Any solution?

THanks
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back