Mirrored ZFS Array will not boot on Supermicro A2SDi motherboard

[Brent Clothier]

I am attempting to install Proxmox (Ver. 5.1-3) on a system with two SSD units in a mirrored ZFS array. However, the system does not boot into Proxmox, and instead, dumps me into an EFI shell. Any assistance is appreciated.

My motherboard is a Supermicro A2SDi-8C+-HLN4F (BIOS Rev. 1.0b). Note that this motherboard uses Intel’s C3758 8-Core Atom processor, which is based Intel’s C3000 series “Denverton” system-on-a-chip (SoC). Here is what is installed on the motherboard:

[1] Four Crucial 4GB registered ECC DIMMs (CT4G4RFS824A) for 16GB total
[2] Two enterprise-grade Micron 5100 MAX SSD units, each with 240GB capacity (MTFDDAK240TCC-1AR1ZAB)

Note that one SSD unit is connected to each of the I-SATA 0 and I-SATA 1 ports of the motherboard. (Note that I-SATA 0 supports the SATA DOM feature, which Supermicro calls “SuperDOM”.)

All components are brand new and flashed to the latest firmware. The motherboard BIOS is set to factory defaults. However, I have used the IPMI console to switch the IMPI LAN port from “Failover” to “Dedicate”. This setting should have no relevance to the problem I am having.

I can install Proxmox to each drive individually with no issues. (The unused drive is disconnected from electrical power and its respective I-SATA port.) The default settings of the Proxmox installer work fine and the system boots into Proxmox.

When attempting to install Proxmox onto a mirrored ZFS array, i.e., ZFS in RAID1, things change. The installer completes the installation process. However, upon reboot, the system displays an EFI console with details of the device mapping table. I provide a snapshot below.

At factory default, the BIOS of the motherboard is set to boot UEFI. (Secure Boot is disabled.) If I switch this setting to Legacy, the outcome remains the same. If I first switch the default to Legacy, and then install Proxmox, the outcome is also the same. I also tried switching the SSD units from I-SATA 0 and I-SATA 1 to I-SATA 3 and I-SATA 4. But no luck there either.

I suspect the problem has something to do with GRUB, which fails to see both SSD units. So the motherboard defaults to booting into the EFI console.

 

[Alwin]

When you installed PVE onto a single drive, did you use ZFS or EXT4? ZFS is not able to handle EFI yet, you need to switch to legacy boot in bios. Also check, if the SATA ports have a EFI/legacy option to set.

 

[Brent Clothier]

When you installed PVE onto a single drive, did you use ZFS or EXT4? ZFS is not able to handle EFI yet, you need to switch to legacy boot in bios. Also check, if the SATA ports have a EFI/legacy option to set.

Thank you for responding, Alwin.

When I installed PVE on a single drive, I used EXT4. I also tried XFS, which works as well. The Proxmox installer does not allow one to install ZFS onto a single drive. So no, I did not try ZFS with a single drive.

Regarding ZFS RAID1, I did have the BIOS set to Legacy mode. Yet the installation would not boot. On this motherboard, Supermicro does not provide EFI and/or Legacy settings for individual SATA ports.

If you have any further suggestions, please let me know. I have also opened a ticket with Supermicro.

 

[Alwin]

When I installed PVE on a single drive, I used EXT4. I also tried XFS, which works as well. The Proxmox installer does not allow one to install ZFS onto a single drive. So no, I did not try ZFS with a single drive.

ZFS raid 0, with one disk, then it is possible to install PVE on ZFS. Afterwards you can create a mirror out of it too.

Boot order or SATA ports may make a difference. Also the disks may need longer to initialize and grub doesn't find them.
https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Troubleshooting_and_known_issues

 

[Brent Clothier]

>ZFS raid 0, with one disk, then it is possible to install PVE on ZFS. Afterwards you can create a mirror out of it too.

I just attempt to install on a single disk using ZFS RAID 0. The unused SSD was disconnected from both power and data plugs. Unfortunately, my A2SDi-8C+-HLN4F motherboard still boots into an EFI console. However, the details of the device mapping table are half that for ZFS RAID 1. (Of course, I am using only one SSDs.) I provide a snapshot of the EFI console below.

 

[Brent Clothier]

>Boot order or SATA ports may make a difference.

I moved the SSDs from I-SATA 0 &1 to I-SATA 2 & 3. However, this change made no difference.

>Also the disks may need longer to initialize and grub doesn't find them.

I believe this issue may be at the root of my problem. To check, I installed FreeNAS (Ver. 11.1-U1) on my system using ZFS RAID 1. FreeNAS is based on FreeBSD (Ver. 11.1). The installation succeed and GRUB loaded FreeNAS just fine. Within FreeNAS, I confirmed that the SSDs were operating as a ZFS RAID 1 array.

Obviously, Proxmox is based on Debian Linux, not FreeBSD. But there must be some difference between the two GRUB loaders. One works, the other does not. I checked the wiki page you linked to. However, some of the solutions there are not relevant to my problem. Others provide no real detail on how to implement them.

 

[Alwin]

FreeBSD is able to boot from UEFI with ZFS, while not yet possible with ZoL. You really need to check your bios settings, as you shouldn't get into a EFI shell at all. It should just tell you that there was no boot device.

 

[nixcz]

@Brent Clothier and anyone else running into this issue on the A2SDi boards: I spent a great amount of time on this too, before I remembered what I had long forgotten (the most non-intuitive part of dealing with UEFI bioses is that CSM must often be explicitly enabled, and this is usually in a completely different menu). For example, some ThinkPad bioses behave this way.

I see you have bios v1.0b, which exposes the Legacy boot options. Version 1.0 (no b) only supports UEFI boot. But even with 1.0b and Legacy boot options enabled, you have to go to the Security menu -> Secure Boot -> CSM Support [Enabled]. Once you enable that, several additional menu options are enabled, including the traditional Hard Disk Drive BBS selection menu to specify SATA drive boot order. Once you see that you know you can boot from mirrored ZFS without problems.

 

[Brent Clothier]

@Brent Clothier and anyone else running into this issue on the A2SDi boards: I spent a great amount of time on this too, before I remembered what I had long forgotten (the most non-intuitive part of dealing with UEFI bioses is that CSM must often be explicitly enabled, and this is usually in a completely different menu). For example, some ThinkPad bioses behave this way.

I see you have bios v1.0b, which exposes the Legacy boot options. Version 1.0 (no b) only supports UEFI boot. But even with 1.0b and Legacy boot options enabled, you have to go to the Security menu -> Secure Boot -> CSM Support [Enabled]. Once you enable that, several additional menu options are enabled, including the traditional Hard Disk Drive BBS selection menu to specify SATA drive boot order. Once you see that you know you can boot from mirrored ZFS without problems.

Thank you for posting this information. I poked around in the Secure Boot menu, but did not think to play around with CSM support. I will have to try your suggestion in a month or two when I purchase another one of the A2SDi motherboards. I transferred my current A2SDi motherboard to a pfSense box (which of course operates on FreeBSD).