Proxmox VE 6.3 zfs root install issues

[jeremiahg7]

Hello,

I installed proxmox on a DL360p with ZFS RAID 10 4 x Kingston DC450R 960GB SSD's running on a h240 in HBA mode. After installation, on first boot, I'm getting the following error:

Attempting Boot From Hard Drive (C

error: unknown device 1.

I saw a similar post with a similar issue and ran the same commands...





Not sure what else to do, any thoughts?

 

[Dunuin]

If a drive got sdX1+2+3 it was created as system/boot drive by proxmox. If it got sdX1+9 is was created as a zfs drive without system/boot.
Not sure if proxmox is doing that this way with raid10 as system drives but it looks a little bit weird. Or did you installed proxmox to just a normal mirror and added a second mirror afterwards?

 

[jeremiahg7]

If a drive got sdX1+2+3 it was created as system/boot drive by proxmox. If it got sdX1+9 is was created as a zfs drive without system/boot.
Not sure if proxmox is doing that this way with raid10 as system drives but it looks a little bit weird. Or did you installed proxmox to just a normal mirror and added a second mirror afterwards?

During installl I selected Raid 10 and the four drives. The installer did all of the work / changes.

 

[jeremiahg7]

Bump, any suggestions on where to go from here?

 

[jeremiahg7]

I tried installing as Raid 0 this time, with all four drives, and I got a new error:



lsblk:



zpool import:

 

[jeremiahg7]

For some reason, when I only stripe with raid 1 it works perfectly...





So the issue is only when it comes to mirroring?

 

[Dunuin]

Maybe a BIOS/UEFI thing? What drives are you trying to boot from? With raid1 all 4 drives allow to boot from (sdX2 available). With raid10 only sda+sdb and as raid0 only sda.

 

[jeremiahg7]

Maybe a BIOS/UEFI thing? What drives are you trying to boot from? With raid1 all 4 drives allow to boot from (sdX2 available). With raid10 only sda+sdb and as raid0 only sda.

I just checked and DL360p Gen 8 doesn't have UEFI.

From what I understand, you boot from the pool? I just select all 4 drives regardless of the configuration (RAID 10, 0, 1) and let the installer handle the rest. is there a configuration I'm missing here?

 

[mmenaz]

Hello,

I installed proxmox on a DL360p with ZFS RAID 10 4 x Kingston DC450R 960GB SSD's running on a h240 in HBA mode. After installation, on first boot, I'm getting the following error:

Hope you are aware that the "R" models are for "read oriented" workload, so you should always choose the "M" variant, that is much more durable and has better performances (at slightly higher costs).

 

[Dunuin]

ZFS is not like a hardware raid. There is no "pool" at boot level. You got 4 individual drives and raid is handled after booting by software.
So you need to boot from the right disk that has a bootloader installed. If using raid10 and only sda+sdb got a boot partition (sdX2) and sdc+sdd not, then you need to setup your BIOS to boot from sda and/or sdb. If your server tries to boot from sdc or sdd (without a boot partition) it won't work.

 

[jeremiahg7]

Hope you are aware that the "R" models are for "read oriented" workload, so you should always choose the "M" variant, that is much more durable and has better performances (at slightly higher costs).

Yes sir, I'm aware. They're web servers, so most of the work is read only.

 

[jeremiahg7]

ZFS is not like a hardware raid. There is no "pool" at boot level. You got 4 individual drives and raid is handled after booting by software.
So you need to boot from the right disk that has a bootloader installed. If using raid10 and only sda+sdb got a boot partition (sdX2) and sdc+sdd not, then you need to setup your BIOS to boot from sda and/or sdb. If your server tries to boot from sdc or sdd (without a boot partition) it won't work.

Wouldn't the installer handle that portion though? I was under the impression you could setup a ZFS pool for the boot drive. Would I need to install on a 5th drive and use the RAID 10 pool for my applications and data then?

 

[Dunuin]

Wouldn't the installer handle that portion though?

Proxmox can't do BIOS settings for you.

I was under the impression you could setup a ZFS pool for the boot drive.

Yes, but you need to tell the BIOS which SSD you want to boot from. Not every SSD of that pool is bootable if it is not raid1.

Would I need to install on a 5th drive and use the RAID 10 pool for my applications and data then?

No, but you still need to setup the right drive to boot from in BIOS.

 

[jeremiahg7]

Proxmox can't do BIOS settings for you.

Yes, but you need to tell the BIOS which SSD you want to boot from. Not every SSD of that pool is bootable if it is not raid1.

No, but you still need to setup the right drive to boot from in BIOS.

LOL - good point, I didn't think about that all. I'll give it a look and report back. Thanks!

 

[jeremiahg7]

Proxmox can't do BIOS settings for you.

Yes, but you need to tell the BIOS which SSD you want to boot from. Not every SSD of that pool is bootable if it is not raid1.

No, but you still need to setup the right drive to boot from in BIOS.

For the life of me, I can't figure out how to set the boot order. The H240 raid controller is in HBA mode so it's not seen in the ACU. The bios only lists the controller itself, not individual drives. I was thinking about just installing it and changing the drive order until it works, is that possible? IE 0-3 was set to a-d... now I'm trying in reverse 0 = d, 1 = c, 2=b, 3=a

 

[bobmc]

I don't think your going to be able to do this - the problem is that the controller is not presenting the pool to the boot loader in the BIOS. Where LSI controllers and their clones are concerned - when you flash them to IT-mode (ie hba mode) you can choose between making them bootable (where bios code is loaded from the controller during post to present a bootable config to the machine and increases the time taken to boot the server) or faster boot time with the controller just simply presenting raw disks. In fact, on the LSI cards, I don't believe you can even boot from single drive unless the card has the bootable bios code enabled.

 

[jeremiahg7]

I don't think your going to be able to do this - the problem is that the controller is not presenting the pool to the boot loader in the BIOS. Where LSI controllers and their clones are concerned - when you flash them to IT-mode (ie hba mode) you can choose between making them bootable (where bios code is loaded from the controller during post to present a bootable config to the machine and increases the time taken to boot the server) or faster boot time with the controller just simply presenting raw disks. In fact, on the LSI cards, I don't believe you can even boot from single drive unless the card has the bootable bios code enabled.

Should I take the 240 out of HBA and setup RAID 10 via the controller? I understand I'll lose shadow copy, but is there anything else I need to worry about? I've seen concerns around performance issues.

 

[Dunuin]

If you use HW raid you can't use ZFS. So no bit rot protection, blocklevel compression, deduplication and all the other nice ZFS features.