ZFS root boot. A tale of 2 Dell R810's.. One works, one doesn't..

[Paul Murdock]

G'day,

I have 2 Dell R810's in nearly the exact same configuration. Only difference is amount of RAM and # of CPU's.

The details:

1) Each has 6 HDD's attached to an LSI 9211-8i HBA card (in IT-mode)
2) I install from USB (Proxmox 5.3.2 and 5.4.1), both install no problem. recognizes the HDD's and create a RAID-Z2 on each R810
3) DELL BIOS mode (NOT UEFI)



Dell #1 - (74GB RAM, 2 CPU's) - after install it boots up just great. It goes straight to the prompt.

Dell #2 (256GB RAM, 4 CPU's) - after install, right as it is supposed to boot from HDD's it just sits there, then about 30 minutes later, it finally boots up. I have tried 2 different LSI 9211-8i's in case the card was bad. It wasn't. After it boots up 30 minutes later, the system acts like normal. zpool status is great and ONLINE. system seems to work great.


Why oh why does it take 30 minutes to boot from the HDD's??

I've thought that maybe the GPT/MBR was messed up on the cards, but I've wiped them, and when I install Proxmox it seems to partition up the drives just great (3 partitions on each disk)

partition 1 - boot partition (I assume GRUB goes here, something like 1MB)
partition 2 - 512MB (I assume this is for some other use?)
parition 3 - 512GB (the majority of the hard drive space)

Anyone have any thoughts on why Dell #2 just takes forever to boot? It's the same setup as the other Dell in almost every respect.

Another point of interest, if I just install Proxmox on /dev/sda using ext4 and reboot, on Dell #2 it just comes right up.. so it has something to do with ZFS perhaps (but only on Dell #2)??


cheers,
Paul

 

[Paul Murdock]

I'm running a Memtest86+ on the 256GB of RAM in case the RAM is the issue.. I did see this particular error on screen a couple of times

[501.882940] print_req_error: I/O error, dev [my hdd's], sector 0

I've changed out the LSI HBA card, so maybe my hard drives are bad??

What gets me, is I had Proxmox installed with my PERC/H700 card and THESE hard drives running RAID-6 just fine for about a week..

This whole issue seems to have happened around taking out the PERC/H700 raid card, putting in the LSI 9211-8i card, and trying ZFS..

I have tried 2 different LSI 9211-8i cards though.. mind boggling.. Those 2 cards were from the same ebay vendor however.. I wonder....

I can pull the LSI 9211-8i from my known working Dell perhaps..

 

[Paul Murdock]

One more data point to try and understand.

On Dell #2, I can install Proxmox 5.4.1 using ZFS RAID-1 on /dev/sda and /dev/sdb and it boots up just fine every time..

It appears to be something with RAID-Z2 across all 6 disks that really messes it up.

I'm beginning to think it has something to do with Proxmox not installing GRUB on all HDD's? How can I check if that is the case?

 

[Paul Murdock]

Found a solution!

see this thread https://forum.proxmox.com/threads/30-min-boot-time.46873/

It turns out a factory reset of iDrac on the Dell was all that was needed. No HBA card problems, no GRUB problems, no disk problems.. just good 'old Dell iDRAC.