[SOLVED] ZFS RAID1 fails to boot after power failure

[helojunkie]

So the power company lost a transformer last night and our proxmox server went down after its large UPS died.

This morning when trying to bring it back online it fails to boot.

Controller sees all drives including the two SSD RAID1 drives that is the boot device.

However when booting it gets past the post and just shows a blinking dash in the upper corner of the screen. No boot, nothing.

I am looking for ideas on how to get it to boot.

Thanks

 

[helojunkie]

Well as luck would have it, a while back I had to replace one of my RAID1 drives and they are hot-swappable. When I replaced the drive, it resilvered just fine and I went about my business.

Fast forward more than a year later (without ever having to reboot the server) and the power goes out and the server won't boot. Plugged in a Prox iso, selected rescue and it booted up just fine.

Turns out that when I replaced the drive I neglected to install grub on it so there was no bootloader. One simple command later and I was back up and running:

grub-install --target=i386-pc /dev/sda

 

[sb-jw]

our proxmox server went down after its large UPS died.

Use APC PowerChute (or alternatives), if your UPS doesn't support it, replace it with a UPS which is able to handle this. The UPS itself can send out an E-Mail when the Power is lost, so you are informed about such Problem.

Turns out that when I replaced the drive I neglected to install grub on it so there was no bootloader.

Keep that in mind: https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Replacing_a_failed_disk_in_the_root_pool
Everytime i have to replace an Disk in ZFS i use this Guide and check it step by step. Its the only way to do not miss a thing

 

[helojunkie]

Use APC PowerChute (or alternatives)

We do, server going down wasn't the problem, it coming back up after I forgot to put grub back on it was . I had just figured that it copied 'everything' not just the ZFS. Lesson learned along with the fact that I should have rebooted anyway just to check! So two lessons learned.

Thanks for the link, I will file that away for the next time!