Proxmox ZFS won't boot anymore after degraded disk

[Wouter Samaey]

I have Proxmox 6.2 with 6x 6TB disks in a RAID-Z2 configuration.
Since a few days ago disk 3 failed and the rpool was degraded.

I replaced the faulty disk with another identical one, but after a few hours I got the message that this disk was also faulty.

The resilvering process did continue after this.

Since the number of errors was very small, I decided to try a "zpool clear", but this didn't seam to do much.

So, I rebooted, and now the server won't come back up and I'm stuck in GRUB rescue.

I'm quite new at this, so any help would be appreciated.

The error in GRUB is: No such device 398d577fa780fa77 (which is de faulty disk)

The situation before the reboot was like this.

root@o1:~# zpool status
pool: rpool
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Thu Sep 10 15:55:40 2020
11.9T scanned at 202M/s, 10.9T issued at 185M/s, 12.0T total
1.43T resilvered, 91.30% done, 0 days 01:38:25 to go
config:

NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
sda3 ONLINE 0 0 0
sdb3 ONLINE 0 0 0
replacing-2 UNAVAIL 0 0 0 insufficient replicas
6418447306550617325 UNAVAIL 0 0 0 was /dev/sdc3
sdc FAULTED 0 12 0 too many errors (resilvering)
sdd3 ONLINE 0 0 0
sde3 ONLINE 0 0 0
sdf3 ONLINE 0 0 0
logs
nvme-INTEL_SSDPE21D480GA_PHM28134004Q480BGN ONLINE 0 0 0
cache
nvme0n1 ONLINE 0 0 0

errors: No known data errors

So in short, I'm screwed and have no idea what to do.

I did try the Resbue boot from a Proxmox ISO USB drive, but it could not auto detect the rpool and that got me nowhere.

I'm pretty sure there can be an easy fix, but what is it?

Should I try re-attaching the initial faulty drive? The resilver did complete though...

Because it is RAIDZ2 the data should have a 2nd disk redundancy.

In GRUB rescue I have tried:

insmod zfs
ls (hd1)
ls (hd2)
ls (hd3)
...

But it keeps saying "Filesystem is unknown"

I also ran "set" and got some info you can see in the screenshot (old school photo)

 

[Wouter Samaey]

If I boot from a Ubuntu 20.04 USB drive (which has also ZFS support), I can see the disks.

So, if I could just repair or re-install grub from this Ubuntu, that would help.

What's the command for this?

 

[apoc]

Sorry I can't help you with the command line but are interested if you know how long the uptime was before you rebooted the system. The description sounds somewhat similar to what I have experienced multiple times but since I had different pools only some backup data was affected. Not the root OS.
Thanks

Maybe that helps:
https://help.ubuntu.com/community/Grub2/Installing

 

[Wouter Samaey]

After some more research, I believe I understand what I did wrong, but still unsure about how to fix.

Is it possible that after doing a “zfs replace” you’re supposed to update grub as well? I did not update grub after the new disk was inserted, so maybe that’s what’s going on?

how to best recover now?

 

[leesteken]

Can you boot with Ubuntu and use gdisk to show us all drives and their partition tables (GPT and/or MBR)? If you installed a recent Proxmox VE with ZFS, I would expect booting via systemd instead of GRUB. But there may be an old GRUB or a partial GRUB partition. Seeing the disk partitions will help determine this, and whether the are ESP partitions anywhere.

If indeed the ESP partitions are missing on the replaced drive, it may be present on one of the other drives. Can you please try booting from one of the other drivers?

 

[Wouter Samaey]

@avw It appears that I have grub indeed.
I managed to mount ZFS on Ubuntu Live USB and can run a few commands there.

A few things I am noticing:

1) is that the partition table on all drives is identical, except for the replacement drive. It lacked a partition at the beginning, but otherwise it's the same. I created the partition by hand, set the type correctly (Bios something) en used "dd" to clone the contents from a health disk there. Since the partition was created later, the number is now 2 instead of 1 on that disk, but the sectors are at the beginning of the disk.

This may be unrelated.

When I look at the disks with GParted, the partitions look different. GParted only shows 1 partition for ZFS and not 3 like fdisk does.

Except now, the replaced disk does look different in GParted.

2) The main issue now seams to be that I should re-install grub, but when I run grub-probe, it says "Unknown filsystem". I am running grub-probe / after mounting and cdrooting to the proxmox installation, so this should work. My guess is something is wrong.

Should I try to remove the replaced disk?
At present it appears the resilvering has started again. Maybe the fact that it isn't complete is an issue by itself?

3) I can access the /boot/grub/grub.cfg file and it mentions the missing disk 398d577fa780fa77. Should I edit this file and put in another disk ID? I have no clue if this is a good or bad idea.

I'm also looking at faster solutions like creating a bootable USB drive from the /boot partition, but this doesn't help since I will again need to install grub, which I can't due to "Unknown filsystem".

 

[Wouter Samaey]

OKay, I was able to fix the grub-probe issue by detaching the faulted drive and thus also stopping the resilvering.
Grub-install is noz working and I'm running it on all disks 1 by 1. Hope I can boot soon now!!

 

[Wouter Samaey]

Grub is working, but still not booting...
Now this baby has shown up.

I'm totally surprised the log device is now an issue. It's never even come up before.

 

[leesteken]

@avw It appears that I have grub indeed.
I managed to mount ZFS on Ubuntu Live USB and can run a few commands there.

A few things I am noticing:

1) is that the partition table on all drives is identical, except for the replacement drive. It lacked a partition at the beginning, but otherwise it's the same. I created the partition by hand, set the type correctly (Bios something) en used "dd" to clone the contents from a health disk there. Since the partition was created later, the number is now 2 instead of 1 on that disk, but the sectors are at the beginning of the disk.

This may be unrelated.

When I look at the disks with GParted, the partitions look different. GParted only shows 1 partition for ZFS and not 3 like fdisk does.

Could this be caused by using fdisk instead of gdisk? I'm not sure whether fdisk sees the GPT and only changed the MBR. As the GPT takes priority, it may explain the difference in GParted. Please check with gdisk whether the GPT and MBR still match.
Then again, this might no longer be the main problem and therefore it may not solve anything...

 

[tatric]

I have just recovered a similar issue twice now and the fix my be the similar for you. Both time i was left a grub rescue.

These are the notes i made after the 1st time:

Proxmox server failed update then rebooted caused it to not boot correctly finishing at the boot rescue
prompt! AAAAAH
Grub rescue is rubbish, list or ls. I could only get one command to work, and googling for results to fix
this didn’t work because I had a new ZFS file system root dir.
Proxmox is installed on the ZFS file system using a pool for the root called rpool.
I tried to reboot in the attempt to use some other kernel but never even got those options, I even tried to
boot from other disks in the BIOS. But only one disk showed to boot from the very basic HP G7
Next I thought about a repair install from a boot usb key. This worked and gave me a root promt but no
zpool list! Using the common zfs list and zpool list showed nothing ! But the four disks were there. Using
lsblk
So I needed to import the disks to get me a file system. Trouble was zpool import rpool imported and
tried to map all the mounts to the boot disk already mount paths! So I zpool export rpool and remounted
with a new comman I found.
Zpool import -a -N -R /mnt
This imported the pool to non-default location. The -N flag (don’t automatically mount) is necessary
because otherwise the rpool root, and the rpool/root/UBUNTU pool, will both try to mount on /mnt .
There was a scary message that said that this file system came from another system and that I had to use -
f to force it to mount on this one.
mount the root system
# zfs mount rpool/ROOT/ubuntu
mount the remaining file systems
# zfs mount -a
Bind the virtual filesystems from the LiveCD environment to the new system and chroot into it:
# mount --rbind /dev /mnt/dev
# mount --rbind /proc /mnt/proc
# mount --rbind /sys /mnt/sys
# chroot /mnt /bin/bash –login
Note: This is using --rbind, not —bind.
# update-grub

I repeated this for each disk -mount the first disk
# mount /dev/disk/by-id/scsi-SATA_disk1-part3 /boot/efi
# grub-install /boot/efi

then rebooted. Got another scary message during a slightly different
boot up where again I had to force the import for a file system that
was not from this system. Then it loaded grub and then crashed!!!
bugger. I rebooted again and the grub came up and I rembered what I

might have caused the whole issue, and booted into an older kernel.
Then it completed boot up!!! normally.
#apt update showed that the a dpkg needed to be fixed. And then apt
update worked again and apt upgrade to complete the process.
Everything back to normal. Lesson if you get disconnected and don’t
check that the update worked fine like it always does. Then just run
and apt update && apt install to make sure before you restart the
server!

The second time i had replaced a disk and its was rebuilding very slowly and after 3 weeks i suffered a power cut and the server rebooted to your issue were the grub disk ****** could not be found.

this time a little different but i booted to a live usb again and imported the zfs pool to /mnt and then i noticed that it was starting to fix itself again and this time the speed was mount up each time i looked from about 2m/s to about 27m/s ! same disks same server the only difference is no OS bumbling away doing nothing. I then even got an estimation on the time to finish, about a day! this was worth it. after if finished inspecting the disk in gparted i had one big zfs partartion and a 8M of empty/reserved but no boot partition. I followed the above notes thinking i could install the grub again. update-grub worked nicely but the last but line i could not mount the missing partition. i tried # grub-install /dev/disk/by-id/diskuid but it spat that back at my face and said that it could not install grub this zfs partition.....its not a ....

i rebooted and it worked so update-grub might of fixed it? or just finishing the rebuild of the disks?

hope you get if fixed.

T

 

[Wouter Samaey]

The solution here is indeed booting from a live USB disk, mounting 3 things and running grub-install. This only works if there is no resilvering going on, so that’s confusing and terribly annoying.

Why the disks are partitioned differently is beyond me. I’m now in the process of replacing all disks with WD Red Pro disks, as this fixes my endless resilvering issue / seeing new disks as faulted.

I hope I wont run into any partition / grub issues by the time I’ve replaceed the last disk, so still looking for an answer why the partitions are different.

on the new disk i have a partition 1 and 9. Why this strange numbering, I dunno... there is no EFI partition, but Im not using EFI to be honnest.

i’d love to see all disks partitioned the same, but “zpool replace“ seams to have a different opinion.

 

[apoc]

on the new disk i have a partition 1 and 9.

This is typical to disks that are part oft a zpool. Nothing to worry imho.
Regarding your resilver-issue.
What exact HDD have you had (6TB)? Are the using SMR by any Chance? I do remember some HDDs use this technology even though not explicitly mentioned. And this gives a lot of trouble in combination with ZFS.

 

[Wouter Samaey]

@tburger yes, they were WD Red 6TB disks with SMR. There's a separate forum thread about this:
https://forum.proxmox.com/threads/z...lted-too-in-a-raidz2-rpool.76290/#post-340171

The WD Red Pro disks with CMR are good replacements.

 

[Wouter Samaey]

@tburger why does ZFS not create a BIOS Boot partition first? If we replace disks and forget to setup a bootloader, you can't boot anymore.
Did the wise people at ZFS On Linux not consider this or is there something I don't know?

 

[apoc]

That is a damn good question I have no answer to.
Technically I indeed am surprised now as well.

I never paid a thought because I run discrete pools and a separate ssd as boot device.