Proxmox ZFS Mirror Boot Drive Failure - How to Restore

[ligistx]

Hi,

I run a proxmox homelab, and have the proxmox installed on 2 SSD's in a ZFS mirror. I also store my VM's on the host drive, but that shouldn't be relivant to what I need to do.

I am somewhat fmailular with ZFS, but really only in truenas where there are simple GUI options to manage everything. With Proxmox, what I need to do is pop in a 3rd (temporary SSD), replicate the mirror to that drive, and then remove the failing boot drive. This will retain my 2 way mirror, but will allow me to remove the failing drive to get it RMAed and replaced. In truenas this is pretty easy, but I am unsure how to go about this in Proxmox since the system boots from this ZFS pool...

Are there any instrucitons on how to do this? I can't seem to find a simple step by step example of how to go about this.

It looks like I am using proxmox boot tool:

Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
1249-18B5 is configured with: uefi (versions: 6.2.16-20-pve, 6.5.11-7-pve, 6.5.13-1-pve, 6.5.13-3-pve)
1249-7D78 is configured with: uefi (versions: 6.2.16-20-pve, 6.5.11-7-pve, 6.5.13-1-pve, 6.5.13-3-pve)

So it looks like I do:
# sgdisk <healthy bootable device> -R <new device>
# sgdisk -G <new device>
# zpool replace -f <pool> <old zfs partition> <new zfs partition>
# proxmox-boot-tool format <new disk's ESP>
# proxmox-boot-tool init <new disk's ESP> [grub]

I am a bit conufsed, what is the <old zfs partition> and <new zfs partition>, and what is <new disk's ESP>?

 

[ligistx]

@Dunuin you have been helpful before, maybe you have some input here?

​

 

[Dunuin]

what is the <old zfs partition> and

Partition 3 of your failed disk.

<new disk's ESP>?

Partition 2 of the third disk you added and cloned the partition table to replace the bad disk.

Don't use the "[grub]" as you are using the ESP.

 

[ligistx]

Partition 3 of your failed disk.

Partition 2 of the third disk you added and cloned the partition table to replace the bad disk.

Don't use the "[grub]" as you are using the ESP.

Ok, just to make sure I get this right...

After running lsblk, I am seeing my nvme drives as:


Lets assume I will pull nvme1n1 out of the system, leaving only nvme0n1 and a stand in SATA drive I will be adding for the short term just to keep things in a mirror (lets assume that will be called "sda"
I would then:
sgdisk <nvme0n1> -R <sda>
sgdisk -G <sda>
zpool replace -f <rpool> <nvme0n1p3> <sda3>
proxmox-boot-tool format <spa2>

and I don't need to do proxmox-boot-tool init <spa2> [grub] ?

Sorry for the likely obvious questions, I just really do not like messing with disks and boot media for my homelab. Since its relatively difficult to backup proxmox, I don't have a great backup mechanism, so I really try and not create any self induced issues.

 

[Dunuin]

zpool replace -f <rpool> <nvme0n1p3> <sda3>

No, should be partition 3 of your failed disk that gets replaced by sda3. So: "zpool replace -f <rpool> <nvme1n1p3> <sda3>"

and I don't need to do proxmox-boot-tool init <spa2> [grub] ?

You need to, but only "proxmox-boot-tool init <sda2>" and not "proxmox-boot-tool init <sda2> grub".

 

[ligistx]

No, should be partition 3 of your failed disk that gets replaced by sda3. So: "zpool replace -f <rpool> <nvme1n1p3> <sda3>"

Oh wow, good catch... I definitely typoed that, would have been not so great if I went ahead and did that.

You need to, but only "proxmox-boot-tool init <sda2>" and not "proxmox-boot-tool init <sda2> grub".

ah ok. Thanks!

 

[Dunuin]

Oh wow, good catch... I definitely typoed that, would have been not so great if I went ahead and did that.

Yes, would result in a lost pool as you are forcing the replace.

 

[ligistx]

Ok

No, should be partition 3 of your failed disk that gets replaced by sda3. So: "zpool replace -f <rpool> <nvme1n1p3> <sda3>"


You need to, but only "proxmox-boot-tool init <sda2>" and not "proxmox-boot-tool init <sda2> grub".

Ok, I guess I should have seen this coming as well, but how do I deal with this exactly? If I physically remove a drive (simulating a full failure... device no longer is recognized at all... in reality, the old drive is just showing SMART errors so its getting swapped and now sitting on my desk ready for RMA once I wipe it of course, but this is a good desaster recovery practice), how do I go about this?

I no longer have the nvme1n1 drive in the system..






It looks like it is saying to use zpool replace. Since I really don't want to mess this up, what would the correct steps here be? I guess its a little different considering I longer have the "old" drive.

sgdisk <nvme0n1> -R <sdk>
sgdisk -G <sdk>
zpool replace -f <rpool> <???> <sdk3>
proxmox-boot-tool format <sdk2>
proxmox-boot-tool init <sdk2>

Do I just substitute "???" with the old drives name in the mirror, "7416205833635307642"?

Or, looking in an actual SSH session:



Should I instead be doing:

sgdisk <nvme0n1> -R <sdk>
sgdisk -G <sdk>
zpool replace -f <rpool> <nvme-eui.002538d5115133f5-part3> <sdk3>
proxmox-boot-tool format <sdk2>
proxmox-boot-tool init <sdk2>

Thanks agian for the help with this. I wish this was made slightly more clear in the MAN pages. Not so difficult in truenas, a bit more difficult here since its a boot volume...

 

[ligistx]

For anyone who may see this in the future, looks like this was correct:

sgdisk /dev/nvme0n1 -R /dev/sdk
sgdisk -G /dev/sdk
zpool replace -f rpool /dev/disk/by-id/nvme-eui.002538d5115133f5-part3 /dev/sdk3
proxmox-boot-tool format /dev/sdk2
proxmox-boot-tool init /dev/sdk2

I performed this and seemingly it worked... Have not actually tried to reboot, or booth from the newly added device, but I imagine this was correct based on the above information for my disks.

 

[Kingneutron]

FYI you should absolutely test that reboot will work reliably from either drive. Otherwise there's no point having a mirror

 

[ligistx]

FYI you should absolutely test that reboot will work reliably from either drive. Otherwise there's no point having a mirror

I agree. I will before I wipe old drive.

 

[Daniel Drucker]

For others:

I followed these instructions to replace half of my boot mirror and got as far as the very last line above, but got "bootctl is not available - make sure systemd-boot is installed".

The solution was, in fact, to apt install systemd-boot.