[SOLVED] zfs boot disk no longer bootable after rpool resilvering

[apax]

Dear passionate proxmox experts,

I have installed proxmox to boot from a mirrored (raid1) ZFS partition. So far so good:

# zpool status rpool
pool: rpool
  state: ONLINE
config:
  NAME STATE READ WRITE CKSUM
  rpool ONLINE 0 0 0
    mirror-0 ONLINE 0 0 0
      nvme-eui.0026b7683f1dbb55-part3 ONLINE 0 0 0
      nvme-eui0026b7683f123d455-part3 ONLINE 0 0 0

# lsblk
nvme0n1 259:3 0 465.8G 0 disk
├─nvme0n1p1 259:4 0 1007K 0 part
├─nvme0n1p2 259:5 0 512M 0 part
└─nvme0n1p3 259:6 0 399.5G 0 part
nvme1n1 259:0 0 465.8G 0 disk
├─nvme1n1p1 259:4 0 1007K 0 part
├─nvme1n1p2 259:5 0 512M 0 part
└─nvme1n1p3 259:6 0 399.5G 0 part

After few days, the second disk failed. I have detached it and replaced with another one. The process when fine and the zpool has been resilvered.

However the partitions (see the lsblk output) is different, as there is only a second partition of 8M instead of the original 2 additional partitions of 1007K and 512M.

# zpool status rpool
  pool: rpool
  state: ONLINE
  scan: resilvered 14.9G in 0 days 00:00:16 with 0 errors on Tue Oct 6 07:55:28 2020
config:
    NAME STATE READ WRITE CKSUM
    rpool ONLINE 0 0 0
        mirror-0 ONLINE 0 0 0
            nvme-eui.0026b7683f1dbb55-part3 ONLINE 0 0 0
            nvme-eui.0026b7683f1e9da5 ONLINE 0 0 0

# lsblk
nvme0n1 259:3 0 465.8G 0 disk
├─nvme0n1p1 259:4 0 1007K 0 part
├─nvme0n1p2 259:5 0 512M 0 part
└─nvme0n1p3 259:6 0 399.5G 0 part
nvme1n1 259:0 0 465.8G 0 disk
├─nvme1n1p1 259:1 0 465.8G 0 part
└─nvme1n1p9 259:2 0 8M 0 part

Proxmox still boots ok on the first disk and the zfs array is ok. However when I tried to remove the first disk, Proxmox does not boot on the second disk. So I am afraid that the effort of having a mirrored boot disk is voided.

Could you help me with some details (step-by-step) instructions on how to rebuild the initial bootable partition table on a zfs bootable mirrored pool ?

I have read all the documentation on pve-efiboot-tool, but it is still unclear in shich order you do things: Is it detach from zpool , re-partition the disk and attach to the zpool and resilver ?

Many, many thanks in advance for your help…

Apax.

 

[Stoiko Ivanov]

The partitioning for the disks done by the installer includes the 512M efi service partiton (ESP) - which is where the kernel+initrd is placed

The procedure for replacing such a disk is explained in the reference documentation:
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#_zfs_administration

I hope this helps!

 

[apax]

Indeed, it helped ! Many thanks, i had skipped the sgdisk command, and now it is all ok.

For reference, here it is how it was solved:

/dev/nvme1n1 is the old disk to keep
/dev/nvme1n1 is the new disk to add

sgdisk /dev/nvme0n1 -R /dev/nvme1n1
sgdisk -G /dev/nvme1n1

#find the new device name
ls /dev/disk/by-id

zpool attach -f rpool nvme-eui.0026b7683f1dbb55-part3 nvme-eui.0026b7683f1e9da5-part3
zpool status rpool

wait to resilver to complete and format partition 2 (the one of 512 MB size)

pve-efiboot-tool format /dev/nvme1n1p2
pve-efiboot-tool init /dev/nvme1n1p2
pve-efiboot-tool refresh

and ...reboot !

Apax

 

[Stoiko Ivanov]

Glad that helped!

Please mark the thread as 'SOLVED' - this helps other users

Thanks!

 

[apax]

Thanks, Stoiko.
Done. Apax

 

[DeD_Morozzzz]

Indeed, it helped ! Many thanks, i had skipped the sgdisk command, and now it is all ok.

For reference, here it is how it was solved:

/dev/nvme1n1 is the old disk to keep
/dev/nvme1n1 is the new disk to add

sgdisk /dev/nvme0n1 -R /dev/nvme1n1
sgdisk -G /dev/nvme1n1

#find the new device name
ls /dev/disk/by-id

zpool attach -f rpool nvme-eui.0026b7683f1dbb55-part3 nvme-eui.0026b7683f1e9da5-part3
zpool status rpool

wait to resilver to complete and format partition 2 (the one of 512 MB size)

Hello.
I have doen everything as you advised, Resilvering went ok, but it seems that "
pve-efiboot-tool format /dev/nvme1n1p2
pve-efiboot-tool init /dev/nvme1n1p2
pve-efiboot-tool refresh
"
part didn't work. new drives are not bootable. Are there any more lines to perform?