Configure bootloader on ZFS root file system after disk replacement

T

thorim

New Member
Dec 31, 2021
2
0
1
Germany
Dear proxmox community, I have proxmox (pve-manager/6.4-6/be2fa32c (running kernel: 5.13.0-19-generic)) installed on a ZFS root file system that is mirrored across two SSDs (sda and sdb). Recently, I had to successively replace both SSDs due to disk failure in the process of which I think I lost my boot partition. At least I don't seem to be able to boot into my proxmox installation anymore. I compiled the information below from chrooting into the proxmox system from a ubuntu 21.10 live system. The system is managed by the proxmox-boot-tool which was missing the two disks it had the IDs stored for when I ran it with the "status" argument from the chroot environment. Therefore, I ran it with the "clean" argument. However, it seems like now I need a dedicated EFI i.e. boot partition on sda and sdb but there doesn't seem to be any space left next to the zfs partitions which I also don't seem to be able to shrink. I then looked into ways to create a zfs boot dataset but I can't seem to figure out how to initialize this dataset with proxmox-boot-tool and/or bootctl. After all, if I'd be able to have a boot dataset, then I wouldn't need proxmox-boot-tool to keep anything in sync, which leaves me confused about whether that's an option at all.
In case I can help with additional information about my setup, please do not hesitate to ask and I'd be glad to deliver it.

zpool list
Code: 
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool   222G   142G  79.6G        -        8G    54%    64%  1.00x    ONLINE  /mnt

zfs list
Code: 
NAME                           USED  AVAIL     REFER  MOUNTPOINT
rpool                          142G  72.7G      104K  /mnt/rpool
rpool/ROOT                    92.6G  72.7G       96K  /mnt/rpool/ROOT
rpool/ROOT/pve-1              92.6G  72.7G     92.6G  /mnt
rpool/data                    49.8G  72.7G      112K  /mnt/rpool/data
rpool/data/subvol-130-disk-0   931M  7.29G      725M  /mnt/rpool/data/subvol-130-disk-0
rpool/data/subvol-150-disk-0   553M  7.46G      553M  /mnt/rpool/data/subvol-150-disk-0
rpool/data/subvol-160-disk-0   946M  7.08G      946M  /mnt/rpool/data/subvol-160-disk-0
rpool/data/subvol-170-disk-0   692M  19.3G      692M  /mnt/rpool/data/subvol-170-disk-0
rpool/data/subvol-180-disk-0  1.36G  8.64G     1.36G  /mnt/rpool/data/subvol-180-disk-0
rpool/data/subvol-190-disk-0   722M  9.30G      722M  /mnt/rpool/data/subvol-190-disk-0
rpool/data/subvol-200-disk-0   782M  9.24G      782M  /mnt/rpool/data/subvol-200-disk-0
rpool/data/vm-100-disk-0        56K  72.7G       56K  -
rpool/data/vm-100-disk-1      7.60G  72.7G     7.60G  -
rpool/data/vm-105-disk-0      2.27G  72.7G     2.27G  -
rpool/data/vm-110-disk-0      23.7G  72.7G     23.7G  -
rpool/data/vm-120-disk-0      10.3G  72.7G     10.3G  -

fdisk -l
Code: 
Disk /dev/loop0: 2.3 GiB, 2470006784 bytes, 4824232 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop1: 4 KiB, 4096 bytes, 8 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop2: 99.3 MiB, 104153088 bytes, 203424 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop3: 242.4 MiB, 254115840 bytes, 496320 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop4: 65.2 MiB, 68378624 bytes, 133552 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop5: 150.4 MiB, 157683712 bytes, 307976 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop6: 61.9 MiB, 64835584 bytes, 126632 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop7: 54.2 MiB, 56856576 bytes, 111048 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 232.9 GiB, 250059350016 bytes, 488397168 sectors
Disk model: Seagate IronWolf
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A4275BB5-DB57-F84E-BB8F-F8B44466BEBB

Device         Start       End   Sectors   Size Type
/dev/sda1       2048 488380415 488378368 232.9G Solaris /usr & Apple ZFS
/dev/sda9  488380416 488396799     16384     8M Solaris reserved 1


Disk /dev/sdb: 232.9 GiB, 250059350016 bytes, 488397168 sectors
Disk model: Seagate IronWolf
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 3224B143-26FF-B44F-BC6E-AE79AE56AB53

Device         Start       End   Sectors   Size Type
/dev/sdb1       2048 488380415 488378368 232.9G Solaris /usr & Apple ZFS
/dev/sdb9  488380416 488396799     16384     8M Solaris reserved 1


Disk /dev/sdd: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: WDC  WUS721010AL
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 5C960D8A-0B53-7C48-B20F-47E1908D4AF7

Device           Start         End     Sectors  Size Type
/dev/sdd1         2048 19532855295 19532853248  9.1T Solaris /usr & Apple ZFS
/dev/sdd9  19532855296 19532871679       16384    8M Solaris reserved 1


Disk /dev/sdc: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: WDC  WUS721010AL
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: CB4392E3-0611-E44E-9622-D3DFEBFE39E1

Device           Start         End     Sectors  Size Type
/dev/sdc1         2048 19532855295 19532853248  9.1T Solaris /usr & Apple ZFS
/dev/sdc9  19532855296 19532871679       16384    8M Solaris reserved 1


Disk /dev/sde: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: WDC  WUS721010AL
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: C8BB300D-0C91-8B4D-BE42-91C8356E97B0

Device           Start         End     Sectors  Size Type
/dev/sde1         2048 19532855295 19532853248  9.1T Solaris /usr & Apple ZFS
/dev/sde9  19532855296 19532871679       16384    8M Solaris reserved 1


Disk /dev/sdf: 3.8 GiB, 4043309056 bytes, 7897088 sectors
Disk model:                 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EF83665C-2F29-48D6-8C3F-80F5A69CFEB7

Device       Start     End Sectors  Size Type
/dev/sdf1       64 6077751 6077688  2.9G Microsoft basic data
/dev/sdf2  6077752 6086215    8464  4.1M EFI System
/dev/sdf3  6086216 6086815     600  300K Microsoft basic data
/dev/sdf4  6090752 7897024 1806273  882M Linux filesystem


Disk /dev/zd0: 1 MiB, 1048576 bytes, 2048 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes


Disk /dev/zd16: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes
Disklabel type: dos
Disk identifier: 0x8e6ef136

Device      Boot Start      End  Sectors Size Id Type
/dev/zd16p1 *     2048 67106815 67104768  32G 83 Linux


Disk /dev/zd32: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes
Disklabel type: dos
Disk identifier: 0xd63687db

Device      Boot Start      End  Sectors Size Id Type
/dev/zd32p1 *     2048 67106815 67104768  32G 83 Linux


Disk /dev/zd48: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes
Disklabel type: gpt
Disk identifier: 2CB27764-9389-4F56-BA0C-07F4CB8E22A9

Device        Start      End  Sectors  Size Type
/dev/zd48p1    2048    67583    65536   32M EFI System
/dev/zd48p2   67584   116735    49152   24M Linux filesystem
/dev/zd48p3  116736   641023   524288  256M Linux filesystem
/dev/zd48p4  641024   690175    49152   24M Linux filesystem
/dev/zd48p5  690176  1214463   524288  256M Linux filesystem
/dev/zd48p6 1214464  1230847    16384    8M Linux filesystem
/dev/zd48p7 1230848  1427455   196608   96M Linux filesystem
/dev/zd48p8 1427456 67108830 65681375 31.3G Linux filesystem


Disk /dev/zd64: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes
Disklabel type: dos
Disk identifier: 0x21dd8da5

Device      Boot Start      End  Sectors Size Id Type
/dev/zd64p1 *     2048 67106815 67104768  32G 83 Linux
 
I only did a zfs replace and from the looks of it should have followed the steps under "Changing a failed bootable device" in the documentation you linked, which explains where I would have gotten the partition table with the boot partition from. Thanks for highlighting the context that I was missing, this makes a lot of sense.

Given that I missed saving/copying the boot partition, would you recommend to create a new partition table with a boot partition on a fresh drive (which presumably needs to be larger than the existing ones), then use that one to replace one of the mirrored set and then reinstall efi and do the same with the other drive?
 
Last edited:
Jep, in case your not-failed drive is still healthy I would replace the failed drive again and this time by following the guide.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom