[TUTORIAL] Expanding Mirrored ZFS UEFI Boot Pool (replacing NVME drives)

[scyto]

***WARNING***

• make sure you have backups, etc etc etc you are messing with your boot disks
• the use of the hardware cloner made me comfortable because i knew worse case i could put the old nvme's back in the machine
• don't blame me if it doesnt work / messes up
• this was done on proxmox 8.4.1

Background

• I have a proxmox machine with had 250GB Kingston PLP drives that were in a mirrored ZFS boot pool, using UEFI to boot
• I decided i needed more room on the host as all my other SATA, NVME drives are passed through to Truenas
• My motherboard only supports 2280 drives.
• But i wanted to make use of some of the good deals on Micron 7400 Pro 4TB 2110 drives.
• I didn't want to reinstall.
• I had used all my PCIE slots and nvme extender adapters didn't work (not sure why)

Hardware Preparation

• Turned off server ;-)
• removed the nvmes in the 2280 sockets on the motherboard
• used an nvme hardware cloner to clone the drives
• add the drives to this! it sits between the pcie slot and pcie4 x4 nvme adapter i am using for a hailo card
• set BIOS to bifurcation 4x4x8 (the documentation on the box of card said 8x4x4 but that didn't work, 4x4x4x4 worked too)
• make sure both drives are seen in the BIOS
• then boot OS

Software Commands
note: i don't know if all of these are needed / or best way, but tried several different sets of instructions, in the end chatgpt got me there

set the pool to auto expand

zpool set autoexpand=on rpool

Get the sector offsets needed for the sgdisk command.

root@pve-nas1:~#
sgdisk -p /dev/nvme9n1
sgdisk -p /dev/nvme8n1

Disk /dev/nvme9n1: 7501476528 sectors, 3.5 TiB
Model: Micron_7400_MTFDKBG3T8TDZ     
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 727C5798-4D54-4F67-A4B6-CB0AA6088BA9
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 468862094
Partitions will be aligned on 8-sector boundaries
Total free space is 1197198 sectors (584.6 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02
   2            2048         2099199   1024.0 MiB  EF00
   3         2099200       467664896   222.0 GiB   BF01


root@pve-nas1:~#
sgdisk -p /dev/nvme8n1

Disk /dev/nvme8n1: 7501476528 sectors, 3.5 TiB
Model: Micron_7400_MTFDKBG3T8TDZ     
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 06A1E705-C7B1-47B7-9FE1-1D417EAF3167
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 468862094
Partitions will be aligned on 8-sector boundaries
Total free space is 1197198 sectors (584.6 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02
   2            2048         2099199   1024.0 MiB  EF00
   3         2099200       467664896   222.0 GiB   BF01

expand first nvme

sgdisk --delete=3 /dev/nvme9n1
sgdisk --new=3:2099200:0 /dev/nvme9n1
sgdisk --typecode=3:bf01 /dev/nvme9n1

expand second nvme

sgdisk --delete=3 /dev/nvme8n1
sgdisk --new=3:2099200:0 /dev/nvme8n1
sgdisk --typecode=3:bf01 /dev/nvme8n1

reboot

check disks