I just finished accomplishing this. It's not too tough once you know how to get past a few nigglies. Many thanks to @Dunuin and to the author linked below. And of course to the awesome ProxMox team.
Prerequisites
This sequence is based on a few assumptions. If your situation is significantly different, you'll need to modify! In particular, I'm using UEFI boot, NOT grub.
We'll go through the following steps to accomplish this:
DETAILS
Create a bootable USB stick with the current ProxMox installer.
Shutdown all VM's. Take a snapshot of rpool. Replicate rpool to new-rpool. Fix up settings appropriately.
Shutdown host, Boot to BIOS. Set USB as first device, NEW storage as second. Boot to USB ProxMox installer, load in debug terminal (shell) mode.
Attach the two partitions to form a mirror for rpool. Verify. DONE!
NOTE: if you skipped to here from above, and didn't need to move over to a smaller disk, then add new to old like this:
DONE.
References
https://aaronlauterer.com/blog/2021/proxmox-ve-migrate-to-smaller-root-disks/
https://forum.proxmox.com/threads/adding-2nd-boot-drive-is-anything-needed-on-partition-1.144164/
https://pve.proxmox.com/wiki/ZFS_on_Linux#_zfs_administration (portions of "Changing a failed bootable device")
Prerequisites
This sequence is based on a few assumptions. If your situation is significantly different, you'll need to modify! In particular, I'm using UEFI boot, NOT grub.
- Functioning ProxMox host, with a single ZFS boot drive using UEFI. (
proxmox-boot-tool statussays so) - You're comfortable with creating USB sticks from ISO files, adjusting boot device sequence in host BIOS settings, etc. (I'll refer to the tools and methods but not explain in detail as that depends completely on your exact hardware etc.)
- You have physical access to the host, including screen, keyboard and USB for boot. Or virtualized equivalent.
- You have a backup in case of a serious mistake.
- You understand my perspective: from cautious experience, I do NOT count on "simple" device names like /dev/nvme0n1 to be consistent. I've been burned too many times. Thus, while it takes a little more work, using /dev/disk/by-id/<whatever> is safer. (Hint: save a few strings in a file that's easy to copy-paste from on-screen. Saves a lot of typos.)
We'll go through the following steps to accomplish this:
- Create a bootable USB stick with the current ProxMox installer.
- Shutdown host, install the new device (I assume it is blank.)
- Before booting again, verify you can boot to the ProxMox installer. Do NOT use it yet! Pull the USB and boot as normal.
- Verify new device is visible to the host.
- Capture the partition table of current rpool device, and appropriate by-id names of current and new devices.
- Define the needed partitions on new. Verify new is smaller than current. (If new is same or larger than current skip steps 7-12 following!)
- Setup boot management on new, create new-rpool on new
- Shutdown all VM's. Take a snapshot of rpool. Replicate rpool to new-rpool. Fix up settings appropriately.
- Shutdown host, Boot to BIOS. Set USB as first device, NEW storage as second. Boot to USB ProxMox installer, load in debug terminal (shell) mode.
- Verify access to rpool and new-rpool. Import and rename the pools (rpool to old-rpool, new-rpool to rpool.) Then export.
- Shutdown, pull USB, boot. All should be well except now you're booting from rpool on the new smaller drive.
- Verify all is well. Destroy old-rpool.
- Attach the two partitions to form a mirrored rpool. Verify. DONE!
DETAILS
Create a bootable USB stick with the current ProxMox installer.
- Follow https://pve.proxmox.com/wiki/Prepare_Installation_Media
- I used Rufus on Windows. Rufus auto-selects the right options.
- Make sure the new device is visible to the BIOS of the host. (This was actually my biggest hassle: I had to make some tough hardware decisions because I didn't have 4 spare PCIe lanes for the new NVMe drive. It was invisible?!!!)
- This is more BIOS work, ensuring you have the correct boot order sequence.
- If you can boot to the installer screen, you're fine. Just shut off the hardware, pull USB, and boot back to the usual ProxMox.
lsblkshould show you current and new devices.
- You do not want to lose the partition table info.
ls /dev/disk/by-idwill show all disks by ID. There may be multiple ID's for the exact same device. I chose the shortest one for current and new.- From here on out, I'm going to write
/dev/disk/by-id/<current-id>but that really means (for example)/dev/disk/by-id/nvme-CT1000T500SSD5_234344CF32EB - To show the current device partition table:
sgdisk -p /dev/disk/by-id/<current-id>(Be sure it follows the pattern in the next step. If not, something is seriously wrong (ie these are not the instructions you need.) - To save the partition table on disk:
sgdisk /dev/disk/by-id/<current-id> -b /tmp/part-backupfile.bin - NOTE: as long as it's the standard partition table, you can easily recreate it without having saved it! They are all the same design. However, the unique UUID's will change... AND, each partition table in detail is specific to one drive or one exact hardware type:
- Although partition three is defined the same (ie "use the rest of the disk"), the actual partition definition will be different, particularly in the case we are dealing with here, of adding a new smaller drive. The end sector number is different. Thus, you can't just load the partition table from old to new! You would have a broken partition and broken GPT (because a secondary copy of GPT is stored at the end of the disk.)
- You already have
sgdisk -pfor current, correct? OK, let's build the new disk:
- (If the final sector number on new is not smaller than current, skip to the last step below!)
- do
ls /dev/disk/by-idagain. You should see your new disk with three new partitions labeled -part1, -part2 and -part3. We're going to use part2 and part3 ... - These commands will format and add the EFI partition to ProxMox boot manager:
- Now let's create new-rpool in new device partition 3:
Shutdown all VM's. Take a snapshot of rpool. Replicate rpool to new-rpool. Fix up settings appropriately.
- Time to shut down your VM's. Yep, we need to go offline for the transfer to go quickly (and we will soon reboot the host again.)
- Now, we'll use send/recv of a complete snapshot to get everything into the new smaller drive:
# Make recursive snapshot of the entire pool
zfs snapshot -r rpool@migrate
# Replicate to new-rpool (I was pleasantly surprised that this went quickly for me)
zfs send -R rpool@migrate | zfs recv new-rpool -F
# set next boot to aim at new-rpool in new-rpool. Don't worry, it will be renamed correctly soon...
zpool set bootfs=new-rpool/ROOT/pve-1 new-rpool
Shutdown host, Boot to BIOS. Set USB as first device, NEW storage as second. Boot to USB ProxMox installer, load in debug terminal (shell) mode.
- I assume you know how to accomplish these steps...
- The current ProxMox installer has debug-install options in the main "Advanced" menu
- Do Not Worry that you're beginning an install! We will NOT let it actually overwrite anything
- HOWEVER, you MUST follow instructions carefully. It is certainly possible to goof and overwrite.
- When you boot the installer into debug-terminal mode, you should see lots of text fly by, then a (root) "#" prompt. You can see a demo of this at the aaronlauter link below.
- You are looking at an initramfs shell. Type "exit" and press enter to get to a live linux.
- Now you have another "#" prompt.
zpool importshould list your available pools, including especially rpool and new-rpool- NOTE: It will say they were used in another system. That's true! You are running the installer Linux, not your normal ProxMox. That's why the ZFS pools are not active.
- NOTE: It will say they were used in another system. That's true! You are running the installer Linux, not your normal ProxMox. That's why the ZFS pools are not active.
- IF you do NOT see the pools in
zpool import, do NOT go further! Pull the USB, shut off the machine, boot to ProxMox and diagnose.
- If you're at this step, you should be in the ProxMox installer and have seen a good result of
zpool importlisting rpool and new-rpool as available. - Let's import and rename (step one) then export again (from the installer.) This will make the new smaller rpool become the live active one in ProxMox:
- Congratulations, just about done
- I assume you know how to do this. Clean up BIOS boot order as needed. Both disks should be visible as bootable, but I would boot from the new smaller disk.
proxmox-boot-tool statusshould look good
zpool statusshould show rpool. old-rpool will likely show inzpool import- Look around. Ensure you are happy with the new rpool, the new drive in use as primary boot.
- Let's finish off the old rpool:
Attach the two partitions to form a mirror for rpool. Verify. DONE!
NOTE: if you skipped to here from above, and didn't need to move over to a smaller disk, then add new to old like this:
But if you DID have to move, you will add old to new:
DONE.
References
https://aaronlauterer.com/blog/2021/proxmox-ve-migrate-to-smaller-root-disks/
https://forum.proxmox.com/threads/adding-2nd-boot-drive-is-anything-needed-on-partition-1.144164/
https://pve.proxmox.com/wiki/ZFS_on_Linux#_zfs_administration (portions of "Changing a failed bootable device")
Last edited: