Can I boot a ZFS install from a USB with just the boot loader installed?

[trevmlt]

I have a ZFS Proxmox 7 installation using the systemd-boot bootloader on an old spinning disk I would like to upgrade to an NVMe drive. My currrent (older) motherboard will not boot from NVMe without a BIOS hack, which I would like to avoid if possible. Is there a way to boot from a USB stick, but run the system from the NVMe with my configuration? I found a couple of bits of info here and there, but nothing recent (most talk involves grub).

 

[trevmlt]

With UEFI/systemd boot this does work on an Asrock EP2C602-4L/D16. After moving the rpool around between the NVMe and spinning disk, it was just a matter of formatting and copying the 1st and 2nd partitions to the USB, and then making sure the sticks UUID was present in /etc/kernel/proxmox-boot-uuids. Rebooted and system came up.

 

[Thdi]

Hi trevmlt,

Thank you for sharing your experience! I’m currently facing a very similar challenge: I’ve installed Proxmox on an NVMe SSD with ZFS, and I need to boot the system from a USB stick because my motherboard doesn’t support native PCIe boot.

Your approach sounds very promising, and I’d really appreciate it if you could share a bit more detail about the process. Specifically, I’m curious about:

1. How did you set up the USB stick? Did you only install the bootloader, or did you need to make additional modifications?

2. What steps were required to link the USB stick with the NVMe root pool?

3. Did you encounter any issues with ZFS integration during boot, and how did you handle them?

4. Are there any specific pitfalls or recommendations you’d suggest to ensure a smooth setup?

Your comment about formatting and copying the first and second partitions and ensuring the UUID is present in /etc/kernel/proxmox-boot-uuids is very helpful. It would be great if you could elaborate a bit on these steps or provide examples.

I’m sure many in the community would benefit from your insights. Thanks in advance for taking the time to share your experience!

Best regards,
T

 

[p.b]

Cant't you use proxmox-boot-tool for that?

If you have an emtpy patition on your USB-Stick it should be just a matter of something like:

proxmox-boot-tool format /dev/disk/by-id/usb-stick-partX
proxmox-boot-tool init /dev/disk/by-id/usb-stick-partX
# or for GRUB
# proxmox-boot-tool init /dev/disk/by-id/usb-stick-partX grub
proxmox-boot-tool refresh

#  not sure if below is needed - might work without
update-initramfs -u -k all

I've used similar commands to (re-)recreate boot partitions on so many internal disks, nvmes and usb-devices without any issues.
The partition layout can easily be done using parted (mklabel, mkpart), sgdisk and wipefs.

 

[Thdi]

Cant't you use proxmox-boot-tool for that?

If you have an emtpy patition on your USB-Stick it should be just a matter of something like:

proxmox-boot-tool format /dev/disk/by-id/usb-stick-partX
proxmox-boot-tool init /dev/disk/by-id/usb-stick-partX
# or for GRUB
# proxmox-boot-tool init /dev/disk/by-id/usb-stick-partX grub
proxmox-boot-tool refresh

#  not sure if below is needed - might work without
update-initramfs -u -k all

I've used similar commands to (re-)recreate boot partitions on so many internal disks, nvmes and usb-devices without any issues.
The partition layout can easily be done using parted (mklabel, mkpart), sgdisk and wipefs.

Hey, thanks for your reply! After doing a lot of research, this thread definitely pointed me in the right direction. I later found this discussion: Move Boot Partition ZFS USB, where the original author of the linked post provides a perfect roadmap for the scenario I was facing.

In the hope that this will help someone else at some point, I am paraphrasing his solution here:

From the orginal thread: "Move boot partition ZFS USB"​

[...] "For anyone wanting to replicate this is the process:
(Assuming an unbootable proxmox 8 install due to raid card & proxmox 8 live iso)

• Delete all existing partitions on your usb or any other device that can boot and is unaffected by raid card. (at least 1 GB space, depending on number of kernels installed)
• To replicate the setup above and have the first sector start at 34, start with gdisk /dev/sdX, press "x" for expert mode, "l" for alignment and set it to 1, press "m" to return back to normal mode
• "n" to create new partition with number 1 (sdX1), first sector: 34, last sector: 2047, Hex: EF02
• "n" to create new partition with number 2 (sdX2), first sector: 2048, last sector: 1050623, Hex: EF00 (might want to increase last sector number if planning on having more kernels)
• "w" to write changes and exit
• Now mount rpool to /mnt with zpool import rpool -R /mnt
• And mount /proc /sys /dev /run with:

Bash:
<span>mount</span> -t proc proc /mnt/proc<br><span>mount</span> -t sysfs sys /mnt/sys<br><span>mount</span> -o <span>bind</span> /dev /mnt/dev<br><span>mount</span> -o <span>bind</span> /run /mnt/run

• chroot /mnt
• proxmox-boot-tool format /dev/sdX2 --force and proxmox-boot-tool init /dev/sdX2
• Now check if the UUID of /dev/sdX2 from blkid /dev/sdX2 is inside /etc/kernel/proxmox-boot-uuids, if not try to run proxmox-boot-tool refresh
• To unmount the system folders we need to run umount for each /mnt/proc /mnt/sys /mnt/dev /mnt/run
• zpool export rpool, try -f to force the export
• Important is to export the zfs pool, because proxmox doesn't boot with force flag on pool import during boot" – @whytf, Dec 20, 2023

---

I may add:
There are certainly other ways to do this, but for me, the most convenient approach was to work with three USB drives and one or two NVMe drives.

• USB1: Proxmox ISO
• USB2: Used for the first Proxmox installation (as a live environment). This has a nice shell application in the GUI, which is helpful if your keyboard mapping is messed up. 'Copy & paste' works like a charm here.
• USB3: The BOOT drive to be.
• NVMe: Final Proxmox installation with ZFS. (I mirrored the root system across two drives, but you can also use a single NVMe with ZFS.)

For me, it made perfect sense to have the installation setup on the USB and the NVMe drive identical—same IP address, etc.—except for ZFS! I would recommend keeping the Proxmox USB installation on a simple drive without ZFS to avoid conflicting ZFS 'rpools'.

Good luck to anyone trying to achieve this!