Installation on NVMe on systems that don't support booting from NVMe

[no-usernames-left]

Hello,

I'd like to install Proxmox on a pair of NVMe drives (using ZFS mirroring) for performance plus redundancy as well as to free up the machine's 3.5" drive bays for very large spinning-rust disks (20+ TB each). There is, of course, a catch: the system (Dell R710) is old enough that it doesn't support booting from NVMe.

I scrubbed the documentation for 8.1 but I can't see a known-good documented process to install the ESP (and/or /boot) on some other media, for example a high-endurance SD card (which the machine in question _can_ boot from and which is super easy to image after installation).

What would you recommend?

Thanks in advance!

 

[leesteken]

Install on the NVMe drives, rescue boot with the Proxmox installer into the new installation add the a ESP on the SD card using proxmox-boot-tool.

 

[_gabriel]

even regular usb for esp is enough to bootloader, as no write is done after booting.
esp part doesn't need to mounted.
proxmox-boot-tool will auto mount if/when you update the kernel.

 

[no-usernames-left]

Install on the NVMe drives, rescue boot with the Proxmox installer into the new installation

It looks like the rescue functionality of the installer doesn't have support for ZFS; when I reboot back into the installer and choose the rescue option, I get an error that rpool couldn't be found, and it errors out.

Any other ideas?

add the a ESP on the SD card using proxmox-boot-tool.

It looks like the partitions have to be set up first before pointing `proxmox-boot-tool` at the specific partition, and I can't find where it's documented. However, I installed Proxmox on just the SD card then rebooted and blew away the third (ZFS) partition to get it set up correctly, then I imaged the SD card for good measure.

 

[leesteken]

It looks like the rescue functionality of the installer doesn't have support for ZFS; when I reboot back into the installer and choose the rescue option, I get an error that rpool couldn't be found, and it errors out.

Any other ideas?

I hoped that that would just work, sorry. I always install new a Proxmox in a VM and then move the virtual disk via USB to a real disk.

It looks like the partitions have to be set up first before pointing `proxmox-boot-tool` at the specific partition, and I can't find where it's documented.

Just use gdisk or GParted Live for a GUI. I don't think Proxmox has a guide for that, but it's "just Linux".

However, I installed Proxmox on just the SD card then rebooted and blew away the third (ZFS) partition to get it set up correctly, then I imaged the SD card for good measure.

This would probably also work: Install Proxmox on the SD card and start it, partition the NVMe drives, attach the partition of both of them as a mirror to the SD card, remove the SD card from the mirror.

 

[no-usernames-left]

I hoped that that would just work, sorry. I always install new a Proxmox in a VM and then move the virtual disk via USB to a real disk.

Just use gdisk or GParted Live for a GUI. I don't think Proxmox has a guide for that, but it's "just Linux".

This would probably also work: Install Proxmox on the SD card and start it, partition the NVMe drives, attach the partition of both of them as a mirror to the SD card, remove the SD card from the mirror.

I thought about making it a 3-way mirror then removing the SD card from the vdev after the fact, but then I worried that the pool would be space-constrained as a result of being "held back" by the smallest member and then I'd have to expand things and it would just be a mess.

So this is what I did:

1. Booted from the install USB
2. Installed onto only the two NVMe SSDs as a ZFS mirror (RAID1)
3. Rebooted
4. Booted from the install USB again
5. Installed onto the SD card as 1-drive ZFS RAID0 and nothing else
6. Rebooted
7. Booted into a live environment with ZFS support (Debian 12 live ISO should work fine, Ventoy is your friend*)
8. Used fdisk to delete the third (ZFS) partition on the SD card
9. Rebooted and set the SD card as the first UEFI boot selection

* Note that Ventoy 1.0.96 doesn't boot Proxmox 8.1 ISO, but it should work in the next release of Ventoy.

I would imagine that it would also work just as well if steps 2 and 5 were swapped.

The kernel was booted from SD, it picked up the rpool on the NVMe SSDs, and the box came up.

I noticed that proxmox-boot-tool status didn't know about it, so I used proxmox-boot-tool init and specified the second (ESP) partition. At that point, proxmox-boot-tool status was aware of it and should thus keep it up-to-date going forward. Hurrah!

Thanks for your help!

EDIT: After rebooting, the box didn't come back up, but was stuck in the initramfs, which was complaining that the pool rpool was last in use on another system (well, another installation I suppose, not wrong).

I had to zpool import rpool -f and then reboot (and /sbin/reboot didn't do anything for some reason, so I used a larger hammer).

 

[alexskysilk]

if your system is capable of UEFI boot but is just missing the nvme boot module, you may be able to insert it as described here: https://winraid.level1techs.com/t/h...t-for-all-systems-with-an-ami-uefi-bios/30901

 

[no-usernames-left]

if your system is capable of UEFI boot but is just missing the nvme boot module, you may be able to insert it as described here: https://winraid.level1techs.com/t/h...t-for-all-systems-with-an-ami-uefi-bios/30901

Thank you! I was aware of that possibility, but this was a lower-risk workaround than possibly b0rking the server's firmware.