[SOLVED] Move boot partition ZFS USB

[whytf]

Hello,

What i am trying to achieve is this:
I have a hp raid card (p420i) that does not allow booting from itself while in HBA mode, therefore i decided, i could try to move the boot partition to a usb drive and have all the drives directly in proxmox with zfs.

However i am not sure if i am doing it correctly, i found these sources (https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_proxmox_boot_tool & https://unix.stackexchange.com/ques...r-debian-to-usb-flash-installed-on-two-nvme-d)

So i got into proxmox TUI in debug mode, installed the proxmox on the drives attached to the raid card with raidz1, converted my usb to gpt with gdisk and followed the second article, however i only got here with this problem:

/sbin/proxmox-boot-tool: 198: grub-install.real: not found

(would paste the log but it is the ipmi video output)

If i am not doing it correctly, how could this be achieved in a different way?

Thanks for help.

 

[whytf]

bump

 

[leesteken]

If you have installed Proxmox and can chroot (via rescue boot) into it, adding the USB drive as ESP with proxmox-boot-tool format+init should work.

/sbin/proxmox-boot-tool: 198: grub-install.real: not found

It's not clear to me why you would get this error. How did you chroot into the installed Proxmox? Maybe a bind mount was missing?

 

[whytf]

If you have installed Proxmox and can chroot (via rescue boot) into it, adding the USB drive as ESP with proxmox-boot-tool format+init should work.

It's not clear to me why you would get this error. How did you chroot into the installed Proxmox? Maybe a bind mount was missing?

Firstly i got into terminal installer debug mode and with ctrl + c when the installer asked for reboot, i got back into terminal and tried these commands

 

[leesteken]

Firstly i got into terminal installer debug mode and with ctrl + c when the installer asked for reboot, i got back into terminal and tried these commands

At that time, there is no working Proxmox yet, so maybe that's why it did not work. Please try rebooting and rescue boot and chroot into the Proxmox installation first.

 

[whytf]

At that time, there is no working Proxmox yet, so maybe that's why it did not work. Please try rebooting and rescue boot and chroot into the Proxmox installation first.

Well, it can't find the rpool for some reason, and it returns me to the main menu of the installer

 

[whytf]

If you have installed Proxmox and can chroot (via rescue boot) into it, adding the USB drive as ESP with proxmox-boot-tool format+init should work.

It's not clear to me why you would get this error. How did you chroot into the installed Proxmox? Maybe a bind mount was missing?

Here are the mounts after terminal installation (cancelled before reboot):

Don't see any boot mount though

 

[leesteken]

Well, it can't find the rpool for some reason, and it returns me to the main menu of the installer

That's disappointing and I don't know how to fix that.

Here are the mounts after terminal installation (cancelled before reboot):
View attachment 59940
Don't see any boot mount though

proxmox-boot-tool mount ESP's separately when you run it, Where is your new installation root mounted? Can you chroot into it and then run proxmox-boot-tool? Or did you try that already?

 

[whytf]

proxmox-boot-tool mount ESP's separately when you run it, Where is your new installation root mounted? Can you chroot into it and then run proxmox-boot-tool? Or did you try that already?

After the installation, i guess that by default the zfs pool is unmounted for restart, zpool list said that there are no available pools, but by manually trying "zpool import rpool", it actually imported:


The problem is that the "pve-1", "ROOT" is empty, as well as "data", so i can't chroot:


And the proxmox-boot-tool init, failed with same error

 

[leesteken]

After the installation, i guess that by default the zfs pool is unmounted for restart, zpool list said that there are no available pools, but by manually trying "zpool import rpool", it actually imported:
View attachment 59942

The problem is that the "pve-1", "ROOT" is empty, as well as "data", so i can't chroot:

The default mountpoint for rpool/ROOT/pve-1 is /. Maybe you can specify a different mountpoint temporarily (like /mnt or any other empty directory) with zpool import rpool -R /mnt?

 

[whytf]

The default mountpoint for rpool/ROOT/pve-1 is /. Maybe you can specify a different mountpoint temporarily (like /mnt or any other empty directory) with zpool import rpool -R /mnt?

Okay so now i am able to chroot into it, but now the proxmox-boot-tool says this:


I guess the /dev has to be mounted to actual /dev from live installer or somehow passthrough the usb to this chroot

I tried zpool import -o altroot=/mnt rpool but i get the same error

 

[leesteken]

Okay so now i am able to chroot into it, but now the proxmox-boot-tool says this:

I guess the /dev has to be mounted to actual /dev from live installer or somehow passthrough the usb to this chroot

Before chrooting you need to bind mount /dev etc. as usual (but maybe you did not know this) like it is done here: https://pve.proxmox.com/wiki/Recover_From_Grub_Failure
Maybe also use /dev/disk/by-path/usb-... or something to point to your USB drive as /dev/sd* is not stable.

EDIT: You also need to partition the USB drive and create the bootable ESP partition yourself first using gdisk, if you have not done so already.
EDIT2: And also run proxmox-boot-tool format before init, but make very sure that you are formatting the right partition.

 

[whytf]

Before chrooting you need to bind mount /dev etc. as usual (but maybe you did not know this) like it is done here: https://pve.proxmox.com/wiki/Recover_From_Grub_Failure
Maybe also use /dev/disk/by-path/usb-... or something to point to your USB drive as /dev/sd* is not stable.

EDIT: You also need to partition the USB drive and create the bootable ESP partition yourself first using gdisk, if you have not done so already.
EDIT2: And also run proxmox-boot-tool format before init, but make very sure that you are formatting the right partition.

So i formatted the usb with EF00 as type, then mkfs.fat -F32 /dev/sdX, then zpool import rpool -R /mnt, then mounted - proc, sys, dev, run to /mnt/(respectively), then chroot into /mnt, and i ran proxmox-boot-tool format /dev/sdX, proxmox-boot-tool init /dev/sdX and proxmox-boot-tool refresh, during the refresh command it didn't copy to the USB for some reason, after reboot i get to "divide by zero" red screen:

 

[leesteken]

So i formatted the usb with EF00 as type, then mkfs.fat -F32 /dev/sdX, then zpool import rpool -R /mnt, then mounted - proc, sys, dev, run to /mnt/(respectively), then chroot into /mnt, and i ran proxmox-boot-tool format /dev/sdX, proxmox-boot-tool init /dev/sdX and proxmox-boot-tool refresh, during the refresh command it didn't copy to the USB for some reason, after reboot i get to "divide by zero" red screen:

mkfs.fat is not necessary because proxmox-boot-tool format does that. proxmox-boot-tool refresh is not necessary because proxmox-boot-tool init should do that.
Can you check /etc/kernel/proxmox-boot-uuids? I don't really know why it would not work, sorry. Is the ESP marked as bootable and ESP?

 

[whytf]

Can you check /etc/kernel/proxmox-boot-uuids?

It contains only my hard drives, not the usb

Is the ESP marked as bootable and ESP?

Do you mean legacy boot?

Now i get this warning when running proxmox-boot-tool init /dev/sdX:

 

[leesteken]

It contains only my hard drives, not the usb

You could add the UUID of the ESP filesystem manuall, but proxmox-boot-tool should add it automatically

View attachment 60115
Do you mean legacy boot?

I don't think that's the right one. Please check with gdisk (or GParted Live ISO) if the partition is EF00. Maybe show the whole output of gdisk -l /dev/sdi.

Not sure if it relates but i get this warning when running proxmox-boot-tool init /dev/sdX:
View attachment 60117

It should not install GRUB on /dev/sdi1 but on /dev/sdi but you could (or maybe should) have a 1007KB EF02 parition before the ESP.
Mine looks like this and maybe you should mimic it?
Number Start (sector) End (sector) Size Code Name 1 34 2047 1007.0 KiB EF02 2 2048 1050623 512.0 MiB EF00

 

[whytf]

You could add the UUID of the ESP filesystem manuall, but proxmox-boot-tool should add it automatically

I don't think that's the right one. Please check with gdisk (or GParted Live ISO) if the partition is EF00. Maybe show the whole output of gdisk -l /dev/sdi.

It should not install GRUB on /dev/sdi1 but on /dev/sdi but you could (or maybe should) have a 1007KB EF02 parition before the ESP.
Mine looks like this and maybe you should mimic it?
Number Start (sector) End (sector) Size Code Name 1 34 2047 1007.0 KiB EF02 2 2048 1050623 512.0 MiB EF00

Thanks for your help all the way, this did the trick perfectly

 

[whytf]

For anyone wanting to replicate this is the process:
(Assuming an unbootable proxmox 8 install due to raid card & proxmox 8 live iso)

• Delete all existing partitions on your usb or any other device that can boot and is unaffected by raid card. (at least 1 GB space, depending on number of kernels installed)
• To replicate the setup above and have the first sector start at 34, start with gdisk /dev/sdX, press "x" for expert mode, "l" for alignment and set it to 1, press "m" to return back to normal mode
• "n" to create new partition with number 1 (sdX1), first sector: 34, last sector: 2047, Hex: EF02
• "n" to create new partition with number 2 (sdX2), first sector: 2048, last sector: 1050623, Hex: EF00 (might want to increase last sector number if planning on having more kernels)
• "w" to write changes and exit
• Now mount rpool to /mnt with zpool import rpool -R /mnt
• And mount /proc /sys /dev /run with:

mount -t proc proc /mnt/proc
mount -t sysfs sys /mnt/sys
mount -o bind /dev /mnt/dev
mount -o bind /run /mnt/run

• chroot /mnt
• proxmox-boot-tool format /dev/sdX2 --force and proxmox-boot-tool init /dev/sdX2
• Now check if the UUID of /dev/sdX2 from blkid /dev/sdX2 is inside /etc/kernel/proxmox-boot-uuids, if not try to run proxmox-boot-tool refresh
• To unmount the system folders we need to run umount for each /mnt/proc /mnt/sys /mnt/dev /mnt/run
• zpool export rpool, try -f to force the export
• Important is to export the zfs pool, because proxmox doesn't boot with force flag on pool import during boot

 

[Kyle]

Thanks for sharing your steps @whytf. They inspired me to undertake and write up: Switching from legacy boot when there is no space for ESP partition.

 

[ntblade]

Hi all,
I'm in a similar position. I installed pve onto 2 x SSDs on and add-in PCIe SATA adapter. The installer see the drives and installs in ZFS RAID 1 but my system is unable to boot from the PCI card.

I'm trying to boot from a usb stick as in the procedure above but the system just boots into what looks like a black grub splash screen that says, "Reboot into Firmware Inerface" and reboots in 10s

I'm unable to boot the system using Rescue mode. I get:

error: nosuch device: rpool
ERROR: unable to find boot disk automatically

If I then try to create a bootable USB I boot in debug mode until the installer runs to disk selection at which point I Alt+Ctr+ F3 and drop to a terminal

I'm able to partition my USB drive correctly, mount rpool and:
mount -t proc proc /mnt/proc
mount -t sysfs sys /mnt/sys
mount -o bind /dev /mnt/dev
mount -o bind /run /mnt/run
chroot /mnt
works fine.

proxmox-boot-tool format /dev/sdX2 --force works fine but proxmox-boot-tool init /dev/sdd2 finishes with output:

Failed to create EFI Boot variable entry: No such file or directory

Can anyone help please? I'm a bit stuck what to try next.

Many thanks

NTB