problems with proxmox-boot-tool

[Thomas Stone]

Hello!

I recently updated from Proxmox VE 6.x to 7.1, on a system which was set up maybe on 5.x with root on ZFS. I read about the issue with booting zfs and read the web page about it at https://pve.proxmox.com/wiki/ZFS:_Switch_Legacy-Boot_to_Proxmox_Boot_Tool. I seem to have exactly the setup up described there, which should be updated.

I also want to move my boot disk from one 512GB nvme drive to another one, so I have the second one blank and ready to play with. /dev/nvme0n1 is the currently-running Proxmox rpool disk and /dev/nvme1n1 is the disk I'm trying to first set up correctly with proxmox-boot-tool and then move my rpool to that disk (not sure how I'll do that yet).

Here is the disk layout on the currently running (old) disk:

Using /dev/nvme0n1
(parted) print                                                           
Model: Samsung SSD 970 PRO 512GB (nvme)
Disk /dev/nvme0n1: 512GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      17.4kB  1049kB  1031kB                     bios_grub
 2      1049kB  512GB   512GB   zfs          zfs
 9      512GB   512GB   8389kB

So on the new disk I created gpt partition table with partition 1 starting at 1MB size 512MB for the vfat booter, and partition 2 for the rest of the disk. Then I run the boot tool to format:

# proxmox-boot-tool format /dev/nvme1n1p1 --force
UUID="781D-2BD9" SIZE="536870912" FSTYPE="vfat" PARTTYPE="0fc63daf-8483-4772-8e79-3d69d8477de4" PKNAME="nvme1n1" MOUNTPOINT=""
Setting partition type of '/dev/nvme1n1p1' to 'c12a7328-f81f-11d2-ba4b-00a0c93ec93b'..
The operation has completed successfully.
Calling 'udevadm settle'..
Formatting '/dev/nvme1n1p1' as vfat..
mkfs.fat 4.2 (2021-01-31)
Done.

and then init

# proxmox-boot-tool init /dev/nvme1n1p1
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
UUID="8B61-4540" SIZE="536870912" FSTYPE="vfat" PARTTYPE="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" PKNAME="nvme1n1" MOUNTPOINT=""
Mounting '/dev/nvme1n1p1' on '/var/tmp/espmounts/8B61-4540'.
Installing grub i386-pc target..
Installing for i386-pc platform.
grub-install.real: warning: this GPT partition label contains no BIOS Boot Partition; embedding won't be possible.
grub-install.real: warning: Embedding is not possible.  GRUB can only be installed in this setup by using blocklists.  However, blocklists are UNRELIABLE and their use is discouraged..
grub-install.real: error: will not proceed with blocklists.

looks bad, so I check

# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
E: /etc/kernel/proxmox-boot-uuids does not exist.

no go. look at partition table

# parted /dev/nvme1n1
GNU Parted 3.4
Using /dev/nvme1n1
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Model: Samsung SSD 970 PRO 512GB (nvme)
Disk /dev/nvme1n1: 512GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End    Size   File system  Name     Flags
 1      1049kB  538MB  537MB  fat32        primary  boot, esp
 2      538MB   512GB  512GB               primary

What am I doing wrong? The instruction lead me to believe that doing this will allow me to boot the current (old) boot disk, though eventually I will want it to boot rpool from partition 2 on the same disk and I'll remove the old disk. Thanks.

 

[leesteken]

I think you need a bios_grub partition from the (unaligned) start of the drive up to the (currently) first partition for GRUB. I don't know how to do that with parted, but gdisk (using the expert menu to change alignment first) can create a EF02 partition from sector 34 to 2047.

 

[Thomas Stone]

I guess I was hoping somehow magically grub didn't need any other space, but it makes sense. Not only that, IT WORKS! Thank you avw!

# parted /dev/nvme1n1
GNU Parted 3.4
Using /dev/nvme1n1
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                           
Model: Samsung SSD 970 PRO 512GB (nvme)
Disk /dev/nvme1n1: 512GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 3      17.4kB  1049kB  1031kB               primary  bios_grub
 1      1049kB  538MB   537MB   fat32        primary  boot, esp
 2      538MB   512GB   512GB                primary

# proxmox-boot-tool init /dev/nvme1n1p1
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
UUID="8B61-4540" SIZE="536870912" FSTYPE="vfat" PARTTYPE="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" PKNAME="nvme1n1" MOUNTPOINT=""
Mounting '/dev/nvme1n1p1' on '/var/tmp/espmounts/8B61-4540'.
Installing grub i386-pc target..
Installing for i386-pc platform.
Installation finished. No error reported.
Unmounting '/dev/nvme1n1p1'.
Adding '/dev/nvme1n1p1' to list of synced ESPs..
Refreshing kernels and initrds..
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/8B61-4540
    Copying kernel 5.13.19-2-pve
    Copying kernel 5.4.151-1-pve
Generating grub configuration file ...
Found background image: /usr/share/images/desktop-base/desktop-grub.png
Found linux image: /boot/vmlinuz-5.13.19-2-pve
Found initrd image: /boot/initrd.img-5.13.19-2-pve
Found linux image: /boot/vmlinuz-5.4.151-1-pve
Found initrd image: /boot/initrd.img-5.4.151-1-pve
done

Now... to clone my rpool. Hmm.

 

[Thomas Stone]

If it were an MBR disk I would not need that... but GPT for the win.

 

[Thomas Stone]

... for those playing along, even thought the new partition for zfs was a bit smaller, I was able to attach it to rpool so I briefly had a mirror (resilvered super fast), then detach the original boot drive. Shut down, pull original drive, reboot. Had to fix boot priorities in my UEFI/Bios but then it booted and so far seems happy.