Sometimes to get the ZFS root file system the way we want requires an install in legacy mode and conversion to uEFI mode, perhaps because of:
If we zpool attach rpool sda2 nvme0n1 to add the desired root drive there won't be a partition to write the BIOS boot. So lets match the existing partition table:
normally the efi partition is formatted vfat32, but our partition is too tiny so vfat16 will do:
So we have grubx64.efi staged, but because we booted legacy, there are no uEFI variables and that's why efibootmgr failed to register the boot entry.
to boot in uEFI let's boot the USB flash drive image file from rEFInd. reboot, turn off bios legacy boot and turn on uEFI boot and select the USB device with the rEFInd image. It will show both the disks with grubx64.efi and will boot either. Now we can add the boot variables to bios:
now it will boot from either drive and we can remove the rEFInd stick and the install drive and expand the volume.
yes I know its not a good idea to mirror a slow USB drive with a fast nvme ssd.
if we wanted to install the legacy grub, overwrite the vfat partition with:
grub-install -d /usr/lib/grub/i386-pc /dev/nvme0n1
What did I miss?
- Installer creates an undesirable partition table
- Installer can't find cdrom image after grub2 load
- hardware requires uEFI bios to initialize (emmc)
- black boot screen perhaps because of uEFI grub2 not recognizing newer CPUs
Code:
root@pve:~# fdisk -l
Disk /dev/sda: 55.9 GiB, 60022480896 bytes, 117231408 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 10C3E371-D891-423D-B341-FD3BB74DD114
Device Start End Sectors Size Type
/dev/sda1 34 2047 2014 1007K BIOS boot
/dev/sda2 2048 117214989 117212942 55.9G Solaris /usr & Apple ZFS
/dev/sda9 117214990 117231374 16385 8M Solaris reserved 1
Code:
root@pve:~# sgdisk -Z -a1 -n1:34:2047 -t1:EF02 -c1:GRUB -a9 -n9:-8M:0 -t9:bf07 -c9:Reserved -a2 -n2:2047:0 -t2:bf01 -c2:ZFS /dev/nvme0n1
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Setting name!
partNum is 0
REALLY setting name!
Information: Moved requested sector from 976756750 to 976756743 in
order to align on 9-sector boundaries.
Setting name!
partNum is 8
REALLY setting name!
Information: Moved requested sector from 2047 to 2048 in
order to align on 2-sector boundaries.
Setting name!
partNum is 1
REALLY setting name!
The operation has completed successfully.
root@pve:~# zpool attach -f rpool sda2 /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_500GB_S466NX0K901726H-part2
root@pve:~# sgdisk -Z -a1 -n1:34:2047 -t1:EF02 -c1:GRUB -a9 -n9:-8M:0 -t9:bf07 -c9:Reserved -a2 -n2:2047:0 -t2:bf01 -c2:ZFS /dev/sdd
***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory.
***************************************************************
Warning! Secondary partition table overlaps the last partition by
33 blocks!
You will need to delete this partition or resize it in another utility.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Setting name!
partNum is 0
REALLY setting name!
Information: Moved requested sector from 976756750 to 976756743 in
order to align on 9-sector boundaries.
Setting name!
partNum is 8
REALLY setting name!
Information: Moved requested sector from 2047 to 2048 in
order to align on 2-sector boundaries.
Setting name!
partNum is 1
REALLY setting name!
The operation has completed successfully.
root@pve:~# zpool attach -f rpool sda2 /dev/disk/by-id/usb-ZALMAN_External_HDD_______XX00000001-0\:0-part2
root@pve:~# zpool status
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sat Nov 10 19:31:58 2018
677M scanned out of 870M at 26.0M/s, 0h0m to go
676M resilvered, 77.87% done
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda2 ONLINE 0 0 0
nvme-Samsung_SSD_970_EVO_500GB_S466NX0K901726H-part2 ONLINE 0 0 0
usb-ZALMAN_External_HDD_______XX00000001-0:0-part2 ONLINE 0 0 0 (resilvering)
errors: No known data errors
root@pve:~# zpool iostat -v
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 870M 54.7G 22 34 938K 829K
mirror 870M 54.7G 55 84 2.24M 1.98M
sda2 - - 10 8 639K 168K
nvme-Samsung_SSD_970_EVO_500GB_S466NX0K901726H-part2 - - 29 47 733K 862K
usb-ZALMAN_External_HDD_______XX00000001-0:0-part2 - - 0 214 260 9.41M
-------------------------------------------------------- ----- ----- ----- ----- ----- -----
root@pve:~# fdisk -l
Disk /dev/nvme0n1: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5336CF94-FBD1-411B-A7CC-C359A8638784
Device Start End Sectors Size Type
/dev/nvme0n1p1 34 2047 2014 1007K BIOS boot
/dev/nvme0n1p2 2048 976756742 976754695 465.8G Solaris /usr & Apple ZFS
/dev/nvme0n1p9 976756743 976773134 16392 8M Solaris reserved 1
Disk /dev/sda: 55.9 GiB, 60022480896 bytes, 117231408 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 10C3E371-D891-423D-B341-FD3BB74DD114
Device Start End Sectors Size Type
/dev/sda1 34 2047 2014 1007K BIOS boot
/dev/sda2 2048 117214989 117212942 55.9G Solaris /usr & Apple ZFS
/dev/sda9 117214990 117231374 16385 8M Solaris reserved 1
Disk /dev/sdd: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: DC51A57B-EFFE-4957-A4F5-26351FEACD0C
Device Start End Sectors Size Type
/dev/sdd1 34 2047 2014 1007K BIOS boot
/dev/sdd2 2048 976756742 976754695 465.8G Solaris /usr & Apple ZFS
/dev/sdd9 976756743 976773134 16392 8M Solaris reserved 1
root@pve:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 55.9G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 55.9G 0 part
└─sda9 8:9 0 8M 0 part
sdd 8:48 0 465.8G 0 disk
├─sdd1 8:49 0 1007K 0 part
├─sdd2 8:50 0 465.8G 0 part
└─sdd9 8:57 0 8M 0 part
mmcblk0 179:0 0 29.7G 0 disk
├─mmcblk0p1 179:1 0 1007K 0 part
├─mmcblk0p2 179:2 0 29.7G 0 part
└─mmcblk0p9 259:4 0 8M 0 part
nvme0n1 259:0 0 465.8G 0 disk
├─nvme0n1p1 259:5 0 1007K 0 part
├─nvme0n1p2 259:6 0 465.8G 0 part
└─nvme0n1p9 259:7 0 8M 0 part
Code:
root@pve:~# mkdosfs -n BOOT_EFI /dev/nvme0n1p1
mkfs.fat 4.1 (2017-01-24)
root@pve:~# mount /dev/disk/by-label/BOOT_EFI /boot/efi
root@pve:~# df /boot/efi
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/nvme0n1p1 988 184 804 19% /boot/efi
root@pve:~# grub-install -d /usr/lib/grub/x86_64-efi /dev/nvme0n1
Installing for x86_64-efi platform.
EFI variables are not supported on this system.
EFI variables are not supported on this system.
grub-install: error: efibootmgr failed to register the boot entry: No such file or directory.
root@pve:~# ls -l /boot/efi/EFI/proxmox/grubx64.efi
-rwxr-xr-x 1 root root 182784 Nov 10 19:38 /boot/efi/EFI/proxmox/grubx64.efi
root@pve:~# umount /boot/efi
root@pve:~# mkdosfs -n BOOT_EFI /dev/sdd1
mkfs.fat 4.1 (2017-01-24)
root@pve:~# mount /dev/disk/by-label/BOOT_EFI /boot/efi
root@pve:~# df /boot/efi
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdd1 988 0 988 0% /boot/efi
root@pve:~# grub-install -d /usr/lib/grub/x86_64-efi /dev/sdd1
Installing for x86_64-efi platform.
EFI variables are not supported on this system.
EFI variables are not supported on this system.
grub-install: error: efibootmgr failed to register the boot entry: No such file or directory.
root@pve:~# ls -l /boot/efi/EFI/proxmox/grubx64.efi
-rwxr-xr-x 1 root root 182784 Nov 10 19:55 /boot/efi/EFI/proxmox/grubx64.efi
root@pve:~# update-initramfs -u # update /etc/zfs/zpool.cache
update-initramfs: Generating /boot/initrd.img-4.15.17-1-pve
root@pve:~# update-grub # update /boot/grub/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.17-1-pve
Found initrd image: /boot/initrd.img-4.15.17-1-pve
Found memtest86+ image: /ROOT/pve-1@/boot/memtest86+.bin
Found memtest86+ multiboot image: /ROOT/pve-1@/boot/memtest86+_multiboot.bin
Adding boot menu entry for EFI firmware configuration
done
Code:
root@pve:~# efibootmgr -v
EFI variables are not supported on this system.
root@pve:~# ls /sys/firmware/efi
ls: cannot access '/sys/firmware/efi': No such file or directory
Code:
root@pve:~# ls /sys/firmware/efi
config_table efivars esrt fw_platform_size fw_vendor runtime runtime-mapsystabvars
root@pve:~# efibootmgr -v
BootCurrent: 0003
Timeout: 1 seconds
BootOrder: 0002,0003,0004,0005,0001,0000
Boot0000* nvme0n1VenHw(99e275e7-75a0-4b37-a2e6-c5385e6c00cb)
Boot0001* mmcblk0VenHw(99e275e7-75a0-4b37-a2e6-c5385e6c00cb)
Boot0002* UEFI : USB : ZALMAN External HDD : PART 0 : OS BootloaderPciRoot(0x0)/Pci(0x14,0x0)/USB(12,0)/HD(1,GPT,f8b891b7-cb78-45e4-9775-97ddf1735a8c,0x22,0x7de)..BO
Boot0003* UEFI : USB : SanDisk : PART 0 : OS BootloaderPciRoot(0x0)/Pci(0x14,0x0)/USB(14,0)/USB(3,0)/HD(1,GPT,30d07d2b-cd7f-44d6-aa6f-db03ee9f3911,0x800,0x2f9f)..BO
Boot0004* UEFI : USB : MX MXUB3SESU-16G 1.00 : PART 1 : OS BootloaderPciRoot(0x0)/Pci(0x14,0x0)/USB(15,0)/HD(2,GPT,eafc5def-6eeb-415e-94d3-48ddd19f63a7,0x214,0x1680)..BO
Boot0005* UEFI : Samsung SSD 970 EVO 500GB : PART 0 : OS BootloaderPciRoot(0x0)/Pci(0x1d,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-25-38-59-81-B0-EF-DE)/HD(1,GPT,fbfd0bd9-face-43a9-bef8-13f145193a06,0x22,0x7de)..BO
Boot0007* USB : SanDisk : PART 0 : Boot DriveBBS(HD,,0x0)..BO
root@pve:~# efibootmgr -qBb 0
root@pve:~# efibootmgr -qBb 1
root@pve:~# efibootmgr -qc -d /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_500GB_S466NX0K901726H -p 1 -L Samsung970EVO -l "\EFI\proxmox\grubx64.efi"
root@pve:~# efibootmgr -qc -d /dev/disk/by-id/usb-ZALMAN_External_HDD_______XX00000001-0\:0 -p 1 -L ZALMAN -l "\EFI\proxmox\grubx64.efi"
root@pve:~# efibootmgr -o 0,1
BootCurrent: 0003
Timeout: 1 seconds
BootOrder: 0000,0001
Boot0000* Samsung970EVO
Boot0001* ZALMAN
Boot0003* UEFI : USB : SanDisk : PART 0 : OS Bootloader
Boot0004* UEFI : USB : MX MXUB3SESU-16G 1.00 : PART 1 : OS Bootloader
Boot0005* UEFI : Samsung SSD 970 EVO 500GB : PART 0 : OS Bootloader
Boot0007* USB : SanDisk : PART 0 : Boot Drive
Code:
root@pve:~# zpool iostat
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
rpool 873M 54.6G 1 19 26.4K 224K
root@pve:~# zpool detach rpool sda2
root@pve:~# zpool online -e rpool nvme-Samsung_SSD_970_EVO_500GB_S466NX0K901726H-part2
root@pve:~# zpool iostat
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
rpool 873M 465G 1 19 26.4K 225Kif we wanted to install the legacy grub, overwrite the vfat partition with:
grub-install -d /usr/lib/grub/i386-pc /dev/nvme0n1
What did I miss?