[SOLVED] ZFS Grub install faliour upgrade 5.4 to 6

[djcmike]

Good morning.

I am in the process of upgrading from 5.4 to 6, but I ran in to the following problem and have not rebooted yet.

grub-install: warning: your core.img is unusually large. It won't fit in the embedding area.
grub-install: error: filesystem `zfs' doesn't support blocklists.

It is a bit of a concern.

The system is running ZFS on two SSDs in mirror.

Hope you can cast some light on the issue

Yours,
Mike Hansen

 

[Stoiko Ivanov]

which version of grub is installed? (PVE ships its own) - please post the output of `pveversion -v`
I hope this helps!

 

[djcmike]

Hi Stoiko.

Thank you for your quick responce.

I rebooted the system and nothing went wrong, but I still have issues with /dev/sda and grub.

The output of 'pveversion -v' is:

Linux pioneer 5.0.21-1-pve #1 SMP PVE 5.0.21-1 (Tue, 20 Aug 2019 17:16:32 +0200) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Aug 26 22:32:27 2019 from 172.16.1.198
pioneer:~# pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.21-1-pve)
pve-manager: 6.0-6 (running version: 6.0-6/c71f879f)
pve-kernel-5.0: 6.0-7
pve-kernel-helper: 6.0-7
pve-kernel-4.15: 5.4-8
pve-kernel-5.0.21-1-pve: 5.0.21-1
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-9-pve: 4.15.18-30
ceph-fuse: 12.2.12-pve1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.11-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-4
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-7
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-64
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-7
pve-cluster: 6.0-5
pve-container: 3.0-5
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-2
pve-qemu-kvm: 4.0.0-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve2

 

[Stoiko Ivanov]

sorry - i missed that the grub version is not included in pveversion output
please post the output of `dpkg -l |grep grub`

 

[djcmike]

Ah yeah I didn't see it either. Well here goes:

ii  grub-common                          2.02+dfsg1-18-pve1              amd64        GRand Unified Bootloader (common files)
ii  grub-efi-amd64-bin                   2.02+dfsg1-18-pve1              amd64        GRand Unified Bootloader, version 2 (EFI-AMD64 modules)
ii  grub-efi-ia32-bin                    2.02+dfsg1-18-pve1              amd64        GRand Unified Bootloader, version 2 (EFI-IA32 modules)
ii  grub-pc                              2.02+dfsg1-18-pve1              amd64        GRand Unified Bootloader, version 2 (PC/BIOS version)
ii  grub-pc-bin                          2.02+dfsg1-18-pve1              amd64        GRand Unified Bootloader, version 2 (PC/BIOS modules)
ii  grub2-common                         2.02+dfsg1-18-pve1              amd64        GRand Unified Bootloader (common files for version 2)

 

[Stoiko Ivanov]

* Ok - the version is the one from PVE...
* How did you setup the system in the first place? (pve-installer?, on top of debian? , via debootstrap?)

could you post the output of `lsblk` and `gdisk -p /dev/sda` (and /dev/sdb or whatever the other device in the zpool mirror is)
Thanks!

 

[djcmike]

I installed it from the ISO from proxmox.com
'lsblk' shows:

NAME     MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda        8:0    0 465,8G  0 disk
├─sda1     8:1    0  1007K  0 part
├─sda2     8:2    0   512M  0 part
└─sda3     8:3    0 465,3G  0 part
sdb        8:16   0 465,8G  0 disk
├─sdb1     8:17   0  1007K  0 part
├─sdb2     8:18   0   512M  0 part
└─sdb3     8:19   0 465,3G  0 part
sdc        8:32   0   5,5T  0 disk
├─sdc1     8:33   0   5,5T  0 part
└─sdc9     8:41   0     8M  0 part
sdd        8:48   0   5,5T  0 disk
├─sdd1     8:49   0   5,5T  0 part
└─sdd9     8:57   0     8M  0 part
sde        8:64   0   5,5T  0 disk
├─sde1     8:65   0   5,5T  0 part
└─sde9     8:73   0     8M  0 part
sr0       11:0    1  1024M  0 rom 
zd0      230:0    0    16G  0 disk
├─zd0p1  230:1    0    14G  0 part
├─zd0p2  230:2    0     1K  0 part
└─zd0p5  230:5    0     2G  0 part
zd16     230:16   0   350G  0 disk
├─zd16p1 230:17   0   342G  0 part
├─zd16p2 230:18   0     1K  0 part
└─zd16p5 230:21   0     8G  0 part
zd32     230:32   0  16,5G  0 disk
zd48     230:48   0     8T  0 disk
└─zd48p1 230:49   0     8T  0 part

And gdisk -p /dev/sda and sdb is:

Partition table scan:

  MBR: MBR only

  BSD: not present

  APM: not present

  GPT: not present





***************************************************************

Found invalid GPT and valid MBR; converting MBR to GPT format

in memory. 

***************************************************************



Disk /dev/sda: 976773168 sectors, 465.8 GiB

Model: WDC WDS500G2B0A-

Sector size (logical/physical): 512/512 bytes

Disk identifier (GUID): 59A79559-9E9D-476B-880B-8C5033236186

Partition table holds up to 128 entries

Main partition table begins at sector 2 and ends at sector 33

First usable sector is 34, last usable sector is 976773134

Partitions will be aligned on 8-sector boundaries

Total free space is 0 sectors (0 bytes)



Number  Start (sector)    End (sector)  Size       Code  Name

   1              34            2047   1007.0 KiB  EF00  EFI System

   2            2048         1050623   512.0 MiB   EF00  EFI System

   3         1050624       976773134   465.3 GiB   BF00  Solaris root

GPT fdisk (gdisk) version 1.0.3



Partition table scan:

  MBR: protective

  BSD: not present

  APM: not present

  GPT: present



Found valid GPT with protective MBR; using GPT.

Disk /dev/sdb: 976773168 sectors, 465.8 GiB

Model: WDC WDS500G2B0A-

Sector size (logical/physical): 512/512 bytes

Disk identifier (GUID): 90C56681-3D7A-4AAF-922B-80D23869D0FD

Partition table holds up to 128 entries

Main partition table begins at sector 2 and ends at sector 33

First usable sector is 34, last usable sector is 976773134

Partitions will be aligned on 8-sector boundaries

Total free space is 0 sectors (0 bytes)



Number  Start (sector)    End (sector)  Size       Code  Name

   1              34            2047   1007.0 KiB  EF02  

   2            2048         1050623   512.0 MiB   EF00  

   3         1050624       976773134   465.3 GiB   BF01

 

[Stoiko Ivanov]

Found invalid GPT and valid MBR; converting MBR to GPT format in memory.

hmm - that is odd - seems the gpt-partition table of sda got messed up somehow (expecting that it was ok (like sdb) at some point)..

How did you setup this system? pve-iso (and which version?)

You could try to replicated the partition-table from sdb to sda (and randomize the GUIDS afterwards) - then try to install grub again

Hope this helps!

 

[djcmike]

Hi Stoiko

Again thanks for your help on this one
I installed proxmox ve 5.2 from the official download site, originally.

Ok grub and zfs isn't my strongest side, so how would I go about restoring from sdb?

Cheers,
Mike

 

[Stoiko Ivanov]

The `sgdisk` commands and grub-install from https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Replacing_a_failed_disk_in_the_root_pool should work
(don't offline the disk in the zpool and don't replace it
source = /dev/sdb
target = /dev/sda
and grub-install to /dev/sda

As always when operating with disks and partitions - make sure you have a working backup!!

 

[djcmike]

Hi again Stolko. Thanks that worked perfectly! Thanks for your excellent help.

Cheers,
Mike

 

[Stoiko Ivanov]

Glad you resolved your issue! Please mark the thread as 'SOLVED' so that others know what to expect..
Thanks!