Grub rescue mode after rpool upgrade

J

Jero

Member
May 20, 2016
33
2
8
43
Hi Guys,

I did a "zpool upgrade rpool" after the pve5 to 6 upgrade and now i can't boot anymore.

I get the following grub error:
error: no such device xxxxxxx
error: unknown filesystem

after chroot:

pveversion -v
Code: 
proxmox-ve: 6.0-2 (running kernel: 5.0.15-1-pve)
pve-manager: 6.0-4 (running version: 6.0-4/2a719255)
pve-kernel-5.0: 6.0-5
pve-kernel-helper: 6.0-5
pve-kernel-4.15: 5.4-6
pve-kernel-5.0.15-1-pve: 5.0.15-1
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.15.18-14-pve: 4.15.18-39
pve-kernel-4.13.13-5-pve: 4.13.13-38
pve-kernel-4.13.13-4-pve: 4.13.13-35
pve-kernel-4.13.13-3-pve: 4.13.13-34
pve-kernel-4.13.13-2-pve: 4.13.13-33
pve-kernel-4.13.4-1-pve: 4.13.4-26
pve-kernel-4.10.17-4-pve: 4.10.17-24
pve-kernel-4.10.17-2-pve: 4.10.17-20
pve-kernel-4.4.76-1-pve: 4.4.76-94
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.62-1-pve: 4.4.62-88
pve-kernel-4.4.59-1-pve: 4.4.59-87
pve-kernel-4.4.44-1-pve: 4.4.44-84
pve-kernel-4.4.35-2-pve: 4.4.35-79
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.24-1-pve: 4.4.24-72
pve-kernel-4.4.21-1-pve: 4.4.21-71
pve-kernel-4.4.19-1-pve: 4.4.19-66
pve-kernel-4.4.16-1-pve: 4.4.16-64
pve-kernel-4.4.15-1-pve: 4.4.15-60
pve-kernel-4.4.13-1-pve: 4.4.13-56
pve-kernel-4.4.10-1-pve: 4.4.10-54
pve-kernel-4.4.8-1-pve: 4.4.8-52
pve-kernel-4.4.6-1-pve: 4.4.6-48
ceph-fuse: 12.2.11+dfsg1-2.1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.10-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-2
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-5
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-61
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-5
pve-cluster: 6.0-4
pve-container: 3.0-4
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-5
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-2
pve-qemu-kvm: 4.0.0-3
pve-xtermjs: 3.13.2-1
pve-zsync: 2.0-1
qemu-server: 6.0-5
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve1

zpool status rpool
Code: 
  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 0 days 00:16:53 with 0 errors on Sun Jul 14 00:40:57 2019
config:

   NAME                                                     STATE     READ WRITE CKSUM
   rpool                                                    ONLINE       0     0     0
     mirror-0                                               ONLINE       0     0     0
       ata-Samsung_SSD_860_PRO_256GB_S42VNF0M107498Y-part2  ONLINE       0     0     0
       ata-Samsung_SSD_860_PRO_256GB_S42VNF0M107490L-part2  ONLINE       0     0     0

errors: No known data errors

zpool get all rpool
Code: 
NAME   PROPERTY                       VALUE                          SOURCE
rpool  size                           238G                           -
rpool  capacity                       34%                            -
rpool  altroot                        -                              default
rpool  health                         ONLINE                         -
rpool  guid                           6740436469329246581            -
rpool  version                        -                              default
rpool  bootfs                         rpool/ROOT/pve-1               local
rpool  delegation                     on                             default
rpool  autoreplace                    off                            default
rpool  cachefile                      -                              default
rpool  failmode                       wait                           default
rpool  listsnapshots                  off                            default
rpool  autoexpand                     on                             local
rpool  dedupditto                     0                              default
rpool  dedupratio                     1.00x                          -
rpool  free                           155G                           -
rpool  allocated                      82,7G                          -
rpool  readonly                       off                            -
rpool  ashift                         12                             local
rpool  comment                        -                              default
rpool  expandsize                     -                              -
rpool  freeing                        0                              -
rpool  fragmentation                  33%                            -
rpool  leaked                         0                              -
rpool  multihost                      off                            default
rpool  checkpoint                     -                              -
rpool  load_guid                      1297468664927249800            -
rpool  autotrim                       on                             local
rpool  feature@async_destroy          enabled                        local
rpool  feature@empty_bpobj            active                         local
rpool  feature@lz4_compress           active                         local
rpool  feature@multi_vdev_crash_dump  enabled                        local
rpool  feature@spacemap_histogram     active                         local
rpool  feature@enabled_txg            active                         local
rpool  feature@hole_birth             active                         local
rpool  feature@extensible_dataset     active                         local
rpool  feature@embedded_data          active                         local
rpool  feature@bookmarks              enabled                        local
rpool  feature@filesystem_limits      enabled                        local
rpool  feature@large_blocks           enabled                        local
rpool  feature@large_dnode            enabled                        local
rpool  feature@sha512                 enabled                        local
rpool  feature@skein                  enabled                        local
rpool  feature@edonr                  enabled                        local
rpool  feature@userobj_accounting     active                         local
rpool  feature@encryption             active                         local
rpool  feature@project_quota          active                         local
rpool  feature@device_removal         enabled                        local
rpool  feature@obsolete_counts        enabled                        local
rpool  feature@zpool_checkpoint       enabled                        local
rpool  feature@spacemap_v2            active                         local
rpool  feature@allocation_classes     enabled                        local
rpool  feature@resilver_defer         enabled                        local
rpool  feature@bookmark_v2            enabled                        local

grub-probe /
Code: 
grub-probe: error: unknown filesystem.

update-grub
Code: 
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.0.15-1-pve
Found initrd image: /boot/initrd.img-5.0.15-1-pve
/usr/sbin/grub-probe: error: unknown filesystem.
Found linux image: /boot/vmlinuz-4.15.18-18-pve
Found initrd image: /boot/initrd.img-4.15.18-18-pve
Found linux image: /boot/vmlinuz-4.15.18-14-pve
Found initrd image: /boot/initrd.img-4.15.18-14-pve
Found linux image: /boot/vmlinuz-4.13.13-5-pve
Found initrd image: /boot/initrd.img-4.13.13-5-pve
Found linux image: /boot/vmlinuz-4.13.13-4-pve
Found initrd image: /boot/initrd.img-4.13.13-4-pve
Found linux image: /boot/vmlinuz-4.13.13-3-pve
Found initrd image: /boot/initrd.img-4.13.13-3-pve
Found linux image: /boot/vmlinuz-4.13.13-2-pve
Found initrd image: /boot/initrd.img-4.13.13-2-pve
Found linux image: /boot/vmlinuz-4.13.4-1-pve
Found initrd image: /boot/initrd.img-4.13.4-1-pve
Found linux image: /boot/vmlinuz-4.10.17-4-pve
Found initrd image: /boot/initrd.img-4.10.17-4-pve
Found linux image: /boot/vmlinuz-4.10.17-2-pve
Found initrd image: /boot/initrd.img-4.10.17-2-pve
Found linux image: /boot/vmlinuz-4.4.76-1-pve
Found initrd image: /boot/initrd.img-4.4.76-1-pve
Found linux image: /boot/vmlinuz-4.4.67-1-pve
Found initrd image: /boot/initrd.img-4.4.67-1-pve
Found linux image: /boot/vmlinuz-4.4.62-1-pve
Found initrd image: /boot/initrd.img-4.4.62-1-pve
Found linux image: /boot/vmlinuz-4.4.59-1-pve
Found initrd image: /boot/initrd.img-4.4.59-1-pve
Found linux image: /boot/vmlinuz-4.4.44-1-pve
Found initrd image: /boot/initrd.img-4.4.44-1-pve
Found linux image: /boot/vmlinuz-4.4.35-2-pve
Found initrd image: /boot/initrd.img-4.4.35-2-pve
Found linux image: /boot/vmlinuz-4.4.35-1-pve
Found initrd image: /boot/initrd.img-4.4.35-1-pve
Found linux image: /boot/vmlinuz-4.4.24-1-pve
Found initrd image: /boot/initrd.img-4.4.24-1-pve
Found linux image: /boot/vmlinuz-4.4.21-1-pve
Found initrd image: /boot/initrd.img-4.4.21-1-pve
Found linux image: /boot/vmlinuz-4.4.19-1-pve
Found initrd image: /boot/initrd.img-4.4.19-1-pve
Found linux image: /boot/vmlinuz-4.4.16-1-pve
Found initrd image: /boot/initrd.img-4.4.16-1-pve
Found linux image: /boot/vmlinuz-4.4.15-1-pve
Found initrd image: /boot/initrd.img-4.4.15-1-pve
Found linux image: /boot/vmlinuz-4.4.13-1-pve
Found initrd image: /boot/initrd.img-4.4.13-1-pve
Found linux image: /boot/vmlinuz-4.4.10-1-pve
Found initrd image: /boot/initrd.img-4.4.10-1-pve
Found linux image: /boot/vmlinuz-4.4.8-1-pve
Found initrd image: /boot/initrd.img-4.4.8-1-pve
Found linux image: /boot/vmlinuz-4.4.6-1-pve
Found initrd image: /boot/initrd.img-4.4.6-1-pve
/usr/sbin/grub-probe: error: unknown filesystem.
Found memtest86+ image: /ROOT/pve-1@/boot/memtest86+.bin
Found memtest86+ multiboot image: /ROOT/pve-1@/boot/memtest86+_multiboot.bin
done

update-initramfs -u
Code: 
update-initramfs: Generating /boot/initrd.img-5.0.15-1-pve
cryptsetup: ERROR: Couldn't resolve device rpool/ROOT/pve-1
cryptsetup: WARNING: Couldn't determine root device
cryptsetup: WARNING: The initramfs image may not contain cryptsetup binaries
    nor crypto modules. If that's on purpose, you may want to uninstall the
    'cryptsetup-initramfs' package in order to disable the cryptsetup initramfs
    integration and avoid this warning.
/usr/bin/ckbcomp: Can not find file "symbols/fr-be" in any known directory

grub-install /dev/disk/by-id/ata-Samsung_SSD_860_PRO_256GB_S42VNF0M107490L
Code: 
Installing for i386-pc platform.
grub-install: error: unknown filesystem.

grub-install /dev/disk/by-id/ata-Samsung_SSD_860_PRO_256GB_S42VNF0M107498Y
Code: 
Installing for i386-pc platform.
grub-install: error: unknown filesystem.


I don't know what to do next..
 
Found it.

When you create a encrypted dataset on the rpool the feature "encryption" becomes active on the zpool. Grub is unable to detect the filesystem when that happens. Fortunately the encryption option reverts back to enabled when destroying the encrypted dataset and grub is detecting the FS as ZFS again.

It a bummer, because i was waiting on ZoL 0.8 for encrypting the LXC dataset :-(

Info from the Ach wiki:
You can create a pool with only the compatible features enabled:

# zpool create -d -o feature@allocation_classes=enabled \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@project_quota=enabled \
-o feature@resilver_defer=enabled \
-o feature@spacemap_histogram=enabled \
-o feature@spacemap_v2=enabled \
-o feature@userobj_accounting=enabled \
-o feature@zpool_checkpoint=enabled \
$POOL_NAME $VDEVS
Click to expand...

Edit: bug report on ZoL:
https://github.com/zfsonlinux/zfs/issues/7153
 
Last edited:
PVE 6 now supports a ZFS root install in UEFI mode. The good thing in that situation is that systemd-boot is used instead of GRUB.

So if you get a chance to reinstall your server switch it to UEFI mode first and these hassles caused by GRUB will be a thing of the past.
 
fabian said:
it's also mentioned in our admin guide: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#zfs_encryption
Click to expand...

Thanks, i didn't now the admin guide.

aaron said:
PVE 6 now supports a ZFS root install in UEFI mode. The good thing in that situation is that systemd-boot is used instead of GRUB.

So if you get a chance to reinstall your server switch it to UEFI mode first and these hassles caused by GRUB will be a thing of the past.
Click to expand...

Yep, reinstall it is.
 
You must log in or register to reply here.
Top Bottom