PVE 4.1 - Grub rescue shell after add ZFS cache & log

[elurex]

Hi all,

I have reinstall dozen times and I cannot figure out how PVE's grub works. I always assume grub is install on the first 2 disk of /dev/sda and /dev/sdb.

When I install my pve 4.1, I use following ZFS pool configuration


after I add two intel 750 nvme sdd as long and cache, after reboot I can no longer boot into pve.


I have tried to manually grub-install

grub-install /dev/sda
grub-install /dev/sda
update-grub
update-initramfs -u

Still After reboot, I can only get into grub-rescue shell



I tried to insmod zfs insmod part_gpt then ls (hd0) , ls (hd0,gpt1) ls (hd0, gpt2) ls (hd0, gpt9) they all return unknow filesystem.

Please help.

I have updated pve 4.1 to latest update and still the same results

 

[fabian]

Please post the resulting grub configuration ("/boot/grub/grub.cfg") and the content of "/etc/default/grub"

 

[elurex]

root@san:~# uname -r
4.4.6-1-pve
root@san:~# pveversion -v
proxmox-ve: 4.1-48 (running kernel: 4.4.6-1-pve)
pve-manager: 4.1-34 (running version: 4.1-34/8887b0fd)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.2.6-1-pve: 4.2.6-36
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-39
qemu-server: 4.0-72
pve-firmware: 1.1-8
libpve-common-perl: 4.0-59
libpve-access-control: 4.0-16
libpve-storage-perl: 4.0-50
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-13
pve-container: 1.0-61
pve-firewall: 2.0-25
pve-ha-manager: 1.0-28
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-7
lxcfs: 2.0.0-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve9~jessie
root@san:~# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/pve-1 boot=zfs"

# Disable os-prober, it might add menu entries for each guest
# root FS on a local partition
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

grub.cfg uploaded as grub.txt

these are cfg BEFORE ADDING zfs cache and log

 

[fabian]

the config that is not working (i.e., after adding the cache and log vdevs and updating the grub config) would be more interesting

 

[elurex]

root@san:/dev# zpool add rpool cache /dev/nvme0n1p1 -f
root@san:/dev# zpool add rpool log /dev/nvme1n1p1 -f
root@san:/dev# zpool status
  pool: rpool
state: ONLINE
  scan: none requested
config:

        NAME         STATE     READ WRITE CKSUM
        rpool        ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            sda2     ONLINE       0     0     0
            sdb2     ONLINE       0     0     0
          mirror-1   ONLINE       0     0     0
            sdc      ONLINE       0     0     0
            sdd      ONLINE       0     0     0
          mirror-2   ONLINE       0     0     0
            sde      ONLINE       0     0     0
            sdf      ONLINE       0     0     0
          mirror-3   ONLINE       0     0     0
            sdg      ONLINE       0     0     0
            sdh      ONLINE       0     0     0
        logs
          nvme1n1p1  ONLINE       0     0     0
        cache
          nvme0n1p1  ONLINE       0     0     0

errors: No known data errors
root@san:/dev# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#  info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/pve-1 boot=zfs"

# Disable os-prober, it might add menu entries for each guest
# root FS on a local partition
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

grub.cfg uploaded as grub.txt

 

[fabian]

did you run "update-grub2" to regenerate the configuration? it is identical

 

[elurex]

after adding cache and log.... booting video

 

[elurex]

did you run "update-grub2" to regenerate the configuration? it is identical

yes I did run update-grub... and I suspect pve4.1 might use /dev/nvme0n1 as grub booting drive instead of /dev/sda

 

[fabian]

I could not reproduce this here, but unfortunately we only have limited NVME testing hardware available.

Since NVME support is fairly new and we want to rule out bugs on our side, we would allow you (this one time, under these special circumstances) to open a free support ticket at https://my.proxmox.com - if you are willing to give us temporary access to the server and iKVM console. Maybe we can find the root of this issue by looking around in the Grub rescue shell.

If you decide to take this route, please put a reference to this forum thread into the ticket so that our support staff can assign it to me or one of my colleagues directly.

 

[elurex]

ticket opened... please gimme some time to create necessary connection profile and account access. currently that server holds no data... everytihing on the hard drive can be wiped, partition or reinstalled

I have tried reinstall that server so many time trying to solve the problem and I have used wipefs -a /dev/sdX and /dev/nvme0n1 many times already

 

[sbielski]

I have the same problem here, running Promox 4.3 with 5x 4TB drives and a 256GB SSD, after adding 10GB as log and the rest as cache the new installed server does not boot anymore (stopping at grub rescue with the same error "no such device"). Any solution available?

 

[elurex]

i would suggest to use usb thumb disk for pve os drive to avoid this zfs +zil+cache+grub2 issue...

 

[fabian]

I have the same problem here, running Promox 4.3 with 5x 4TB drives and a 256GB SSD, after adding 10GB as log and the rest as cache the new installed server does not boot anymore (stopping at grub rescue with the same error "no such device"). Any solution available?

SSD or NVME? in the latter case I would suggest checking for firmware updates (both for the NVME disk and the motherboard)