Grub Rescue ZFS Errors on Proxmox 5.3

R

redmop

Well-Known Member
Feb 24, 2015
121
2
58
Fresh Proxmox 5.3 install and updated.

After I've moved my data over and rebooted a couple more times, it started giving me this error right before the grub rescue> prompt "error: compression algorithm inherit not supported"

If I "ls (hd0,gpt3)/ROOT@/, it gives me the following error: "error: compression algorithm 119 not supported"

4 disk raidz with compression=on and changed to compesssion=lz4 after this happened.

If I boot pmagic, zfs mounts just fine and I can chroot and read everything. I've reinstalled zfs, zfs-initramfs, grub*,, updated initramfs and grub, and ran grub-install. (https://forum.proxmox.com/threads/need-to-regenerate-grub-at-each-reboot.44810/#post-215439)

Any ideas on how I can get this going again?
 
Dell Poweredge T130 with no raid card. Proxmox 5.3 fresh install via the installler fully updated raid z with compression=on. Ran on 4.x for years.
 
Just tried reproducing the setup here - (RAIDZ1 rpool with compression on, rpool/ROOT/pve-1 having compression inherited) and it boots...

which grub version do you have installed? (2.02-pve6 here)
 
2.02-pve6
It booted a couple times before this happened. It fully installed and patched.

I've ran grub-install several times. I've force reinstalled grub*, I've regenerated initramfs (I don't think that will help anyway)
 
* you ran `grub-install` on all disks that are part of the RAIDZ?
* could you please post the output of:
** `zpool status`
** `zfs list`
** `zfs get all rpool/ROOT/pve-1` (and all its ancestors for the compression)
** `zpool history`


As for a fix/workaround - you could try to `zfs send/recv` your root dataset (first to some external drive and then back to a new dataset, which has compressions=lz4 explicitly , and afterwards rename it to the original pve-1 name)
 
Yes, grub-install on all the disks. It also did that when I reinstalled grub.

root@PartedMagic:~# zpool status
pool: rpool
state: ONLINE
scan: scrub repaired 0B in 0h0m with 0 errors on Wed Jan 2 21:18:11 2019
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
wwn-0x6668794066075013120x-part3 ONLINE 0 0 0
wwn-0x13675550682808209408x-part3 ONLINE 0 0 0
wwn-0x13202954171141214208x-part3 ONLINE 0 0 0
wwn-0x13669317444016164864x-part3 ONLINE 0 0 0

root@PartedMagic:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 6.32T 3.89T 151K /rpool
rpool/ROOT 2.05G 3.89T 140K /rpool/ROOT
rpool/ROOT/pve-1 2.05G 3.89T 2.05G /
rpool/borgtest 140K 3.89T 140K /rpool/borgtest
rpool/data 140K 3.89T 140K /rpool/data
rpool/migrate-tmp 6.28T 3.89T 151K /rpool/migrate-tmp
rpool/migrate-tmp/proxmox3 1.39T 3.89T 128K /rpool/migrate-tmp/proxmox3
rpool/migrate-tmp/proxmox3/vms 1.39T 3.89T 140K /rpool/migrate-tmp/proxmox3/vms
rpool/migrate-tmp/proxmox3/vms/old 128K 3.89T 128K /rpool/migrate-tmp/proxmox3/vms/old
rpool/migrate-tmp/proxmox3/vms/pve-config 47.0M 3.89T 587K /rpool/migrate-tmp/proxmox3/vms/pve-config
rpool/migrate-tmp/proxmox3/vms/sync-15m 6.45G 3.89T 128K /rpool/migrate-tmp/proxmox3/vms/sync-15m
rpool/migrate-tmp/proxmox3/vms/sync-15m/vm-202-disk-1 6.45G 3.89T 6.45G -
rpool/migrate-tmp/proxmox3/vms/sync-1d 128K 3.89T 128K /rpool/migrate-tmp/proxmox3/vms/sync-1d
rpool/migrate-tmp/proxmox3/vms/sync-1h 128K 3.89T 128K /rpool/migrate-tmp/proxmox3/vms/sync-1h
rpool/migrate-tmp/proxmox3/vms/sync-custom 36.9G 3.89T 128K /rpool/migrate-tmp/proxmox3/vms/sync-custom
rpool/migrate-tmp/proxmox3/vms/sync-custom/vm-100-disk-1 36.9G 3.89T 36.9G -
rpool/migrate-tmp/proxmox3/vms/vm-102-disk-1 16.5G 3.89T 4.68G -
rpool/migrate-tmp/proxmox3/vms/vm-103-disk-1 30.9G 3.89T 7.97G -
rpool/migrate-tmp/proxmox3/vms/vm-103-disk-2 1.24T 3.89T 1.11T -
rpool/migrate-tmp/proxmox3/vms/vm-105-disk-1 7.09G 3.89T 2.18G -
rpool/migrate-tmp/proxmox3/vms/vm-106-disk-1 21.1G 3.89T 11.3G -
rpool/migrate-tmp/proxmox3/vms/vm-112-disk-1 8.39G 3.89T 4.39G -
rpool/migrate-tmp/proxmox3/vms/vm-130-disk-1 20.5G 3.89T 13.9G -
rpool/migrate-tmp/proxmox3/vms/vm-202-disk-1 11.5G 3.89T 11.5G -
rpool/migrate-tmp/proxmox4 4.89T 3.89T 128K /rpool/migrate-tmp/proxmox4
rpool/migrate-tmp/proxmox4/vms 4.89T 3.89T 140K /rpool/migrate-tmp/proxmox4/vms
rpool/migrate-tmp/proxmox4/vms/old 110G 3.89T 128K /rpool/migrate-tmp/proxmox4/vms/old
rpool/migrate-tmp/proxmox4/vms/old/vm-129-disk-1.bak 20.9G 3.89T 7.31G -
rpool/migrate-tmp/proxmox4/vms/old/vm-129-disk-2.bak 89.2G 3.89T 638G -
rpool/migrate-tmp/proxmox4/vms/pve-config 53.5M 3.89T 587K /rpool/migrate-tmp/proxmox4/vms/pve-config
rpool/migrate-tmp/proxmox4/vms/sync-15m 128K 3.89T 128K /rpool/migrate-tmp/proxmox4/vms/sync-15m
rpool/migrate-tmp/proxmox4/vms/sync-1d 128K 3.89T 128K /rpool/migrate-tmp/proxmox4/vms/sync-1d
rpool/migrate-tmp/proxmox4/vms/sync-1h 128K 3.89T 128K /rpool/migrate-tmp/proxmox4/vms/sync-1h
rpool/migrate-tmp/proxmox4/vms/sync-custom 128K 3.89T 128K /rpool/migrate-tmp/proxmox4/vms/sync-custom
rpool/migrate-tmp/proxmox4/vms/vm-117-disk-1 19.1G 3.89T 2.72G -
rpool/migrate-tmp/proxmox4/vms/vm-118-disk-1 2.24G 3.89T 2.24G -
rpool/migrate-tmp/proxmox4/vms/vm-126-disk-1 3.71G 3.89T 2.25G -
rpool/migrate-tmp/proxmox4/vms/vm-126-disk-2 81.4K 3.89T 81.4K -
rpool/migrate-tmp/proxmox4/vms/vm-129-disk-1 760G 3.89T 3.97G -
rpool/migrate-tmp/proxmox4/vms/vm-129-disk-2 3.03T 3.89T 788G -
rpool/migrate-tmp/proxmox4/vms/vm-202-disk-1 6.45G 3.89T 6.45G -
rpool/migrate-tmp/proxmox4/vms/vm-209-disk-1 8.21G 3.89T 8.21G -
rpool/migrate-tmp/proxmox4/vms/vm-211-disk-1 65.5G 3.89T 65.5G -
rpool/migrate-tmp/proxmox4/vms/vm-219-disk-1 30.4G 3.89T 3.83G -
rpool/migrate-tmp/proxmox4/vms/vm-219-disk-2 894G 3.89T 654G -
rpool/migrate-tmp/proxmox4/vms/vm-300-disk-1 3.70G 3.89T 2.25G -

root@PartedMagic:~# zfs get all rpool/ROOT/pve-1
NAME PROPERTY VALUE SOURCE
rpool/ROOT/pve-1 type filesystem -
rpool/ROOT/pve-1 creation Wed Jan 2 20:16 2019 -
rpool/ROOT/pve-1 used 2.05G -
rpool/ROOT/pve-1 available 3.89T -
rpool/ROOT/pve-1 referenced 2.05G -
rpool/ROOT/pve-1 compressratio 2.03x -
rpool/ROOT/pve-1 mounted no -
rpool/ROOT/pve-1 quota none default
rpool/ROOT/pve-1 reservation none default
rpool/ROOT/pve-1 recordsize 128K default
rpool/ROOT/pve-1 mountpoint / local
rpool/ROOT/pve-1 sharenfs off default
rpool/ROOT/pve-1 checksum on default
rpool/ROOT/pve-1 compression lz4 inherited from rpool
rpool/ROOT/pve-1 atime off inherited from rpool
rpool/ROOT/pve-1 devices on default
rpool/ROOT/pve-1 exec on default
rpool/ROOT/pve-1 setuid on default
rpool/ROOT/pve-1 readonly off default
rpool/ROOT/pve-1 zoned off default
rpool/ROOT/pve-1 snapdir hidden default
rpool/ROOT/pve-1 aclinherit restricted default
rpool/ROOT/pve-1 createtxg 10 -
rpool/ROOT/pve-1 canmount on default
rpool/ROOT/pve-1 xattr on default
rpool/ROOT/pve-1 copies 1 default
rpool/ROOT/pve-1 version 5 -
rpool/ROOT/pve-1 utf8only off -
rpool/ROOT/pve-1 normalization none -
rpool/ROOT/pve-1 casesensitivity sensitive -
rpool/ROOT/pve-1 vscan off default
rpool/ROOT/pve-1 nbmand off default
rpool/ROOT/pve-1 sharesmb off default
rpool/ROOT/pve-1 refquota none default
rpool/ROOT/pve-1 refreservation none default
rpool/ROOT/pve-1 guid 7448222599031128143 -
rpool/ROOT/pve-1 primarycache all default
rpool/ROOT/pve-1 secondarycache all default
rpool/ROOT/pve-1 usedbysnapshots 0B -
rpool/ROOT/pve-1 usedbydataset 2.05G -
rpool/ROOT/pve-1 usedbychildren 0B -
rpool/ROOT/pve-1 usedbyrefreservation 0B -
rpool/ROOT/pve-1 logbias latency default
rpool/ROOT/pve-1 dedup off default
rpool/ROOT/pve-1 mlslabel none default
rpool/ROOT/pve-1 sync standard inherited from rpool
rpool/ROOT/pve-1 dnodesize legacy default
rpool/ROOT/pve-1 refcompressratio 2.03x -
rpool/ROOT/pve-1 written 2.05G -
rpool/ROOT/pve-1 logicalused 3.86G -
rpool/ROOT/pve-1 logicalreferenced 3.86G -
rpool/ROOT/pve-1 volmode default default
rpool/ROOT/pve-1 filesystem_limit none default
rpool/ROOT/pve-1 snapshot_limit none default
rpool/ROOT/pve-1 filesystem_count none default
rpool/ROOT/pve-1 snapshot_count none default
rpool/ROOT/pve-1 snapdev hidden default
rpool/ROOT/pve-1 acltype off default
rpool/ROOT/pve-1 context none default
rpool/ROOT/pve-1 fscontext none default
rpool/ROOT/pve-1 defcontext none default
rpool/ROOT/pve-1 rootcontext none default
rpool/ROOT/pve-1 relatime off default
rpool/ROOT/pve-1 redundant_metadata all default
rpool/ROOT/pve-1 overlay off default

root@PartedMagic:~# zpool history
History for 'rpool':
2019-01-02.20:16:20 zpool create -f -o cachefile=none -o ashift=12 rpool raidz1 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3
2019-01-02.20:16:20 zfs create rpool/ROOT
2019-01-02.20:16:21 zfs create rpool/data
2019-01-02.20:16:21 zfs create rpool/ROOT/pve-1
2019-01-02.20:16:21 zfs set atime=off rpool
2019-01-02.20:16:21 zfs set compression=on rpool
2019-01-02.20:16:26 zfs set sync=disabled rpool
2019-01-02.20:29:04 zfs set sync=standard rpool
2019-01-02.20:29:04 zfs set mountpoint=/ rpool/ROOT/pve-1
2019-01-02.20:29:05 zpool set bootfs=rpool/ROOT/pve-1 rpool
2019-01-02.20:29:05 zpool export rpool
2019-01-02.20:47:25 zpool import -N rpool
2019-01-02.20:53:21 zpool import -N rpool
2019-01-04.11:47:09 zfs rename rpool/migrate-tmp/proxmox4/vms/sync-15m/vm-117-disk-1 rpool/migrate-tmp/proxmox4/vms/vm-117-disk-1
2019-01-04.11:47:20 zfs rename rpool/migrate-tmp/proxmox4/vms/sync-15m/vm-118-disk-1 rpool/migrate-tmp/proxmox4/vms/vm-118-disk-1
2019-01-04.11:47:32 zfs rename rpool/migrate-tmp/proxmox4/vms/sync-15m/vm-202-disk-1 rpool/migrate-tmp/proxmox4/vms/vm-202-disk-1
2019-01-04.11:47:40 zfs rename rpool/migrate-tmp/proxmox4/vms/sync-15m/vm-209-disk-1 rpool/migrate-tmp/proxmox4/vms/vm-209-disk-1
2019-01-04.11:48:33 zfs rename rpool/migrate-tmp/proxmox4/vms/sync-1d/vm-126-disk-1 rpool/migrate-tmp/proxmox4/vms/vm-126-disk-1
2019-01-04.11:48:38 zfs rename rpool/migrate-tmp/proxmox4/vms/sync-1d/vm-126-disk-2 rpool/migrate-tmp/proxmox4/vms/vm-126-disk-2
2019-01-04.11:48:53 zfs rename rpool/migrate-tmp/proxmox4/vms/sync-1d/vm-219-disk-1 rpool/migrate-tmp/proxmox4/vms/vm-219-disk-1
2019-01-04.11:48:59 zfs rename rpool/migrate-tmp/proxmox4/vms/sync-1d/vm-219-disk-2 rpool/migrate-tmp/proxmox4/vms/vm-219-disk-2
2019-01-04.11:49:11 zfs rename rpool/migrate-tmp/proxmox4/vms/sync-1d/vm-300-disk-1 rpool/migrate-tmp/proxmox4/vms/vm-300-disk-1
2019-01-04.11:49:33 zfs rename rpool/migrate-tmp/proxmox4/vms/sync-1h/vm-129-disk-1 rpool/migrate-tmp/proxmox4/vms/vm-129-disk-1
2019-01-04.11:49:57 zfs rename rpool/migrate-tmp/proxmox4/vms/sync-1h/vm-129-disk-2 rpool/migrate-tmp/proxmox4/vms/vm-129-disk-2
2019-01-04.11:50:16 zfs rename rpool/migrate-tmp/proxmox4/vms/sync-custom/vm-211-disk-1 rpool/migrate-tmp/proxmox4/vms/vm-211-disk-1
2019-01-04.19:41:31 zpool import -d /dev/disk/by-id/ -f -aN
2019-01-04.20:09:35 zpool export rpool
2019-01-04.20:09:59 zpool export rpool
2019-01-04.20:25:44 zpool export rpool
2019-01-04.20:25:59 zpool import rpool -R /mnt
2019-01-04.20:26:05 zpool export rpool
2019-01-04.20:52:22 zpool import -d /dev/disk/by-id/ -f -aN
2019-01-04.20:52:40 zpool export rpool
2019-01-04.20:53:01 zpool import rpool -R /mnt
2019-01-04.21:11:23 zfs set compression=lz4 rpool
2019-01-04.21:11:37 zpool export rpool
2019-01-04.21:52:24 zpool import -d /dev/disk/by-id/ -f -aN
2019-01-04.22:01:34 zpool export rpool
2019-01-04.22:01:55 zpool import rpool -R /mnt
2019-01-04.22:10:22 zpool export rpool
2019-01-04.22:11:16 zpool import rpool -R /mnt
2019-01-05.00:12:41 zpool import -d /dev/disk/by-id/ -f -aN
2019-01-05.17:23:46 zpool export rpool
2019-01-05.17:24:01 zpool import rpool -R /mnt
2019-01-05.17:32:09 zpool export rpool
2019-01-08.18:20:53 zpool import -d /dev/disk/by-id/ -f -aN
 
I removed the creates/rollbacks/etc from the send receives from "zfs history". They are just putting the VM data back in place.
 
You must log in or register to reply here.
Top Bottom