ZFS failure after ProxMox update

[Ashley Nicholls]

Aftering updating via the webUI the machine was rebooted and I got some very scary messages on the console (sorry no screenshots) with references to VERIFY3 and arc.c errors.

I USB booted nixos and could import the rpool (zpool import -o readonly=on -f -N) without issue and copy files from the filesystem. zpool showed some checksum errors and I was unwilling to import r/w for a scrub.

I then attempted to rescue boot from the ProxMox 5.4 CD but got errors regarding an incompatible compression algorithm 122.

After trawling the forums I found an entry regarding grub boot errors and one of the recomendations was to boot the installer in debug, exit the installer and attempt to import the rpool. When attempting this it seems that ZFS can't find any of my drives by ID anymore!



From dmesg/fdisk and such the disks appear to be present and named correctly


Just wondering if anyone else has seen this or had any recommendations for my next step.

Thanks!

 

[Ashley Nicholls]

Upon further investigation I can now import the rpool into the ProxMox Installer CD debug mode.

I found this post: https://github.com/zfsonlinux/zfs/issues/3145 and have confirmed that deleteing the /dev/disk/by-id/ata* nodes i can now import the zpool correctly.

Nixos still correctly imports the zpool without any hackery. Nixos is running zfs 0.7.13-1 whereas proxmox is running 0.7.13-pve1 (whether or not this is the same version I'm unsure).

Any help would be apprectiated as this does appear to be an issue with the proxmox ZFS system and I'd like to know why this happened before reinstalling my system to have it happen next update.

 

[Ashley Nicholls]

I have modified the initrd and editied /etc/defaults/zfs and scripts/zfs to force -the "d /dev" flag to be used when doing the import.
This now allows my system to boot correctly.

I'm hopeful the change to /etc/defaults/zfs will be enough but I'm not sure how to trigger an initrd-rebuild in proxmox.

 

[wolfgang]

Hi,

did you try to adapt the ZPOOL_IMPORT_PATH as mentioned in the issue?

 

[Ashley Nicholls]

Correct, that is the modification I've made to /etc/defaults/zfs on the initrd to get it to boot.
I've now modified it on the running rootfs and would like to trigger an initrd rebuild to see if this fix will persist across kernel upgrades.

Is there a dpkg reconfigure or alternative I can use to trigger the ramdisk rebuild?

 

[wolfgang]

Proxmox VE has native Kernel build for ZFS do there is no dpkg.

You can run

update-initramfs -u

 

[Ashley Nicholls]

Thanks for that Wolfgang.

I have set the following in /etc/default/zfs:

ZPOOL_IMPORT_PATH="/dev"

Then rebuilt the initramfs as you recommended:

update-initramfs -u

Rebooted the system and it is now booting correctly!

 

[Ashley Nicholls]

Just in case anyone else needs to resuce their system from this, here is how I patched the initrd image.
I used the nixos graphical livecd as my recovery disk.

Boot nixos live, once you've logged in as root

# import our rpool
zpool import -f rpool
# change the root mountpoint and mount it
zfs set mountpoint="/mnt" rpool/ROOT/pve-1
zfs mount rpool/ROOT/pve-1
# edit the zfs defaults file
nano /mnt/etc/default/zfs # add ZPOOL_IMPORT_PATH="/dev"
# unpack initrd - check your initrd version
mkdir /mnt/tmp/initrd
cd /mnt/tmp/initrd
gzip -dc /mnt/boot/initrd.img-4.15.18-14-pve | cpio -idmv
# notice we're using etc/ not /etc/ here
nano etc/default/zfs # add ZPOOL_IMPORT_PATH="/dev"
# recompress initrd
find . | cpio -o -H newc | gzip -9 >  /mnt/boot/initrd.img-4.15.18-14-pve
# Okay let's unmount
cd
zfs unmount rpool/ROOT/pve-1
zfs set mountpoint=/ rpool/ROOT/pve-1
# export rpool and we're done
zpool export rpool

You should now be able to reboot into your proxmox system!