ZFS failure after ProxMox update

Ashley Nicholls

New Member
Jan 29, 2017
Aftering updating via the webUI the machine was rebooted and I got some very scary messages on the console (sorry no screenshots) with references to VERIFY3 and arc.c errors.

I USB booted nixos and could import the rpool (zpool import -o readonly=on -f -N) without issue and copy files from the filesystem. zpool showed some checksum errors and I was unwilling to import r/w for a scrub.

I then attempted to rescue boot from the ProxMox 5.4 CD but got errors regarding an incompatible compression algorithm 122.

After trawling the forums I found an entry regarding grub boot errors and one of the recomendations was to boot the installer in debug, exit the installer and attempt to import the rpool. When attempting this it seems that ZFS can't find any of my drives by ID anymore!


From dmesg/fdisk and such the disks appear to be present and named correctly

Just wondering if anyone else has seen this or had any recommendations for my next step.

Upon further investigation I can now import the rpool into the ProxMox Installer CD debug mode.

I found this post: https://github.com/zfsonlinux/zfs/issues/3145 and have confirmed that deleteing the /dev/disk/by-id/ata* nodes i can now import the zpool correctly.

Nixos still correctly imports the zpool without any hackery. Nixos is running zfs 0.7.13-1 whereas proxmox is running 0.7.13-pve1 (whether or not this is the same version I'm unsure).

Any help would be apprectiated as this does appear to be an issue with the proxmox ZFS system and I'd like to know why this happened before reinstalling my system to have it happen next update.
I have modified the initrd and editied /etc/defaults/zfs and scripts/zfs to force -the "d /dev" flag to be used when doing the import.
This now allows my system to boot correctly.

I'm hopeful the change to /etc/defaults/zfs will be enough but I'm not sure how to trigger an initrd-rebuild in proxmox.

did you try to adapt the ZPOOL_IMPORT_PATH as mentioned in the issue?
Correct, that is the modification I've made to /etc/defaults/zfs on the initrd to get it to boot.
I've now modified it on the running rootfs and would like to trigger an initrd rebuild to see if this fix will persist across kernel upgrades.

Is there a dpkg reconfigure or alternative I can use to trigger the ramdisk rebuild?
Proxmox VE has native Kernel build for ZFS do there is no dpkg.

You can run
update-initramfs -u
Thanks for that Wolfgang.

I have set the following in /etc/default/zfs:
Then rebuilt the initramfs as you recommended:
update-initramfs -u
Rebooted the system and it is now booting correctly!
Just in case anyone else needs to resuce their system from this, here is how I patched the initrd image.
I used the nixos graphical livecd as my recovery disk.

Boot nixos live, once you've logged in as root
# import our rpool
zpool import -f rpool
# change the root mountpoint and mount it
zfs set mountpoint="/mnt" rpool/ROOT/pve-1
zfs mount rpool/ROOT/pve-1
# edit the zfs defaults file
nano /mnt/etc/default/zfs # add ZPOOL_IMPORT_PATH="/dev"
# unpack initrd - check your initrd version
mkdir /mnt/tmp/initrd
cd /mnt/tmp/initrd
gzip -dc /mnt/boot/initrd.img-4.15.18-14-pve | cpio -idmv
# notice we're using etc/ not /etc/ here
nano etc/default/zfs # add ZPOOL_IMPORT_PATH="/dev"
# recompress initrd
find . | cpio -o -H newc | gzip -9 >  /mnt/boot/initrd.img-4.15.18-14-pve
# Okay let's unmount
zfs unmount rpool/ROOT/pve-1
zfs set mountpoint=/ rpool/ROOT/pve-1
# export rpool and we're done
zpool export rpool

You should now be able to reboot into your proxmox system!


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!