Cannot import zpool by-id - bug specific to pve's zfsutils-linux pkg

[lampbearer]

On a fresh install of Proxmox on baremetal with RAID1/zfs, proxmox makes a very poor choice - it builds the RAID array using /dev/sdx notation (going totally against everything recommended by zfs). For many reasons /dev/sdx is a very BAD idea since it doesn't take much rebooting or messing around on most systems to see the specific drives jump to different sdx labels depending upon whether flash drives are inserted at time of boot, the speed of SAS controllers, etc.

So, I tried the reasonable thing to fix this -- booted into the proxmox installer in debug mode out to a command prompt and tried to import the zpool as by-id and re-export it so that on the next boot proxmox gets it right.

Flat out - nothing remotely related to /dev/disk/by-id works at all even with 0.7.4-pve-bpo9 of zfsutils-linux.
Here are the commands:
zpool import -d /dev/disk/by-id rpool (the error message is "no such pool or dataset")
Trust me, the pool exists. The error message is bogus.
zpool import rpool works perfectly fine.

So now we move to an attempted workaround:
zpool detach rpool sdb2
zpool attach rpool sda2 /dev/disk/by-id/mydevice-part2
Same error message "no such pool or dataset"

Tried some MORE messing around:
uninstall zfsutils-linux
try using the debian repositories (missing symbols - so pve has done something strange)
Try to fetch the symbols (after adding pve into the sources) - no dice, not found
try to install zfs-initramfs or zfs-dkms, no dice.

This is EXTREMELY repeatable. Can someone please fix this.

 

[wolfgang]

Hi,

first do not compare zfs on bsd with zfs on linux.
They are not the same.

Also if you like to import dev/by-id you only have to uncommand the line
"ZPOOL_IMPORT_PATH="/dev/disk/by-vdev:/dev/disk/by-id""
in /etc/default/zfs
Update grub.

 

[lampbearer]

Hi,

first do not compare zfs on bsd with zfs on linux.
They are not the same.

Thank you for the response. I'm not sure I understand this comment. I never referenced BSD and understand that Proxmox rides on Debian not BSD. I'm guessing that you are implying is that /dev/sdx is acceptable in the proxmox environment due to different behavior of differently-compiled and sourced code. But I can say with absolute certainty that allocation letters do change when usb flash devices are plugged in, drives are moved, etc. Could you clarify, please? Is there some way that /dev/sdx in proxmox properly follows devices always and avoids race conditions from hardware devices (even modern ones)? I first tried a set-up with 3 flash drives in a 3-way mirror and could not boot twice without some drive being offline and resilvering.

With 20 drives, there are other reasons to follow a better referencing scheme, but I am certain there is a point to be made regarding the design.

I did try uncommenting the path line, but I did not update grub so I appreciate that suggestion. I also have since found that the problem is in using the path not the disk reference - /dev/disk/by-id/mydrive-part2. Apparently dropping the path and party and just using mydrive works and it seems as if full-disk is required even when slices do in fact involve partitions. ZFS seems to find the correct partition regardless and you cannot use the -part2 even if you originally only gave zfs a partition and not a full disk. The reason for the latter is the need to create and use an EFI partition which debian doesn't yet fully support properly. I find this behavior strange, but it did allow a workaround by detach as sdx and attach as mydrive (no paths, no partitions).

Any additional clarification on your comment is most welcome.

 

[wolfgang]

Normally udev zfs rules should keep the correct order.
/dev/disk/* are only symlinks to /dev/sdx so it is better use it direct.
It is true if you have special udev rules the correct order can disturb.
I have test it here with our Testlab with usb flash drives but can't reproduce it.
May be you have installed specials programs what change the default udev rules?