[SOLVED] Can't mount encrypted ZFS dataset - zfs mount fails sillently without an error

[tlaguz]

Hi,
I have problems with mounting encrypted zfs dataset after boot.
Pool is properly imported and visible in `zpool status` output.
Then I manually load keys with: `zfs load-key -a` - still no issues.
And here comes the mounting part. Dataset has `canmount=on` and `mountpoint=/mnt/ssd` properties set.
Directory /mnt/ssd is empty and is not a Proxmox storage.
Command `zfs mount pool-ssd` fails silently. Dataset is not mounted and it is confirmed by `zfs mount` and by mounted property.
What have I tried:

• removing /mnt/ssd directory
• exporting/importing pool
• changing mountpoint to other directory - this works but only till next reboot.
  Then the situation repeats and I have to change mountpoint once again.

I can't make head or tail of it. There is no error, or any other clue. I tried to import and mount this dataset
on different server - one with only debian without proxmox and it works flawlessly. However after
installing proxmox there is the same problem on both machines.

> zfs --version
zfs-0.8.4-2~bpo10+1
zfs-kmod-0.8.4-pve1

> pveversion
pve-manager/6.2-15/48bd51b6 (running kernel: 5.4.65-1-pve)

 

[fiona]

Hi,
seems like zfs utils are the one packaged by Debian instead of the ones from Proxmox, while the kernel module is from Proxmox. Maybe that mismatch is causing the strange behavior. Could you post the output of apt show zfsutils-linux? How did you configure your apt sources?

 

[tlaguz]

Hi Fabian, thanks for the reply!
Indeed `zfsutils-linux` seems to be installed from debian repository:

> apt show zfsutils-linux
Package: zfsutils-linux
Version: 0.8.4-2~bpo10+1
Priority: optional
Section: contrib/admin
Source: zfs-linux
Maintainer: Debian ZFS on Linux maintainers <pkg-zfsonlinux-devel@alioth-lists.debian.net>
Installed-Size: 1,285 kB
Provides: zfsutils

I followed this tutorial in order to install Debian on ZFS: https://openzfs.github.io/openzfs-docs/Getting Started/Debian/Debian Buster Root on ZFS.html
Then I simply installed proxmox on it.

> cat /etc/apt/sources.list
deb http://deb.debian.org/debian buster main contrib
deb-src http://deb.debian.org/debian buster main contrib

> cat /etc/apt/sources.list.d/buster-backports.list
deb http://deb.debian.org/debian buster-backports main contrib
deb-src http://deb.debian.org/debian buster-backports main contrib

> cat /etc/apt/sources.list.d/pve-install-repo.list
deb http://download.proxmox.com/debian/pve buster pve-no-subscription

> cat /etc/apt/sources.list.d/pve-enterprise.list
#deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise

> cat /etc/apt/preferences.d/90_zfs
Package: libnvpair1linux libuutil1linux libzfs2linux libzfslinux-dev libzpool2linux python3-pyzfs pyzfs-doc spl spl-dkms zfs-dkms zfs-dracut zfs-initramfs zfs-test zfsutils-linux zfsutils-linux-dev zfs-zed
Pin: release n=buster-backports
Pin-Priority: 990

Now that I see the problem I guess removing `/etc/apt/preferences.d/90_zfs` should fix at least version mismatch. I will try this when I be able to reboot the machine and will see if this solves the problem.

 

[tlaguz]

I've restarted the server and confirm that version mismatch was the problem. After removing `90_zfs` and running `apt upgrade && apt autoremove` no abnormal behavior detected!

 

[fiona]

Glad you were able to solve the problem. Could you please mark the thread as [SOLVED]? This helps others with similar problems to find solutions more easily.