ZFS [BUG] in latest 0.8.4-pve1

[Republicus]

After an update to ZFS 0.8.4-pve now two storage systems with encrypted datasets will not mount child datasets.

ZFS is treating child/sub datasets as directories. Both systems have an 'encrypted_data' dataset with underlying datasets inheriting encryption details.

root@node05:~# zfs load-key -r raid10bak/encrypted_data
Enter passphrase for 'raid10bak/encrypted_data': 
1 / 1 key(s) successfully loaded

root@node05:~# zfs mount raid10bak/encrypted_data
cannot mount '/raid10bak/encrypted_data': directory is not empty

This happened today as both systems were updated.

The only way Ive managed to mount the child datasets is:

root@node05:~# zfs set overlay=on raid10bak
root@node05:~# zfs mount -a

These datasets are empty, excepting other datasets, which should (and did) mount properly before this update.

root@node05:~# zfs list
NAME                                   USED  AVAIL     REFER  MOUNTPOINT
raid10bak                             2.21T  8.32T      296G  /raid10bak
raid10bak/encrypted_data              1.92T  8.32T      240K  /raid10bak/encrypted_data
raid10bak/encrypted_data/pve          1.92T  8.32T      200K  /raid10bak/encrypted_data/pve
raid10bak/encrypted_data/pve/backups  1.92T  8.32T     1.92T  /raid10bak/encrypted_data/pve/backups

 

[Stoiko Ivanov]

cannot mount '/raid10bak/encrypted_data': directory is not empty

it would be interesting to know, what kind of content was inside the mountpoint
could you unmount the datasets below `/raid10bak/encrypted_data`
and paste the output of:
`find /raid10bak/encrypted_data`

please post also the contents of your '/etc/pve/storage.cfg'

EDIT: added the request for storage.cfg

Thanks!

 

[Republicus]

Sorry for the delay. I ran into some serious work-stoppage with the latest kernel.

As you requested @Stoiko Ivanov

Showing zfs filesystems/datasets

root@node05:~# zfs list
NAME                                   USED  AVAIL     REFER  MOUNTPOINT
raid10bak                             2.21T  8.32T      296G  /raid10bak
raid10bak/encrypted_data              1.92T  8.32T      240K  /raid10bak/encrypted_data
raid10bak/encrypted_data/pve          1.92T  8.32T      200K  /raid10bak/encrypted_data/pve
raid10bak/encrypted_data/pve/backups  1.92T  8.32T     1.92T  /raid10bak/encrypted_data/pve/backups

storage.cfg

root@node05:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content images,backup
        maxfiles 32
        shared 0

nfs: nfs-storage
        export /raid50vol/pve/nfs-storage
        path /mnt/pve/nfs-storage
        server 10.12.12.170
        content images,rootdir
        nodes node03,node02,node01,node05,node04
        options rw,rsize=65536,wsize=65536,hard,proto=rdma,port=20049,intr,_netdev,auto,noatime,sync,vers=3

nfs: nfs-async
        export /raid50vol/pve/nfs-storage
        path /mnt/pve/nfs-async
        server 10.12.12.170
        content vztmpl,rootdir,images
        nodes node01,node02,node03,node04,node05
        options rw,rsize=65536,wsize=65536,hard,proto=rdma,port=20049,intr,_netdev,auto,noatime,async,vers=3

nfs: lxc-templates
        export /raid50vol/pve/lxc-templates
        path /mnt/pve/lxc-templates
        server 10.12.12.170
        content vztmpl
        nodes node03,node02,node04,node05,node01
        options rw,rsize=65536,wsize=65536,hard,proto=rdma,port=20049,intr,_netdev,auto,noatime,async,vers=3

nfs: iso-images
        export /raid50vol/pve/iso-images
        path /mnt/pve/iso-images
        server 10.12.12.170
        content iso
        nodes node01,node02,node03,node04,node05
        options rw,rsize=65536,wsize=65536,hard,proto=rdma,port=20049,intr,_netdev,auto,noatime,async,vers=3

zfspool: raid10bak
        pool raid10bak
        content images,rootdir
        mountpoint /raid10bak
        nodes node05

nfs: nfs-encrypt
        export /raid50vol/encrypted_data/pve
        path /mnt/pve/nfs-encrypt
        server 10.12.12.170
        content images,rootdir
        options rw,rsize=65536,wsize=65536,hard,proto=rdma,port=20049,intr,_netdev,auto,noatime,async,vers=3

nfs: backup-storage
        export /raid10bak/encrypted_data/pve/backups
        path /mnt/pve/backup-storage
        server 10.12.12.175
        content backup
        maxfiles 5
        nodes node05,node04,node02,node03,node01
        options rw,rsize=65536,wsize=65536,hard,proto=rdma,port=20049,intr,_netdev,auto,noatime,async,vers=3

Splitting this message because of character limit (why?)

 

[Republicus]

root@node05:~# find /raid10bak/encrypted_data
/raid10bak/encrypted_data
/raid10bak/encrypted_data/pve
/raid10bak/encrypted_data/pve/backups
/raid10bak/encrypted_data/pve/backups/dump
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-101-2020_05_16-05_34_43.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-101-2020_05_16-05_34_43.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71250-2020_05_16-05_38_14.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-80011-2020_05_23-05_30_02.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71101-2020_05_16-06_07_02.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-70002-2020_05_23-05_39_31.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-80010-2020_05_23-07_37_09.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71162-2020_05_23-05_48_22.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71010-2020_05_16-06_03_12.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-71160-2020_05_16-05_56_44.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-100-2020_05_24-14_35_52.tmp
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-100-2020_05_24-14_35_52.tmp/qemu-server.conf
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-70001-2020_05_24-14_41_57.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-70100-2020_05_23-05_39_29.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-10010-2020_05_16-05_30_02.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71100-2020_05_16-06_03_48.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-70001-2020_05_16-05_30_02.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-70150-2020_05_23-08_23_48.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-102-2020_05_16-05_30_02.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-20002-2020_05_23-05_30_02.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-80010-2020_05_23-07_37_09.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71250-2020_05_16-05_38_14.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-105-2020_05_23-08_15_52.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-80100-2020_05_23-05_32_23.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-80100-2020_05_23-05_32_23.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71104-2020_05_16-06_14_05.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-71160-2020_05_23-06_01_05.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-70001-2020_05_24-15_11_11.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-70001-2020_05_23-05_30_03.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-70002-2020_05_16-05_45_30.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-100-2020_05_24-14_14_48.vma.zst
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71172-2020_05_23-05_51_11.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-100-2020_05_23-05_30_02.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-20004-2020_05_16-05_51_33.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-102-2020_05_23-05_30_02.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-80160-2020_05_23-05_49_37.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-70010-2020_05_23-08_23_30.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-80100-2020_05_16-05_56_25.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-70002-2020_05_23-05_39_31.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-80102-2020_05_16-06_22_15.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-70100-2020_05_16-05_36_37.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-103-2020_05_23-05_46_06.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-11001-2020_05_23-08_18_03.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-100-2020_05_23-05_30_02.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-70011-2020_05_23-07_33_08.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-70105-2020_05_23-05_40_56.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71172-2020_05_23-05_51_11.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71250-2020_05_23-05_52_53.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-105-2020_05_16-16_47_23.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71103-2020_05_16-06_13_24.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-20002-2020_05_16-05_30_02.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71100-2020_05_23-05_47_55.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71102-2020_05_16-06_12_30.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-103-2020_05_16-05_42_50.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-10020-2020_05_16-05_52_15.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-103-2020_05_16-05_42_50.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71101-2020_05_23-05_52_17.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-70001-2020_05_24-14_41_57.vma.zst
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-20003-2020_05_16-05_40_41.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-10020-2020_05_16-05_52_15.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71200-2020_05_23-05_52_15.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71010-2020_05_16-06_03_12.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-80100-2020_05_16-05_56_25.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-70002-2020_05_16-05_45_30.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-70110-2020_05_23-05_42_14.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-10010-2020_05_23-07_50_10.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71102-2020_05_16-06_12_30.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-102-2020_05_23-05_30_02.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-20004-2020_05_23-07_33_04.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-20003-2020_05_16-05_40_41.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-101-2020_05_23-05_37_59.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-70105-2020_05_23-05_40_56.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-100-2020_05_24-14_35_52.vma.dat
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-20005-2020_05_16-05_58_56.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-71160-2020_05_16-05_56_44.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-80102-2020_05_16-06_22_15.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-20005-2020_05_16-05_58_56.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-20002-2020_05_16-05_30_02.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71151-2020_05_23-05_43_21.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-80011-2020_05_23-05_30_02.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71151-2020_05_23-16_02_21.tar.zst
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-10020-2020_05_23-08_13_04.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-70011-2020_05_23-07_33_08.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-20004-2020_05_16-05_51_33.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71162-2020_05_23-05_48_22.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-11001-2020_05_16-05_53_05.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-103-2020_05_23-05_46_06.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-20002-2020_05_23-05_30_02.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-101-2020_05_23-05_37_59.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-70107-2020_05_23-05_41_46.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-70001-2020_05_16-05_30_02.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71103-2020_05_16-06_13_24.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-70110-2020_05_23-05_42_14.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-70100-2020_05_23-05_39_29.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-20003-2020_05_23-05_37_02.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-100-2020_05_16-05_30_02.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-102-2020_05_16-05_30_02.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-100-2020_05_24-14_14_48.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-20003-2020_05_23-05_37_02.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-100-2020_05_16-05_30_02.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71151-2020_05_23-16_02_21.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-71160-2020_05_23-18_29_53.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-70107-2020_05_23-05_41_46.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71104-2020_05_16-06_14_05.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-70001-2020_05_24-15_11_11.vma.zst
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71200-2020_05_23-05_52_15.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-105-2020_05_23-08_15_52.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-70001-2020_05_23-05_30_03.vma.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-70010-2020_05_23-08_23_30.tar.lzo
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-70100-2020_05_16-05_36_37.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71151-2020_05_23-05_43_21.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-qemu-80160-2020_05_16-06_01_42.vma.dat
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71250-2020_05_23-05_52_53.log
/raid10bak/encrypted_data/pve/backups/dump/vzdump-lxc-71150-2020_05_16-06_14_46.log