Error mounting ZFS dataset into container

[wgallt]

I have two unprivileged lxc containers with users mapped to the host. I have a zfs pool called data with subdirectories. Container 100 mounts one of the subdirectories fine. Container 102, using the exact same config just using a different subdirectory fails to mount and the container will not start. These containers are nearly identical. The error doesn't give me enough information and I'm not sure how to debug this properly. Any help is appreciated. The error starting container is:

safe_mount: 1220 Invalid argument - Failed to mount "/data/downloads" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/data/downloads"
mount_entry: 2439 Invalid argument - Failed to mount "/data/downloads" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/data/downloads"
lxc_setup: 4412 Failed to setup mount entries

Here are the configs for both.
100.conf:

arch: amd64
cores: 2
features: nesting=1
hostname: nextcloud
memory: 1024
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.68.1,hwaddr=C2:F7:0F:84:14:BE,ip=192.168.68.31/24,type=veth
ostype: debian
rootfs: app:subvol-100-disk-0,size=8G
swap: 512
unprivileged: 1
lxc.idmap: u 0 100000 33
lxc.idmap: g 0 100000 33
lxc.idmap: u 33 33 1
lxc.idmap: g 33 33 1
lxc.idmap: u 34 100034 65502
lxc.idmap: g 34 100034 65502
lxc.mount.entry: /data/nextcloud data/nextcloud none bind,create=dir 0 0

102.conf

arch: amd64
cores: 1
features: nesting=1
hostname: qbittorrent
memory: 1024
net0: name=eth0,bridge=vmbr1,firewall=1,gw=192.168.68.1,hwaddr=BE:81:DB:01:F1:24,ip=192.168.68.32/24,type=veth
ostype: debian
rootfs: app:subvol-102-disk-0,size=8G
swap: 512
unprivileged: 1
lxc.idmap: u 0 100000 1005
lxc.idmap: g 0 100000 1005
lxc.idmap: u 1005 1005 1
lxc.idmap: g 1005 1005 1
lxc.idmap: u 1006 101006 64530
lxc.idmap: g 1006 101006 64530
lxc.mount.entry: /data/downloads data/downloads none bind,create=dir 0 0

subuid:

root:100000:65536
root:1005:1
root:33:1

subgid:

root:100000:65536
root:1005:1
root:33:1

Host directory permissions:
drwxrwxr-x 5 app app 5 Aug 21 2022 downloads

drwxrwx--- 13 www-data www-data 18 Apr 2 12:52 nextcloud

 

[fweber]

Hi, there is an older thread with a similar error [1] -- there, the culprit was that another filesystem was mounted within the directory (in your case, /data/downloads). Could you check if this is the case here, e.g. using findmnt?

[1]: https://forum.proxmox.com/threads/102687/#post-506535

 

[wgallt]

I ran findmnt and this was the result:

├─/data                          data                      zfs        rw,noatime,xattr,posixacl
│ ├─/data/share                  data/share                zfs        rw,noatime,xattr,posixacl
│ ├─/data/downloads              data/downloads            zfs        rw,noatime,xattr,posixacl
│ │ ├─/data/downloads/watched    data/downloads/watched    zfs        rw,noatime,xattr,posixacl
│ │ ├─/data/downloads/incomplete data/downloads/incomplete zfs        rw,noatime,xattr,posixacl
│ │ └─/data/downloads/complete   data/downloads/complete   zfs        rw,noatime,xattr,posixacl
│ └─/data/nextcloud              data/nextcloud            zfs        rw,noatime,xattr,posixacl

Would it be because watched, incomplete and complete are nested datasets? This is a zfs pool migrated from TrueNAS Scale. I wonder if i delete these and just create them as directories if that would take care of it.

 

[wgallt]

I was able to mount the path by setting the source to the nested dataset directly. Thanks for pointing me in the right direction.