Podman on LXC with ZFS backed volume and Overlay

[h0tw1r3]

Now that ZFS with overlay support is available in ZFS 2.2 and Proxmox 8, I decided to test it out with podman in an lxc container.
Out of the box, it didn't work. Giving the standard message from podman that overlays are not supported on a ZFS file system. FWIW docker gives the same message, but I haven't tested / don't know if this method works.

Basically All you need to do is create a mount wrapper and configure podman to use it.

I'm using a Debian 12 container with the latest Podman (4.8) from http://downloadcontent.opensuse.org/repositories/home:/alvistack/Debian_12/
I think the stock 4.3 may work also, but I haven't tested it.

create /usr/local/bin/overlayzfsmount

#!/bin/sh
exec /bin/mount -t overlay overlay "$@"

make sure it's executable: chmod +x /usr/local/bin/overlayzfsmount

edit /etc/containers/storage.conf, mine only contains:

[storage]
driver = "overlay"
runroot = "/run/containers/storage"
graphroot = "/var/lib/containers/storage"

[storage.options]
pull_options = {enable_partial_images = "false", use_hard_links = "false", ostree_repos=""}
mount_program = "/usr/local/bin/overlayzfsmount"

[storage.options.overlay]
mountopt = "nodev"

The important setting is `mount_program`.

Thats it! You should now be able to start containers without fuse.

root@docker2:~# docker image pull docker.io/library/debian:12
Trying to pull docker.io/library/debian:12...
Getting image source signatures
Copying blob sha256:bc0734b949dcdcabe5bfdf0c8b9f44491e0fce04cb10c9c6e76282b9f6abdf01
Copying config sha256:2a033a8c63712da54b5a516f5d69d41606cfb5c4ce9aa1690ee55fc4f9babb92
Writing manifest to image destination
2a033a8c63712da54b5a516f5d69d41606cfb5c4ce9aa1690ee55fc4f9babb92
root@docker2:~# docker run -it --rm --name test docker.io/library/debian:12
root@3b38b7aacd55:/#

 

[el_pedr0]

very timely info for me. Thank you.
But possible typo in your storage.conf
Should "/usr/local/bin/overzfsmount" be "/usr/local/bin/overlayzfsmount"?

 

[h0tw1r3]

very timely info for me. Thank you.
But possible typo in your storage.conf
Should "/usr/local/bin/overzfsmount" be "/usr/local/bin/overlayzfsmount"?

fixed, thanks!

 

[el_pedr0]

I've just tried this with podman 4.3.1 and it seems to be working in rootless.

If you're going to do rootless podman, remember to edit ~/.config/containers/storage.conf rather than /etc/containers/storage.conf. And you might want to use the rootless_storage_path directive instead of graphroot

I lost a lot of time trying to get fuse to work. So this is very welcome.

 

[hyu]

What should be displayed by podman info under store before and after this change?

I am not using a Debian LXC but NixOS where the configuration is a bit different, and mine displays this:

store:
  configFile: /home/sysadmin/.config/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 1
    stopped: 8
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/sysadmin/.local/share/containers/storage
  graphRootAllocated: 8589934592
  graphRootUsed: 3984457728
  graphStatus:
    Backing Filesystem: zfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 6
  runRoot: /run/user/1001/containers
  transientStore: false
  volumePath: /home/sysadmin/.local/share/containers/storage/volumes

Interested to know if this is good or not.

 

[LangDuTienTu]

Sorry but I don't understand your purpose
I have podman rootless inside LXC and it works with ZFS, for both mounpoint a directory or a datasheet

 

[ajeun]

For anyone else that runs into this. I followed the instructions but found that it didn't work for all containers. Running debian worked fine, but trying to run portainer failed with an error like

mount: /var/lib/containers/storage/overlay/b3463251ee4e78750cea908510c8973c06e48eec24dee2bb2fafa9063763da47/merged: wrong fs type, bad option, bad superblock on overlay, missing codepage or helper program, or other error.

Following https://github.com/containers/podman/issues/20324#issuecomment-2589290167 I modified /usr/local/bin/overlayzfsmount

#!/bin/sh
LIBMOUNT_FORCE_MOUNT2=always exec /bin/mount -t overlay overlay "$@"

and that worked for me.

 

[LangDuTienTu]

For anyone else that runs into this. I followed the instructions but found that it didn't work for all containers. Running debian worked fine, but trying to run portainer failed with an error like

mount: /var/lib/containers/storage/overlay/b3463251ee4e78750cea908510c8973c06e48eec24dee2bb2fafa9063763da47/merged: wrong fs type, bad option, bad superblock on overlay, missing codepage or helper program, or other error.

Following https://github.com/containers/podman/issues/20324#issuecomment-2589290167 I modified /usr/local/bin/overlayzfsmount

#!/bin/sh
LIBMOUNT_FORCE_MOUNT2=always exec /bin/mount -t overlay overlay "$@"

and that worked for me.

Im curious, why need this.

I have podman inside LXC and it works out of the box without any modification