Proxmox 8.1 / ZFS 2.2: Docker in Privileged Container

[problame]

I'm excited about the 8.1 release because it ships ZFS 2.2.
This should enable running Docker inside privileged LXCs. [1]
(Unprivileged LXC is a different beast, let's keep this thread focussed on the privileged LXC use case)

And indeed, it's super simple:

1. Create privileged LXC on a ZFS storage
2. Enable features: nesting=1 either via GUI or in the /etc/pve/lxc/$vmid.conf
3. Inside the LXC
   1. Install Docker, e.g., using convenience script
   2. Ensure docker indeed uses overlay2 driver: docker info | grep 'Storage Driver'
   3. Run a container: `docker run hello-world`

However, you'll observe that sometimes container build / create /destroy is slow.
The data path is fast, but, these "management" operations are slow.

My assessment is that this is due to a ZFS performance bug.
I filed an issue upstream: https://github.com/openzfs/zfs/issues/15581

I'll update this thread once there are significant developments.

If anyone from Proxmox is interested in enabling this use case, I think throwing person hours behind above issue is the way to go.


[1] NB: Pre OpenZFS 2.2, the only practical option for a CoW Docker graph driver on top of ZFS was their "ZFS Storage Driver" which built on top ZFS clones.
OpenZFS 2.2 (shipped in Proxmox 8.1) adds support for whiteouts at the filesystem level, and thus `overlayfs` can now be used efficiently.

 

[leesteken]

Please be aware that Proxmox explicitly warns against running Docker in a container: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pct

 

[problame]

I’m very much aware of that “recommendation”. Let’s not debate its merits here and focus on what’s functionally feasible as of OpenZFS 2.2.

 

[giddy]

I'm excited about the 8.1 release because it ships ZFS 2.2.
This should enable running Docker inside privileged LXCs. [1]
(Unprivileged LXC is a different beast, let's keep this thread focussed on the privileged LXC use case)

And indeed, it's super simple:

1. Create privileged LXC on a ZFS storage
2. Enable features: nesting=1 either via GUI or in the /etc/pve/lxc/$vmid.conf
3. Inside the LXC
   1. Install Docker, e.g., using convenience script
   2. Ensure docker indeed uses overlay2 driver: docker info | grep 'Storage Driver'
   3. Run a container: `docker run hello-world`

However, you'll observe that sometimes container build / create /destroy is slow.
The data path is fast, but, these "management" operations are slow.

My assessment is that this is due to a ZFS performance bug.
I filed an issue upstream: https://github.com/openzfs/zfs/issues/15581

I'll update this thread once there are significant developments.

If anyone from Proxmox is interested in enabling this use case, I think throwing person hours behind above issue is the way to go.


[1] NB: Pre OpenZFS 2.2, the only practical option for a CoW Docker graph driver on top of ZFS was their "ZFS Storage Driver" which built on top ZFS clones.
OpenZFS 2.2 (shipped in Proxmox 8.1) adds support for whiteouts at the filesystem level, and thus `overlayfs` can now be used efficiently.

Hi. I followed the instructions but was unable to make it work. I get the following error -

docker: Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded: running `/usr/sbin/apparmor_parser apparmor_parser -Kr /var/lib/docker/tmp/docker-default1469373804` failed with output: apparmor_parser: Unable to replace "docker-default".  Permission denied; attempted to load a profile while confined?

I am running Proxmox VE 8.1.3 and ZFS 2.2.2-pve1