5.3: Docker on LXC on ZFS


Active Member
Nov 1, 2018

I'm toying with running Docker inside a CentOS 7 LXC on a Proxmox whose storage is ZFS. The container has:

$ cat /etc/pve/lxc/209.conf
features: nesting=1

and is a privileged container.

For starters, the only storage driver that allows me to start the Docker service is vfs. ZFS is not supported OOB (I suppose some tinkering might help, but...).

Starting with VFS, starting a container fails:

$ sudo docker run -it --rm centos
/usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:327: setting cgroup config for procHooks process caused \"failed to write c 10:200 rwm to devices.allow: write /sys/fs/cgroup/devices/system.slice/docker-29dec37417e09a1b8e118bd9e7402578307c8c73ad562ee887f37efa7bc85449.scope/devices.allow: operation not permitted\"".
ERRO[0016] error getting events from daemon: context canceled
With privileged containers device usage is very restricted.
If you need a privileged container you'll need to allow the use of /dev/net/tun (the c:10:200 it's trying to allow access to, used for tunnel and tap device setup):
# /etc/pve/lxc/$vmid.conf
lxc.cgroup.devices.allow = c 10:200 rwm

With an unprivileged centos7 container with `features: nesting=1,keyctl=1` and docker-ce I can run the above `docker run` command you mentioned without any extra lines in the config.

I'm not sure why it would fail on `c:10:200` which is /dev/net/tun though - the device node is not available on the unprivileged container by default either - however, the cgroup does permit access to it in that case, so the operation you posted above wouldn't error either.
OK, using an unprivileged container helps. With keyctl I can run Docker. The overlay storage driver detects its on ZFS and refuses to work, but VFS works. I need to play with this as I suspect VFS is terrible.

However, switching to an unprivileged container breaks my FreeIPA client setup.
overlay2 detects you are in a COW filesystem and rejects to work.

The zfs driver doesn't seem to work
Host: Proxmox 5.3 with zfs storage
Container with nesting=1
Docker installed inside the container

The Docker storage driver "vfs" works fine, but is a bit slow.
When using the "overlay2" driver (I did 'modprobe overlay' on the host) then it complains:
level=error msg="'overlay2' is not supported over zfs" storage-driver=overlay2
Error starting daemon: error initializing graphdriver: backing file system is unsupported for this graph driver
When using the "zfs" driver then it complains:
Error starting daemon: error initializing graphdriver: prerequisites for driver not satisfied (wrong filesystem?)

Any idea how to get either of the two drivers working?
With "features: nesting=1" Docker runs nicely in LXC container (with ZFS) but there is a huge annoyance.
Disk usage is trough the roof. It takes +30Gb to run a simple docker command to build 3 containers.
I can't figure it out. Why is Docker such a hog on LXC containers?
IIRC the VFS storage driver doesn't do the copy-on-write trick to just store differences between layers, and there seems to be no driver which can work on LXC + ZFS which does the trick.


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!