5.3: Docker on LXC on ZFS

[koalillo]

Hi,

I'm toying with running Docker inside a CentOS 7 LXC on a Proxmox whose storage is ZFS. The container has:

$ cat /etc/pve/lxc/209.conf
...
features: nesting=1
...

and is a privileged container.

For starters, the only storage driver that allows me to start the Docker service is vfs. ZFS is not supported OOB (I suppose some tinkering might help, but...).

Starting with VFS, starting a container fails:

$ sudo docker run -it --rm centos
...
/usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:327: setting cgroup config for procHooks process caused \"failed to write c 10:200 rwm to devices.allow: write /sys/fs/cgroup/devices/system.slice/docker-29dec37417e09a1b8e118bd9e7402578307c8c73ad562ee887f37efa7bc85449.scope/devices.allow: operation not permitted\"".
ERRO[0016] error getting events from daemon: context canceled

 

[wbumiller]

With privileged containers device usage is very restricted.
If you need a privileged container you'll need to allow the use of /dev/net/tun (the c:10:200 it's trying to allow access to, used for tunnel and tap device setup):

# /etc/pve/lxc/$vmid.conf
(...)
lxc.cgroup.devices.allow = c 10:200 rwm

With an unprivileged centos7 container with `features: nesting=1,keyctl=1` and docker-ce I can run the above `docker run` command you mentioned without any extra lines in the config.

I'm not sure why it would fail on `c:10:200` which is /dev/net/tun though - the device node is not available on the unprivileged container by default either - however, the cgroup does permit access to it in that case, so the operation you posted above wouldn't error either.

 

[koalillo]

OK, using an unprivileged container helps. With keyctl I can run Docker. The overlay storage driver detects its on ZFS and refuses to work, but VFS works. I need to play with this as I suspect VFS is terrible.

However, switching to an unprivileged container breaks my FreeIPA client setup.

 

[koalillo]

Solved my issues with FreeIPA https://forum.proxmox.com/threads/c...be-mapped-in-an-unprivileged-container.49544/

Now I wonder, is there a better storage driver than VFS to run inside an LXC container on a ZFS fs?

 

[morph027]

Not tried yet ... overlay2 probably?

 

[koalillo]

overlay2 detects you are in a COW filesystem and rejects to work.

The zfs driver doesn't seem to work

 

[Sebastian256]

Host: Proxmox 5.3 with zfs storage
Container with nesting=1
Docker installed inside the container

The Docker storage driver "vfs" works fine, but is a bit slow.
When using the "overlay2" driver (I did 'modprobe overlay' on the host) then it complains:
level=error msg="'overlay2' is not supported over zfs" storage-driver=overlay2
Error starting daemon: error initializing graphdriver: backing file system is unsupported for this graph driver
When using the "zfs" driver then it complains:
Error starting daemon: error initializing graphdriver: prerequisites for driver not satisfied (wrong filesystem?)

Any idea how to get either of the two drivers working?

 

[SamTzu]

With "features: nesting=1" Docker runs nicely in LXC container (with ZFS) but there is a huge annoyance.
Disk usage is trough the roof. It takes +30Gb to run a simple docker command to build 3 containers.
I can't figure it out. Why is Docker such a hog on LXC containers?

 

[koalillo]

IIRC the VFS storage driver doesn't do the copy-on-write trick to just store differences between layers, and there seems to be no driver which can work on LXC + ZFS which does the trick.