5.3: Docker on LXC on ZFS

koalillo

New Member
Nov 1, 2018
11
0
1
39
Hi,

I'm toying with running Docker inside a CentOS 7 LXC on a Proxmox whose storage is ZFS. The container has:

$ cat /etc/pve/lxc/209.conf
...
features: nesting=1
...

and is a privileged container.

For starters, the only storage driver that allows me to start the Docker service is vfs. ZFS is not supported OOB (I suppose some tinkering might help, but...).

Starting with VFS, starting a container fails:

$ sudo docker run -it --rm centos
...
/usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:327: setting cgroup config for procHooks process caused \"failed to write c 10:200 rwm to devices.allow: write /sys/fs/cgroup/devices/system.slice/docker-29dec37417e09a1b8e118bd9e7402578307c8c73ad562ee887f37efa7bc85449.scope/devices.allow: operation not permitted\"".
ERRO[0016] error getting events from daemon: context canceled
 

wbumiller

Proxmox Staff Member
Staff member
Jun 23, 2015
647
88
48
With privileged containers device usage is very restricted.
If you need a privileged container you'll need to allow the use of /dev/net/tun (the c:10:200 it's trying to allow access to, used for tunnel and tap device setup):
Code:
# /etc/pve/lxc/$vmid.conf
(...)
lxc.cgroup.devices.allow = c 10:200 rwm
With an unprivileged centos7 container with `features: nesting=1,keyctl=1` and docker-ce I can run the above `docker run` command you mentioned without any extra lines in the config.

I'm not sure why it would fail on `c:10:200` which is /dev/net/tun though - the device node is not available on the unprivileged container by default either - however, the cgroup does permit access to it in that case, so the operation you posted above wouldn't error either.
 

koalillo

New Member
Nov 1, 2018
11
0
1
39
OK, using an unprivileged container helps. With keyctl I can run Docker. The overlay storage driver detects its on ZFS and refuses to work, but VFS works. I need to play with this as I suspect VFS is terrible.

However, switching to an unprivileged container breaks my FreeIPA client setup.
 

koalillo

New Member
Nov 1, 2018
11
0
1
39
overlay2 detects you are in a COW filesystem and rejects to work.

The zfs driver doesn't seem to work
 

Sebastian256

New Member
Aug 12, 2017
2
0
1
29
Host: Proxmox 5.3 with zfs storage
Container with nesting=1
Docker installed inside the container

The Docker storage driver "vfs" works fine, but is a bit slow.
When using the "overlay2" driver (I did 'modprobe overlay' on the host) then it complains:
level=error msg="'overlay2' is not supported over zfs" storage-driver=overlay2
Error starting daemon: error initializing graphdriver: backing file system is unsupported for this graph driver
When using the "zfs" driver then it complains:
Error starting daemon: error initializing graphdriver: prerequisites for driver not satisfied (wrong filesystem?)

Any idea how to get either of the two drivers working?
 

SamTzu

Active Member
With "features: nesting=1" Docker runs nicely in LXC container (with ZFS) but there is a huge annoyance.
Disk usage is trough the roof. It takes +30Gb to run a simple docker command to build 3 containers.
I can't figure it out. Why is Docker such a hog on LXC containers?
 

koalillo

New Member
Nov 1, 2018
11
0
1
39
IIRC the VFS storage driver doesn't do the copy-on-write trick to just store differences between layers, and there seems to be no driver which can work on LXC + ZFS which does the trick.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!