Toying with gvisor to run Docker inside an LXC container in a safe? way

[koalillo]

Hi,

The other day I stumbled upon gvisor, which is a Google container driver which can be used with Docker that apparently uses an approach which is "mostly user-mode"; it emulates a Linux kernel in usermode, so it might be a way to run Docker containers inside an LXC container without nesting cgroups stuff and all that.

However, I stopped at:

https://github.com/google/gvisor/issues/1499

. Maybe someone more knowledgeable can give it a shot? There's quite a few promising container drivers out there which might allow a way to run Docker inside LXC without compromising security (e.g. there's some interesting progress in other areas at https://forum.proxmox.com/threads/docker-in-lxc-problem-after-pve-kernel-update.56948/ , I'm pinging T.Herrmann there...).

Cheers,

Álex

 

[T.Herrmann]

No experiences with gvisor.

I think on the long run there will be two ways for HA with Docker Container>> Kubernetes and PVE with LXC Docker in cluster mode.

Biggest Problem with PVE and LXC with Docker is the vfs only support as file-system for docker images. No Overlay2 or ZFS is supported until now. This makes no problem for small docker images but if you use images with 0.5 million files it makes a huge differences.

 

[koalillo]

Well, I'm thinking of a non-HA scenario- just being able to do development/CI on a container (e.g. run Jenkins on LXC and be able to do builds requiring Docker, using an LXC container as a dev workstation, etc.).

 

[T.Herrmann]

Ok

 

[oguz]

hi,
you're better off running jenkins in a VM if you want to do docker builds

 

[koalillo]

I'm running it in a VM currently, but I'm running a 35€/month 48gb RAM server and LXC allows me to pack in a lot of stuff... Right now ideally I'd require 4 VMs for all the Docker stuff I want to do, which represents significant overhead on that.

Anyway, the gVisor bug I filed was idiotic- it does seem that I need to play more with storage to get things running.

This is a bit frustrating because it wouldn't be so difficult to do Docker stuff inside LXC in a reasonable way; gVisor, udocker, proot... there are lots of scattered bits and pieces of a solution, but they don't quite fit. I'm thinking right now that it wouldn't be so difficult to write a shim that emulates most of the Docker functionality I want to do with proot...

 

[koalillo]

Just toyed a bit with Singularity in a CentOS 8 container. It seems... to work, so that's interesting. However, while it can use Docker Hub images and it's pretty similar, it's not a drop-in replacement (however, it seems Kubernetes can be configured to use it, so maybe you could run Kubernetes in an LXC container using Singularity?).

However, non-root usage doesn't seem to work.