[SOLVED] Run Docker inside LXC

[Bugbear]

Hi,

I'm running Proxmox 7.2 want to run a few applications as docker containers but save the overhead a VM would bring by having Docker inside a Proxmox LXC.
Could find those two articles on how to accomplish that with an unprivileged LXC but it doesn't work on my system.
https://forum.proxmox.com/threads/docker-in-an-lxc.32938/
https://www.solaris-cookbook.eu/virtualisation/proxmox/proxmox-lxc-running-docker-inside-container/

Procedure:

root@PVE:~# cat /etc/pve/lxc/112.conf
arch: amd64
cores: 4
cpulimit: 3
features: nesting=1
hostname: Codex
memory: 16384
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=7E:18:7E:D0:78:F3,ip=dhcp,ip6=dhcp,type=veth
ostype: ubuntu
rootfs: local-lvm:vm-112-disk-0,size=128G
swap: 0
unprivileged: 1
#################### docker section ####################
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:

## Reboot container
## Try to run a docker container inside the LXC

root@Codex:~# docker run --rm hello-world
docker: Error response from daemon: Could not check if docker-default AppArmor profile was loaded: open /sys/kernel/security/apparmor/profiles: permission denied.

Do you have ideas what I did wrong?
Any suggestions appreciated!

 

[Dunuin]

I also needed the keyctl feature enabled to run docker in an unprivileged LXC.
Its by the way not recommended to run docker inside a LXC, because a VM would be more secure and more reliable causing less problems.

 

[Bugbear]

Oh, yeah, I had forgotten that ..

So is everything that should be needed as follows?

• Enable Nesting
• Enable Keyctl
• Add to /etc/pve/lxc/<ID>.conf:

lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:

And is it certainly lxc.cgroup.devices.allow: a and not lxc.cgroup2.devices.allow: a? Both are somehow not working for me, I tried it multiple times.

 

[Dunuin]

PVE 7.X by default doesn't support cgroup anymore (just cgroup2). I can't remember that I needed to edit the "/etc/pve/lxc/<ID>.conf" back then when I set it up under PVE 6.X.

 

[vesalius]

Check here for some scripts to make some common LXC setups (including docker) easy.

https://github.com/tteck/Proxmox

 

[liberodark]

Hi,

I recommend this options but that mostly depend on what you need to do :
In PVE 7.x is not really needed to edit the lxc configuration.

When you use :

lxc.apparmor.profile: unconfined
lxc.cap.drop:

It is mainly because you have a specific need. You should not use its default options but precisely avoid putting them if possible.

PVE 7.x
Debian + Nesting 1 (Unpriviled Container : No)

lxc.apparmor.profile: unconfined
lxc.cap.drop:

AlmaLinux 8 + Nesting 1 (Unpriviled Container : No)

PVE 6.x
Debian + Nesting 1 (Unpriviled Container : No)

lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
lxc.mount.auto:

Centos + Nesting 1 + Fuse 1 (Unpriviled Container : No)

lxc.cgroup.devices.allow: a
lxc.mount.auto: sys
lxc.cap.drop:

PS : Warning are only valid if you do not use the ZFS which involves other settings.

Best Regards

 

[Bugbear]

Thank you all for the support! You were right, editing the ct-id.conf was not necessary.
Without that, so only with nesting and keyctl docker works perfectly inside my (Ubuntu 22.04) LXC.
For all trying the same, in order to be on the safe side make sure you do this configurations to the LXC before first launch of the LXC, at least in my testing it turned out not to be very consistent when changing the config afterwards.

 

[SimonMcNair]

big thanks to tteck for the scripts

 

[Rivolity]

Hello,
Enlighten me !
What is the purpose of running a docker inside an LXC ? Why don't you run your containers (after adapting your script) directly to LXC ? Since the purpose of LXC on Proxmox is running your containers without adding another level of containerization (virtualization of OS).

Lxc main focus is system containers. Containers, which offer an environment as close as possible to the one you’d get from a VM, but without the overhead that comes with running a separate kernel and simulating all the hardware, but running Docker in LXC is a slightly controversial idea.

Docker aims at running a single application in an isolated, self-contained environment. These are generally referred to as “Application Containers”, rather than “System Containers”. You manage a Docker instance from the host, using the Docker Engine command line interface.

It is not recommended to run docker directly on your Proxmox VE host.

If you want to run application containers, for example, Docker images, it is best to run them inside a Proxmox Qemu VM.

 

[SimonMcNair]

I think, but am not sure, that lxc provides a certain level of isolation, but not as much as a vm. If you run docker in lxc some containers may not work if they require access to the kernel that lxc will not permit ?

If someone could clarify or correct me I would appreciate it.

 

[Bugbear]

So, of course everything depends on the specific scenario ...
But in my case I'd like to run some applications inside of docker runtime. As you mentioned @Rivolity, it is not recommended to run docker directly on your Proxmox VE host.
Therefore there are two options left now: Run Docker inside a Qemu VM or inside of a LXContainer.
Qemu VM would objectively be the preferred option as it gives you that extra piece of isolation. But with a limited amount of resources, LXC allows me to have some degree of isolation from the host but still be reasonably lightweight and enjoy the other benefits of LXC