LXC + unprivileged + Docker OK | but LXC + privileged + Docker broken?

[metjeh]

Apparently after my maintenance update of Proxmox I have come accross the following Bug / Oddity:

Running an unprivileged=YES LXC (Fedora) with Docker => OK

Create container via CLI, the following config is generated

lxc.cgroup.relative = 0
lxc.cgroup.dir.monitor = lxc.monitor/9040
lxc.cgroup.dir.container = lxc/9040
lxc.cgroup.dir.container.inner = ns
lxc.arch = amd64
lxc.include = /usr/share/lxc/config/fedora.common.conf
lxc.include = /usr/share/lxc/config/fedora.userns.conf
lxc.seccomp.profile = /var/lib/lxc/9040/rules.seccomp
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1
lxc.mount.auto = sys:mixed
lxc.monitor.unshare = 1
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.console.path = none
lxc.cgroup2.devices.deny = c 5:1 rwm
lxc.tty.max = 0
lxc.environment = TERM=linux
lxc.uts.name = 9040-nextcloud
lxc.cgroup2.memory.max = 34376515584
lxc.cgroup2.memory.high = 34242297856
lxc.cgroup2.memory.swap.max = 0
lxc.rootfs.path = /var/lib/lxc/9040/rootfs
lxc.net.0.type = veth
lxc.net.0.veth.pair = veth9040i0
lxc.net.0.hwaddr = BC:24:11:00:90:40
lxc.net.0.name = eth0
lxc.net.0.mtu = 1500
lxc.net.0.script.up = /usr/share/lxc/lxcnetaddbr
lxc.cgroup2.cpuset.cpus = 0,2-3,5

Running an unprivileged=NO LXC (Fedora) with Docker => not OK

lxc.cgroup.relative = 0
lxc.cgroup.dir.monitor = lxc.monitor/9040
lxc.cgroup.dir.container = lxc/9040
lxc.cgroup.dir.container.inner = ns
lxc.arch = amd64
lxc.include = /usr/share/lxc/config/fedora.common.conf


lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1

lxc.monitor.unshare = 1


lxc.console.path = none
lxc.cgroup2.devices.deny = c 5:1 rwm
lxc.tty.max = 0
lxc.environment = TERM=linux
lxc.uts.name = 9040-nextcloud
lxc.cgroup2.memory.max = 34376515584
lxc.cgroup2.memory.high = 34242297856
lxc.cgroup2.memory.swap.max = 0
lxc.rootfs.path = /var/lib/lxc/9040/rootfs
lxc.net.0.type = veth
lxc.net.0.veth.pair = veth9040i0
lxc.net.0.hwaddr = BC:24:11:00:90:40
lxc.net.0.name = eth0
lxc.net.0.mtu = 1500
lxc.net.0.script.up = /usr/share/lxc/lxcnetaddbr
lxc.cgroup2.cpuset.cpus = 0-1,3-4

To get the priviliged lxc container to work properly I have to
- manually modify the lxc config at /var/lib/lxc/9040/config and
- then use lxc-start 9040
- pct enter 9040 and docker run --rm hello-world

lxc.include = /usr/share/lxc/config/fedora.userns.conf
lxc.seccomp.profile = /var/lib/lxc/9040/rules.seccomp

As soon as i stop the container and restart it via pct start 9040 the config at /var/lib/lxc/9040/config is overwritten and the two lines are removed.

What I am suspecting is that with a recent update of the PCT tool, there might be a bug while generating the lxc config?



EDIT: see answer below ... I suspect that something with the seccomp / permission seems to be off in a privileged lxc ... although one would suspect, that it should have "more" freedom?

 

[metjeh]

Further investigation shows the following scenarios:

1) LXC unprivileged + Fedora + Docker => OK

2) LXC privileged + Fedora + Docker => broken (docker complains that it can't start shim / permission)

to "fix" 2) you need to either

resort to manually adding:

# fix fedora privileged lxc not wanting docker without priviliged
lxc.cgroup2.devices.allow: a
lxc.cap.drop:

or run the docker images inside with --privileged: true (but this will not work if you use docker compose to build images!)

I am aware that Docker in LXC is not supported but the above is for reference. However I wonder why docker in unprivileged lxc is ok while in privileged not?!