Docker LXC Unprivileged container on Proxmox 7 with ZFS

[kamzata]

I'm using Proxmox 7.0-11 on ZFS filesystem and I'm trying to use Dokku (which uses Docker) on a Ubuntu 20.04 LXC Unprivileged container.

On the container, I enabled the nesting and keyctl features right after created using the Ubuntu 20.04 template. Here the config:

root@srv001:~# pct config 104
arch: amd64
cores: 4
features: keyctl=1,nesting=1
hostname: dokku
memory: 2048
net0: name=eth0,bridge=vmbr0,gw=192.168.1.1,hwaddr=72:41:4B:AE:C1:DF,ip=192.168.1.104/24,type=veth
ostype: ubuntu
rootfs: local-zfs:subvol-104-disk-0,size=24G
swap: 2048
unprivileged: 1

Then, after upgrade, I run:

wget https://raw.githubusercontent.com/dokku/dokku/v0.26.6/bootstrap.sh;
DOKKU_TAG=v0.26.6 bash bootstrap.sh

in order to install Dokku.

The installation seems worked but the disk space rapidly increase from 2GB to 10GB.

Then, after creating a Dokku apps, I deployed my Hello World node application. This took a lot of time (around 30 minutes) and disk space (which it's now 20GB). Running a ncdu / on the container filesystem shows the folder /var/lib/docker/vfs/dir/ takes basically the whole disk space (with tens of files). Furthermore, every single additional Dokku command is reallllly slow and takes more and more disk space.

I've never used Docker before so maybe you can help me. How can I fix this?

 

[tom]

I've never used Docker before

Please use Docker inside a VM (this is the recommended way) and not in LXC container.

 

[kamzata]

Please use Docker inside a VM (this is the recommended way) and not in LXC container.

Any chance of use it on a LXC container?

 

[tom]

Any chance of use it on a LXC container?

Probably yes, but this is not supported and you therefore you will not get much help for this here.

 

[zorrobiwan]

I experience the same behavior

(still) Proxmox 6.4 on ZFS and LXC containers
Any operation in Docker is really slow and disk usage is growing
Building images takes hours and I got time out on starting containers in a stack so I have to start them several times.

So, thx, now I know that Docker on LXC is not recommended. I've been searching information about a recommended way to use Docker with Proxmox but found anything relevant.

 

[LnxBil]

I've been searching information about a recommended way to use Docker with Proxmox but found anything relevant.

It's the same for any other hypervisor. It does not make sense to run it directly on the hypervisor. Just install a VM (or a bunch of them) and run Docker (or Swarm or K8S) on it and everything works as it should be.

 

[MMartinez]

I've been using docker on LXC with Proxmox (6.4 and earlier versions) for 3 years. It works fine. The purpose of using Debian/ubuntu LXC containers is that I need to give many machines to a group of about 20 students with just one fisical server and it would not be able to run 20 (or even more) kvm VM.

With the release of PVE 7.0 I read about problems with cgroups2 (or similar) so I haven't updated that server to last version. It would be great to solve the problems to run docker on LXC as for teaching purposes is great to give LXC containers to the students as they are light and easier to manage (you can do a pct enter even without knowing the password) than KVM.

Regards,

Manuel Martínez

 

[zorrobiwan]

Interesting @MMartinez !

Are you using ZFS on your physical server ?

 

[MMartinez]

No, sorry for the confusion. It is not on ZFS, the physical server is an HP DL380 Gen7 and the containers run on a hardware raid mounted via fstab and registered as a directory in proxmox.

 

[zorrobiwan]

No, sorry for the confusion. It is not on ZFS, the physical server is an HP DL380 Gen7 and the containers run on a hardware raid mounted via fstab and registered as a directory in proxmox.

ok thx for the reply !

 

[vesalius]

I've been using docker on LXC with Proxmox (6.4 and earlier versions) for 3 years. It works fine. The purpose of using Debian/ubuntu LXC containers is that I need to give many machines to a group of about 20 students with just one fisical server and it would not be able to run 20 (or even more) kvm VM.

With the release of PVE 7.0 I read about problems with cgroups2 (or similar) so I haven't updated that server to last version. It would be great to solve the problems to run docker on LXC as for teaching purposes is great to give LXC containers to the students as they are light and easier to manage (you can do a pct enter even without knowing the password) than KVM.

Regards,

Manuel Martínez

Link could be helpful.

https://www.reddit.com/r/Proxmox/co.../?utm_source=share&utm_medium=web2x&context=3

 

[kamzata]

Just upgraded to Proxmox 7.1. Anyway, before the upgrade from 7.0 version, I had already created a VM running Ubuntu 20.04 and Docker runs great. Since I still would like run Docker in a LXC container, does anyone know if the upgrade to 7.1 made this now possible?

 

[ikus060]

I'm also running a couple of Docker Deamon on LXC without issues on PVE 6.6.

The problem you encounter scare me. Will test this new PVE 7.1 version before upgrading.

 

[Helmut101]

Please update here with your experiences. Still with Kernel 6.4 due to the unpriviliged LXC/Docker issues.

 

[ikus060]

I'm running Docker in LXC unpriviliged on Proxmox v7 with a ZFS storage.

I've install Docker with fuse-overlayfs. That solve all the problem. It's working out of the box.

 

[kamzata]

I'm running Docker in LXC unpriviliged on Proxmox v7 with a ZFS storage.

I've install Docker with fuse-overlayfs. That solve all the problem. It's working out of the box.

That's a great news. How do you install Docker with fuse-overlayfs? More precisely I'm trying to install Dokku which uses Docker.

 

[ikus060]

Essentially, I've follow this:
https://c-goes.github.io/posts/proxmox-lxc-docker-fuse-overlayfs/

 

[kamzata]

Essentially, I've follow this:
https://c-goes.github.io/posts/proxmox-lxc-docker-fuse-overlayfs/

Is it enough fast? Does it have any "space increasing" problem?

 

[MrPowerGamerBR]

Is it enough fast? Does it have any "space increasing" problem?

I'm not @ikus060 but I tried following the guide and it worked!

I was using a 20GBs disk size LXC container just for a simple Docker container + PostgreSQL, because anything less than that Docker would complain about "disk quota exceeded".

First, some "gotcha"'s that I found while following the guide:

• The name of the package that you need to install in your Proxmox host is "fuse-overlayfs"... yeah, the name is kinda obvious but who knows. Installing the package will remove "fuse" and replace it with "fuse3", but according to this thread, it shouldn't cause any issues!
• In the container, you need to chmod the "fuse-overlayfs" binary: "chmod 777 /usr/local/bin/fuse-overlayfs". If you don't do that, Docker won't recognize the overlayfs driver!
• You don't need to use "docker.io", I installed Docker following Docker's guide on their website, and it worked fine.

And after all of that, I was able to use just a 4GB disk for the LXC container! Just to be sure that it was the "fuse-overlayfs" driver that fixed the issue, I changed the driver to "vfs" and I wasn't able to run my containers because "disk quota exceeded", so yay!

I haven't tested the performance, but it seems to work fine... maybe even a bit faster than vfs? The "Extracting" part of pulling a container image on a fuse-overlayfs backed LXC container was waaaay faster than vfs, but maybe this is just placebo.

When using "df -h", it shows that it is using fuse-overlayfs

root@username:/# df -h
Filesystem                 Size  Used Avail Use% Mounted on
rpool/subvol-40018-disk-0  4.0G  2.5G  1.6G  61% /
none                       492K  4.0K  488K   1% /dev
udev                        63G     0   63G   0% /dev/tty
tmpfs                       63G     0   63G   0% /dev/shm
tmpfs                       26G  220K   26G   1% /run
tmpfs                      5.0M     0  5.0M   0% /run/lock
fuse-overlayfs             4.0G  2.5G  1.6G  61% /var/lib/docker/fuse-overlayfs/d2ab45c55d20d88a1d01a05712f7e9b100c662f3577638c9b4575492229fd683/merged
fuse-overlayfs             4.0G  2.5G  1.6G  61% /var/lib/docker/fuse-overlayfs/e0c23ac22c5c2d9f421d6849ed8a8af718b8def0a55cc046bb0a28bf9d931c4a/merged

 

[Helmut101]

Thanks for the report.

There is also another approach/guide. Check this ansible notebook that contains all steps:

- create zvol and format with ext4
- mount the drive and chown all files to 100000:100000 with chmod 0711
- can be mounted in docker with overlayfs2 and nesting=1,keyctl=1 enabled (as unpriviliged)
- caveat: Works with Ubuntu Docker container only (see docker docs)

This guide was Proxmox 6.x, it is unclear whether this still works in 7.x. I just got my JBOD and will hopefully test ZFS soon, but would be good to know ahead of time.