Docker LXC Unprivileged container on Proxmox 7 with ZFS

kamzata

Active Member
Jan 21, 2011
179
7
38
Venezia - Italy
I'm using Proxmox 7.0-11 on ZFS filesystem and I'm trying to use Dokku (which uses Docker) on a Ubuntu 20.04 LXC Unprivileged container.

On the container, I enabled the nesting and keyctl features right after created using the Ubuntu 20.04 template. Here the config:

Bash:
root@srv001:~# pct config 104
arch: amd64
cores: 4
features: keyctl=1,nesting=1
hostname: dokku
memory: 2048
net0: name=eth0,bridge=vmbr0,gw=192.168.1.1,hwaddr=72:41:4B:AE:C1:DF,ip=192.168.1.104/24,type=veth
ostype: ubuntu
rootfs: local-zfs:subvol-104-disk-0,size=24G
swap: 2048
unprivileged: 1

Then, after upgrade, I run:

Bash:
wget https://raw.githubusercontent.com/dokku/dokku/v0.26.6/bootstrap.sh;
DOKKU_TAG=v0.26.6 bash bootstrap.sh

in order to install Dokku.

The installation seems worked but the disk space rapidly increase from 2GB to 10GB.

Then, after creating a Dokku apps, I deployed my Hello World node application. This took a lot of time (around 30 minutes) and disk space (which it's now 20GB). Running a ncdu / on the container filesystem shows the folder /var/lib/docker/vfs/dir/ takes basically the whole disk space (with tens of files). Furthermore, every single additional Dokku command is reallllly slow and takes more and more disk space.

I've never used Docker before so maybe you can help me. How can I fix this?
 
  • Like
Reactions: zorrobiwan

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
15,525
911
163
Any chance of use it on a LXC container?
Probably yes, but this is not supported and you therefore you will not get much help for this here.
 
  • Like
Reactions: kamzata

zorrobiwan

New Member
Jun 10, 2020
24
3
3
49
I experience the same behavior

(still) Proxmox 6.4 on ZFS and LXC containers
Any operation in Docker is really slow and disk usage is growing
Building images takes hours and I got time out on starting containers in a stack so I have to start them several times.

So, thx, now I know that Docker on LXC is not recommended. I've been searching information about a recommended way to use Docker with Proxmox but found anything relevant.
 

LnxBil

Famous Member
Feb 21, 2015
6,059
740
133
Germany
I've been searching information about a recommended way to use Docker with Proxmox but found anything relevant.
It's the same for any other hypervisor. It does not make sense to run it directly on the hypervisor. Just install a VM (or a bunch of them) and run Docker (or Swarm or K8S) on it and everything works as it should be.
 
  • Like
Reactions: zorrobiwan

MMartinez

Active Member
Dec 11, 2014
41
4
28
I've been using docker on LXC with Proxmox (6.4 and earlier versions) for 3 years. It works fine. The purpose of using Debian/ubuntu LXC containers is that I need to give many machines to a group of about 20 students with just one fisical server and it would not be able to run 20 (or even more) kvm VM.

With the release of PVE 7.0 I read about problems with cgroups2 (or similar) so I haven't updated that server to last version. It would be great to solve the problems to run docker on LXC as for teaching purposes is great to give LXC containers to the students as they are light and easier to manage (you can do a pct enter even without knowing the password) than KVM.

Regards,

Manuel Martínez
 
Last edited:
  • Like
Reactions: Helmut101

MMartinez

Active Member
Dec 11, 2014
41
4
28
No, sorry for the confusion. It is not on ZFS, the physical server is an HP DL380 Gen7 and the containers run on a hardware raid mounted via fstab and registered as a directory in proxmox.
 
Last edited:

vesalius

Active Member
Aug 19, 2020
121
30
28
I've been using docker on LXC with Proxmox (6.4 and earlier versions) for 3 years. It works fine. The purpose of using Debian/ubuntu LXC containers is that I need to give many machines to a group of about 20 students with just one fisical server and it would not be able to run 20 (or even more) kvm VM.

With the release of PVE 7.0 I read about problems with cgroups2 (or similar) so I haven't updated that server to last version. It would be great to solve the problems to run docker on LXC as for teaching purposes is great to give LXC containers to the students as they are light and easier to manage (you can do a pct enter even without knowing the password) than KVM.

Regards,

Manuel Martínez
Link could be helpful.

https://www.reddit.com/r/Proxmox/co.../?utm_source=share&utm_medium=web2x&context=3
 

kamzata

Active Member
Jan 21, 2011
179
7
38
Venezia - Italy
Just upgraded to Proxmox 7.1. Anyway, before the upgrade from 7.0 version, I had already created a VM running Ubuntu 20.04 and Docker runs great. Since I still would like run Docker in a LXC container, does anyone know if the upgrade to 7.1 made this now possible?
 

ikus060

New Member
Nov 18, 2021
13
6
3
36
I'm also running a couple of Docker Deamon on LXC without issues on PVE 6.6.

The problem you encounter scare me. Will test this new PVE 7.1 version before upgrading.
 

MrPowerGamerBR

New Member
Oct 27, 2020
20
3
3
22
Is it enough fast? Does it have any "space increasing" problem?
I'm not @ikus060 but I tried following the guide and it worked!

I was using a 20GBs disk size LXC container just for a simple Docker container + PostgreSQL, because anything less than that Docker would complain about "disk quota exceeded".

First, some "gotcha"'s that I found while following the guide:

  • The name of the package that you need to install in your Proxmox host is "fuse-overlayfs"... yeah, the name is kinda obvious but who knows. Installing the package will remove "fuse" and replace it with "fuse3", but according to this thread, it shouldn't cause any issues!
  • In the container, you need to chmod the "fuse-overlayfs" binary: "chmod 777 /usr/local/bin/fuse-overlayfs". If you don't do that, Docker won't recognize the overlayfs driver!
  • You don't need to use "docker.io", I installed Docker following Docker's guide on their website, and it worked fine.

And after all of that, I was able to use just a 4GB disk for the LXC container! Just to be sure that it was the "fuse-overlayfs" driver that fixed the issue, I changed the driver to "vfs" and I wasn't able to run my containers because "disk quota exceeded", so yay!

I haven't tested the performance, but it seems to work fine... maybe even a bit faster than vfs? The "Extracting" part of pulling a container image on a fuse-overlayfs backed LXC container was waaaay faster than vfs, but maybe this is just placebo. :p

When using "df -h", it shows that it is using fuse-overlayfs
Code:
root@username:/# df -h
Filesystem                 Size  Used Avail Use% Mounted on
rpool/subvol-40018-disk-0  4.0G  2.5G  1.6G  61% /
none                       492K  4.0K  488K   1% /dev
udev                        63G     0   63G   0% /dev/tty
tmpfs                       63G     0   63G   0% /dev/shm
tmpfs                       26G  220K   26G   1% /run
tmpfs                      5.0M     0  5.0M   0% /run/lock
fuse-overlayfs             4.0G  2.5G  1.6G  61% /var/lib/docker/fuse-overlayfs/d2ab45c55d20d88a1d01a05712f7e9b100c662f3577638c9b4575492229fd683/merged
fuse-overlayfs             4.0G  2.5G  1.6G  61% /var/lib/docker/fuse-overlayfs/e0c23ac22c5c2d9f421d6849ed8a8af718b8def0a55cc046bb0a28bf9d931c4a/merged
 
Dec 2, 2020
44
16
8
31
Thanks for the report.

There is also another approach/guide. Check this ansible notebook that contains all steps:

- create zvol and format with ext4
- mount the drive and chown all files to 100000:100000 with chmod 0711
- can be mounted in docker with overlayfs2 and nesting=1,keyctl=1 enabled (as unpriviliged)
- caveat: Works with Ubuntu Docker container only (see docker docs)

This guide was Proxmox 6.x, it is unclear whether this still works in 7.x. I just got my JBOD and will hopefully test ZFS soon, but would be good to know ahead of time.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!