Docker LXC Unprivileged container on Proxmox 7 with ZFS

timdonovan said:
@Helmut101 BTW based on my recent findings I currently would not advise using LXC in Proxmox *at all*. I have shared the issue here:

https://forum.proxmox.com/threads/u...ventually-locks-up-pve-host-processes.108362/

But tl;dr unprivileged LXC containers using FUSE are knocking out an entire PVE host.
Click to expand...
Thanks for reporting! Yes, I heard similar reports for this docker-FUSE approach. Which is why I do not use FUSE - everything runs stable with /var/lib/docker on a separate host mount (as describes in the blog post). I haven't actually tested FUSE/fuse-overlayfs myself (primarily because I don't have a need, currently). My current uptime for the host is 84 days, not a single issue - so I assume this is related to the way you set up the LXC/docker.
 
Last edited:
  • Like
Reactions: luison
Hello Guys,

sry for digging this thread out of the dirt. I have tested lxc + docker + fuse for over 180 days in my cluster in more than 9 containers.
I dont have any error or something else. BUT i do it on another way like the most users.
You can use a static build for fuse-overlayfs with this static build i got never a error.
 
techsolo12 said:
Hello Guys,

sry for digging this thread out of the dirt. I have tested lxc + docker + fuse for over 180 days in my cluster in more than 9 containers.
I dont have any error or something else. BUT i do it on another way like the most users.
You can use a static build for fuse-overlayfs with this static build i got never a error.
Click to expand...
I can easily get docker running using:
Bash: 
apt install fuse-overlayfs
apt install docker.io
Unprivileged, fuse, keyctl, nesting

It runs without errors unless I try to migrate to another node or do a pbsbackup. Both will often work, but inevitably one will fail, and that fail can lock up the entire node.

Have you been doing migrations and/or pbsbackups over those 180 days?
 
  • Like
Reactions: Helmut101
BruceX said:
It runs without errors
Click to expand...
If that is your only goal, then go for it. Running in directly on the host on ZFS is the only other viable option besides running a VM that has ZFS in it and run Docker on top of that ZFS - if you want to run it inside of ZFS (meaning Docker with its ZFS storage plugin).
 
BruceX said:
I can easily get docker running using:
Bash: 
apt install fuse-overlayfs
apt install docker.io
Unprivileged, fuse, keyctl, nesting

It runs without errors unless I try to migrate to another node or do a pbsbackup. Both will often work, but inevitably one will fail, and that fail can lock up the entire node.

Have you been doing migrations and/or pbsbackups over those 180 days?
Click to expand...

Yes, i have done multiple times. I have a two node cluster with a seprate node as pbs. The pbs is qdevice for the cluster. I do 2 times in a week a fullbackup. So over 26 backups per container and over 19 migration in the 180 days. I use 9 containers with docker.
 
  • Like
Reactions: BruceX
LnxBil said:
If that is your only goal, then go for it. Running in directly on the host on ZFS is the only other viable option besides running a VM that has ZFS in it and run Docker on top of that ZFS - if you want to run it inside of ZFS (meaning Docker with its ZFS storage plugin).
Click to expand...
Its been running since September. techsolo12 seemed to suggest that a different version of fuse-overlayfs didn't have the migration/replication/backup issues.
 
Last edited:
techsolo12 said:
Yes, i have done multiple times. I have a two node cluster with a seprate node as pbs. The pbs is qdevice for the cluster. I do 2 times in a week a fullbackup. So over 26 backups per container and over 19 migration in the 180 days. I use 9 containers with docker.
Click to expand...
Was your solution "static build for fuse-overlayfs" or something else? If so, can you provide a pointer to the static build procedure you used? Thanks!
 
BruceX said:
Was your solution "static build for fuse-overlayfs" or something else? If so, can you provide a pointer to the static build procedure you used? Thanks!
Click to expand...

I have wrote yesterday a german guide on my blog. Hopefully i have tomorrow time to release it. After release i post the url here! So you can test it for yourself. :)

One thing i had fotgot to say is, if i do a snapshot i got a error. But the snapshot will work properbly. Why this error, i dont know. Since i dont use snapshot only full backups it was not important for me.

But I am not responsible if there are problems in your environment with this "fix". ;)
 
Last edited:
Today i updated my nodes to PVE 7.3.3 and also the static build from fuse-overlayfs to version 1.10 and got a error when i do
fuse-overlayfs --version
The error was something with lazy load... So please don't update the static build at the moment.

EDIT!: I have to correct something... After the update from 7.2.11 to 7.3.3 my tutorial don't work anymore. One of the nodes currently get unstable and go in lock after some minutes. So unfortunately, this won't work anymore with a version greater than 7.3.x
In one or two days, i remove it from my blog and also from this forum. I don't want that anybodies have the same sad situation like me.

With best regards,
techsolo12
 
Last edited:
Maybe something for all of you who stumble across this thread. I found the following on github during my research about this topic and it works just perfect. I want to share it with you.

It is nearly the same like others in this thread wrote already but because of the name convetion this is completely integrated in pve (after the manual steps)
Snapshots and backups work without problems.
No fusefs necessary.

https://github.com/nextcloud/all-in-one/discussions/1490
 
  • Like
Reactions: Helmut101
I'll need to look into this more closely, but it follows what I described here
https://du.nkel.dev/blog/2021-03-25_proxmox_docker/

.. and it has worked flawlessly since 3 years, tested up till the current Proxmox 7.3-4.

The latter steps look like this is solving the migration issue, thanks for sharing!

Regarding the other issues stated here earlier: I really suggest, anybody who does something remotely productive, to not use fuse-overlayfs for the lxc-docker nesting.
 
Last edited:
Are these workarounds still needed for docker in an unprivileged container backed by ZFS?

I just installed 7.3.3 on a new machine and created an LXC container. I loaded the overlayfs module on the host and configured docker in the container to use overlayfs2 driver. This is the output of "docker info":

Code: 
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.16.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 23.0.1
 Storage Driver: overlay2
  Backing Filesystem: zfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.85-1-pve
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 512MiB
 Name: dockertest
 ID: 88e90dc2-23ae-4b12-99cf-c1665ed3c4f2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Could it be that this merge here (https://github.com/truenas/zfs/pull/101) actually added the missing bits to enable overlayfs support on zfs in zfs 2.1?
 
  • Like
Reactions: Helmut101
Hi folks,

i too started a few containers and most of them just run fine. But with that one: "ghcr.io/paperless-ngx/paperless-ngx:latest" i get a :
Code: 
failed to register layer: ApplyLayer exit status 1 stdout: stderr: unlinkat /usr/local/lib/python3.9/site-packages/pkg_resources/tests/data/my-test-package-source: invalid argument

The same docker-compose file works both on overlay2 with ext4 and xfs. Only with overlay2 backed by zfs the stack would not start. So there may still be some workarounds needed.
 
  • Like
Reactions: kam821 and Helmut101
scaredweasel said:
Hi folks,

i too started a few containers and most of them just run fine. But with that one: "ghcr.io/paperless-ngx/paperless-ngx:latest" i get a :
Code: 
failed to register layer: ApplyLayer exit status 1 stdout: stderr: unlinkat /usr/local/lib/python3.9/site-packages/pkg_resources/tests/data/my-test-package-source: invalid argument

The same docker-compose file works both on overlay2 with ext4 and xfs. Only with overlay2 backed by zfs the stack would not start. So there may still be some workarounds needed.
Click to expand...

Same problem with pihole/pihole.

Bash: 
root@pihole:/root# docker-compose up
Pulling pihole (pihole/pihole:latest)...
latest: Pulling from pihole/pihole
bb263680fed1: Pull complete
de9654fb76c7: Pull complete
4f4fb700ef54: Pull complete
cbe6380c3a6b: Pull complete
34575e9e3344: Pull complete
44048c8f578a: Pull complete
ee827dbc8bf8: Extracting [==================================================>]  30.34MB/30.34MB
6c22f91571c7: Download complete
67421bc419c4: Download complete
ERROR: failed to register layer: ApplyLayer exit status 1 stdout:  stderr: unlinkat /var/cache/apt/archives: invalid argument
 
  • Like
Reactions: frozenfoxx
I just started having this problem yesterday. I use the terraform provider to provision unprivileged LXC containers and then ansible to install Docker and run the containers. This wasn't an issue last Thursday but yesterday I stopped being able to deploy Docker containers in the LXC container with the same "stderr: unlikat" message as above. Even destroying and recreating the LXC doesn't fix it.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom