Running docker containers in proxmox containers

[droidus]

I am sure that this doesn't make a lot of sense, but I am trying to install software that is recommended to be installed using Docker. I've tried this twice now, and I seem to run into the same issue, which makes me think I can't run Docker containers within the proxmox native containers. Here is the error message that I encounter:

docker run -d -p 80:8080 miniflux/miniflux:latest
3fc1662492a8447c43cb2a6761fdeccc306003cffc70c08d0c8c4c42771abbe6
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"proc\\\" to rootfs \\\"/var/lib/docker/overlay2/f9b1c95f723519652c87162a208e7fcb8817d68e13f9e082fbeaf8370e6b79c9/merged\\\" at \\\"/proc\\\" caused \\\"permission denied\\\"\"": unknown.

I assume this is because the proxmox container does not have all of the necessary FS hierarchy/structure required to run docker containers?

 

[mir]

You need to run in a privileged container. By default a LXC container is created unprivileged. See here: https://forum.proxmox.com/threads/docker-under-proxmox.81618/post-360134

 

[Dunuin]

Unprivileged LXC is fine for docker but you need to enable the "nesting" and "keyctl" features first.

 

[mir]

Unprivileged LXC is fine for docker but you need to enable the "nesting" and "keyctl" features first.

You are correct. Using https://pve.proxmox.com/wiki/Unprivileged_LXC_containers to convert from privileged to unprivileged container. Though, if you have disabled login as any user including root with password and only allow certificate based logins should narrow safety down ;-)

 

[droidus]

I know of the nesting option, but have not heard of the keyctl one. What is that, or what does it mean/do?

 

[mir]

I know of the nesting option, but have not heard of the keyctl one. What is that, or what does it mean/do?

From https://[your_pve_host]/pve-docs/chapter-pct.html#pct_options:
keyctl=<boolean> (default = 0)
For unprivileged containers only: Allow the use of the keyctl() system call. This is required to use docker inside a container. By default unprivileged containers will see this system call as non-existent. This is mostly a workaround for systemd-networkd, as it will treat it as a fatal error when some keyctl() operations are denied by the kernel due to lacking permissions. Essentially, you can choose between running systemd-networkd or docker.

keyctl: https://linux.die.net/man/1/keyctl

 

[droidus]

Thank you for sharing that link. Interestingly, reading from the beginning to try and understand all of this, I came upon this:
"

If you want to run application containers, for example, Docker images, it is recommended that you run them inside a Proxmox Qemu VM. This will give you all the advantages of application containerization, while also providing the benefits that VMs offer, such as strong isolation from the host and the ability to live-migrate, which otherwise isn’t possible with containers.

"
So they are essentially saying, don't use containers, and VMs are more secure than a container? Aren't containers isolated as well, if you use them in unprivileged mode?

 

[mir]

Thank you for sharing that link. Interestingly, reading from the beginning to try and understand all of this, I came upon this:
"

If you want to run application containers, for example, Docker images, it is recommended that you run them inside a Proxmox Qemu VM. This will give you all the advantages of application containerization, while also providing the benefits that VMs offer, such as strong isolation from the host and the ability to live-migrate, which otherwise isn’t possible with containers.

"
So they are essentially saying, don't use containers, and VMs are more secure than a container? Aren't containers isolated as well, if you use them in unprivileged mode?

I think that part is from a time where unprivileged containers was not production ready and default was to run using a privileged container. A privileged containers root is mapped to the host root so breaking out of the container means that you get root privileges on the host while breaking out of an unprivileged container means you are only gaining privileges of a normal user but with a set of very strict and harmless privileges. See https://linuxcontainers.org/lxc/security/

 

[Dunuin]

So they are essentially saying, don't use containers, and VMs are more secure than a container? Aren't containers isolated as well, if you use them in unprivileged mode?

VMs are completely isolated and unprivileged containers are still only half isolated. So if you want maximum security I would always use VMs and not LXCs. But unprivileged LXCs are good enough if you are running low end hardware and need to do compromises.

 

[mir]

But unprivileged LXCs are good enough if you are running low end hardware and need to do compromises.

Not only that but if you run micro services or docker based services spooling a complete operating system is a waste of ressorces. Only if your micro services or docker based services must have a 100% uptime then go for a VM since LXC containers cannot live migrate still.

 

[tripflex]

For anybody coming here later and seeing this, if the container is unprivileged just goto Options from the container menu (main UI), click on Features then select keyctl and nesting, save, voila!

 

[majorgear]

For anybody coming here later and seeing this, if the container is unprivileged just goto Options from the container menu (main UI), click on Features then select keyctl and nesting, save, voila!

This is so easy, it should be illegal.

Thanks for sharing this info. I have been running docker inside vm’s because I thought that was the only way to do it.

 

[majorgear]

I had the same issue with a privileged container.

Without nesting I got this error executing "docker run hello-world"

error mounting "proc" to rootfs at "/proc": mount proc:/proc (via /proc/self/fd/6)

With nesting enabled, it worked fine.

 

[RolandK]

don't use lxc for this, use a virtual machine

 

[majorgear]

don't use lxc for this, use a virtual machine

This is the correct way to run containers.

 

[cloudycurse]

don't use lxc for this, use a virtual machine

This thread was a whole discussion on why you can use docker in an lxc if unpriviledged. What is your reason for not doing it?

 

[Dunuin]

What is your reason for not doing it?

The Wiki tells you some arguments why not to use LXCs to run docker:

If you want to run application containers, for example, Docker images, it is recommended that you run them inside a Proxmox Qemu VM. This will give you all the advantages of application containerization, while also providing the benefits that VMs offer, such as strong isolation from the host and the ability to live-migrate, which otherwise isn’t possible with containers.

I don't trust most docker container maintainers, so I want them to run as isolated and as unprivileged as possible. I for example could publish a nice container, a lot of people will then try and use it and after one year I add some ransomware to that container that wipes every data it can access locally or on network shares of your LAN. You never know which maintainer is trustworthy and who is not and the only way to make sure you are not getting shipped malicious code with the container is to read and understand each line of code of the container before pulling a new version. No one is doing that...you are either missing the knowledge to understand in detail what the code is doing or you just don't get the time to analyze millions of lines of code each month when upgrading a container.