Docker support in Proxmox

[chr1s]

sorry that I reopened this old discussion with my comment(s). People still seem to have strong opinions about how containers fit in their infrastructure. Didn’t mean to start a religious discussion about it.

To conclude from my side, I really like the overall direction Proxmox is taking in recent times (firewall, cloudinit, etc. But also the new backup server!). I think many people are using Proxmox as a lightweight private cloud these days. At least it prevents me from running a fat, complex OpenStack installation. I only wish the cloudinit support would be more complete (set userdata in the web interface, etc) and there was officially supported Terraform support.

Regarding containers: On any public cloud OCI/Docker containers run in VMs (alone for security reasons). So what Proxmox lacks a bit here is orchestration/management of that. I would wish for better support here (similar to the direction VMware is taking with their Kubernetes features), but I can understand if it’s out of scope.

But better cloudinit support would be really important imho so that Proxmox can at least fully serve a AWS EC2 like usecase and users can then run Kubernetes distros on top. Things are close, but not yet fully there.

 

[hr556]

sorry that I reopened this old discussion with my comment(s). People still seem to have strong opinions about how containers fit in their infrastructure. Didn’t mean to start a religious discussion about it.

To conclude from my side, I really like the overall direction Proxmox is taking in recent times (firewall, cloudinit, etc. But also the new backup server!). I think many people are using Proxmox as a lightweight private cloud these days. At least it prevents me from running a fat, complex OpenStack installation. I only wish the cloudinit support would be more complete (set userdata in the web interface, etc) and there was officially supported Terraform support.

Regarding containers: On any public cloud OCI/Docker containers run in VMs (alone for security reasons). So what Proxmox lacks a bit here is orchestration/management of that. I would wish for better support here (similar to the direction VMware is taking with their Kubernetes features), but I can understand if it’s out of scope.

But better cloudinit support would be really important imho so that Proxmox can at least fully serve a AWS EC2 like usecase and users can then run Kubernetes distros on top. Things are close, but not yet fully there.

That's exactly the thing, I would also love more cloud-init capabilities in terms of user-data directly setable in the WebUI as well as feature-parity with the cloud-init implementation in Openstack.

Reason: many KVM images with cloud-init support only work well using Openstack, with Cloudstack or any other KVM-based infra it's often a workaround-mess to get the same functionality.

 

[morph027]

I only wish the cloudinit support would be more complete (set userdata in the web interface, etc) ...

You can at least use cloudinit data from snippets, which can be edited through the webinterface.

https://pve.proxmox.com/wiki/Cloud-Init_Support#_custom_cloud_init_configuration

 

[Ramalama]

Very old thread and it's basically dead, so i don't need to reply to anyone

But with openzfs 2.2, docker will be working flawless inside lxc containers.
I'm preparing a test right now, just have some troubles atm compiling the kernel with zfs for Proxmox. Basically im trying to compile the 6.3 kernel with openzfs 2.2rc1.

However long story short, i think that running docker/kubernetes/containerd is anyway better suited inside an lxc containers.
I mean you can do backups and migrate them etc...
Imagine how much work that would be for the Proxmox devs, to implement native docker support that is cluster"izable"

However, i would rather suggest to implement lxd, to proxmox.
Lxd utilizes lxc at the same way as Proxmox does, but it expands lxc containers to be clusterizable.
Means allows for proper HA, without the shutdown/migrate/restart route + some other nice features.
Lxd utilizes kvm either, which is useless to us, but it adds at least mainly a lot features to lxc.

Especially since docker won't be an issue anymore on lxc containers with openzfs.

Btw, im running docker containers on Unprivileged lxc containers on ext4 backend storage since almost forever, and it worked always very perfect.

Lately im running even docker containers inside unprivileged lxc container with overlay2 and nesting only on zfs.
And it works with 98% of the containers amazingly well either.
I stumbled only over 2 docker images so far, that didn't worked with this method, one of them is speedtest-tracker.

However, it's a suggestion about lxd. So if it comes or not, doesn't matter for me very much either, since im extremely happy with Proxmox anyway.

Cheers

 

[LnxBil]

But with openzfs 2.2, docker will be working flawless inside lxc containers.

That's great news. I've been wanting this for years after experiencing zfs delegation in jails with FreeBSD.

 

[stefanobertoli]

Using with success docker inside unprivileged lxc on top of ceph. Nothing too complex but never had an issue. As I wrote in this thread some time ago, + 1 to not use docker directly un proxmox host.

 

[Gramdalf]

That's great news. I've been wanting this for years after experiencing zfs delegation in jails with FreeBSD.

Very similar situation here. Just to add to the previous comment, openzfs2.2 does multiple things:

• OverlayFS (`overlay2`) is functional on top of zfs 2.2
  • This will hopefully come relatively soon, and should allow the default docker install (which uses overlay2) to function in an LXC container stored on a ZFS filesystem, no ext4 zvols required
  • Bonus tip: You might need to update your pool to a new zfs version for it to work properly
• Support for ZFS managment within containers
  • I suspect this will take longer to be integrated with LXC and by proxy Proxmox - but from a quick look at the patch notes ("...added namespace delegation support for containers"), it seems like there are a set of features that would enable containers (LXC, perhaps even docker?) to manage a ZFS pool (or a subset of it, like a dataset/subvol - not exactly sure on the details there). I'm not familiar with BSD Jails, but it sounds quite similar.

I might create a separate forum post about this, but a note for the Proxmox Devs:
Like I mentioned above, it would be nice if the pve version of zfs 2.2 could be released sooner rather than later - It would be nice to have `overlay2` support at the least even if proper support for namespace delegation (among other things) takes longer to implement.
Edit:
Found this thread after posting: https://forum.proxmox.com/threads/o...-2-for-proxmox-ve-8-available-on-test.135635/
I'm not sure when it will be available in the `pve-no-subscription` repository, but it's available in the testing repository.

 

[Ramalama]

Very similar situation here. Just to add to the previous comment, openzfs2.2 does multiple things:

• OverlayFS (`overlay2`) is functional on top of zfs 2.2
  • This will hopefully come relatively soon, and should allow the default docker install (which uses overlay2) to function in an LXC container stored on a ZFS filesystem, no ext4 zvols required
  • Bonus tip: You might need to update your pool to a new zfs version for it to work properly
• Support for ZFS managment within containers
  • I suspect this will take longer to be integrated with LXC and by proxy Proxmox - but from a quick look at the patch notes ("...added namespace delegation support for containers"), it seems like there are a set of features that would enable containers (LXC, perhaps even docker?) to manage a ZFS pool (or a subset of it, like a dataset/subvol - not exactly sure on the details there). I'm not familiar with BSD Jails, but it sounds quite similar.

I might create a separate forum post about this, but a note for the Proxmox Devs:
Like I mentioned above, it would be nice if the pve version of zfs 2.2 could be released sooner rather than later - It would be nice to have `overlay2` support at the least even if proper support for namespace delegation (among other things) takes longer to implement.

Pve kernel 6.5 with openzfs 2.2 is released.
And all the docker issues are solved, tested here already.

 

[Gramdalf]

Pve kernel 6.5 with openzfs 2.2 is released.
And all the docker issues are solved, tested here already.

Sort of - it requires the `pvetest` repository, it hasn't been released to the "stable" yet. Updated the original post.

 

[Ramalama]

Sort of - it requires the `pvetest` repository, it hasn't been released to the "stable" yet. Updated the original post.

If i can make you feel better, im running that kernel/openzfs test already on 5 servers here, its perfect

 

[SInisterPisces]

Pve kernel 6.5 with openzfs 2.2 is released.
And all the docker issues are solved, tested here already.

If we're setting up a new CT to run Docker, do we need to do anything specific to the CT itself to get things working properly? Or, do we just install and use Docker as if it were bare metal?

I'm assuming I'll need to upgrade my zpool(s). I've never had to do that before. A bit scary.

 

[Ramalama]

If we're setting up a new CT to run Docker, do we need to do anything specific to the CT itself to get things working properly? Or, do we just install and use Docker as if it were bare metal?

I'm assuming I'll need to upgrade my zpool(s). I've never had to do that before. A bit scary.

Yeah you need to upgrade your zpools.
But i did that already on ton of systems, almost all in live environments.
The upgrade takes like a millisecond and you don't need to shutdown any vm on the running pool or anything.

About upgrading your pools i wouldn't worry even a second.
Only downgrades are impossible, means your pools won't work anymore with anything below zfs 2.2.

Otherwise im running docker almost always on unprivileged lxc containers without anything special in them.
Just uninstalling apparmor always in all my lxc containers, cause apparmor is simply stupid.
The idea is great but they didn't thought to make the apparmor configuration easy.

On privileged lxc containers docker runs without apparmor great either.

Cheers

 

[SInisterPisces]

Yeah you need to upgrade your zpools.
But i did that already on ton of systems, almost all in live environments.
The upgrade takes like a millisecond and you don't need to shutdown any vm on the running pool or anything.

About upgrading your pools i wouldn't worry even a second.
Only downgrades are impossible, means your pools won't work anymore with anything below zfs 2.2.

Otherwise im running docker almost always on unprivileged lxc containers without anything special in them.
Just uninstalling apparmor always in all my lxc containers, cause apparmor is simply stupid.
The idea is great but they didn't thought to make the apparmor configuration easy.

On privileged lxc containers docker runs without apparmor great either.

Cheers

Thanks! If you were building an LXC from scratch to work as a Docker host (or a VM, I guess), which distro would you use?

I'm planning to use the Debian LXC template; I prefer Ubuntu but don't like that I can't uninstall snapd anymore. I could just disable it, but it makes me wonder what they'll force on users in the future.

 

[Gramdalf]

Thanks! If you were building an LXC from scratch to work as a Docker host (or a VM, I guess), which distro would you use?

I'm planning to use the Debian LXC template; I prefer Ubuntu but don't like that I can't uninstall snapd anymore. I could just disable it, but it makes me wonder what they'll force on users in the future.

I personally would use either alpine or NixOS. NixOS is nice since you can manage the entire LXC configuration declaratively, while Alpine is good because it is so much more lightweight than pretty much anything out there. You need to enable the alpine edge repository to install a recent version of docker, however.

 

[billa44]

Does anyone interested to realize docker support in Proxmox?

I do

 

[keeka]

Just uninstalling apparmor always in all my lxc containers

I have not removed apparmor but I have noticed it is not started in PVE debian containers:

Bullesye

~# journalctl -b | grep apparmor
Aug 10 22:31:22 ct1 apparmor.systemd[62]: Not starting AppArmor in container

Bookworm

~# journalctl -b | grep apparmor
Aug 10 23:40:03 ct2 systemd[1]: apparmor.service: Starting requested but asserts failed.
Aug 10 23:40:03 ct2 systemd[1]: Assertion failed for apparmor.service - Load AppArmor profiles.

For no other reason than the logging above, it seems to be handled more gracefully in Debian 11 container template.

A couple of quirks I have noticed, running docker in a (debian) CT:

1. Disk charts not depicting any IO (since PVE8). However this started working again following a recent upgrade (perhaps kernel and/or libpve-perl?).

2. Network IO chart not showing expected thruput. The chart also shows a strange cumulative effect. IO linearly increasing between guest restarts. This seems to be associated with tcp keepalive traffic generated by portainer. If I stop portainer, that sawtooth pattern goes. But network traffic still seems under reported.


But for the sheer convenience and low resource use I am finding docker in lxc to be working well.

 

[verulian]

I also would love to see Docker supported in addition to LXC on Proxmox. Not having an integrated Docker capacity in the admin interface is just horrible.

 

[Ramalama]

I think tbh, that we will regret it.
Docker has Compared to LXC Containers a huge amount of Options/Features and i don't believe that we will get anything usefull in the GUI, so we will still prefer the CLI.
I mean we still don't even have Numa Support, which makes Proxmox mostly useless on Dual-Socket or Newer Single-Socket AMD Plattforms like Genoa compared to other Hypervisors. It's not completely a Proxmox Fault, more of a KVM/Qemu still has no support issue, but there is a ton of Improvements even with Scripts that the Proxmox-Team could do.

Additionally in the last time there is not really any Development going on, maybe Summer-Break or something.

And yeah i know, im a very pesimistic person. But just in my Opinion there are other things that should be improved first, before we get something new half-backed that steals time from the Devs. Sorry :-(

But i understand very well, that in a perspective of a Home-User, Proxmox is Completely Amazing and just Docker-Support is missing.

Cheers

 

[alexskysilk]

Docker=infrastructure as code
proxmox=type 1(ish) hypervisor.

apples and oranges, which is to say different targeted usecases. you CAN deploy docker workloads to a pve environment (at the host/vm/lxc level) but to make full use out of docker you really want the whole enchilada (git, jenkins/terraform, kubernetes/swarm, etc) this space already is served by existing solutions that are designed for it.

 

[verulian]

You can think that all you want @Ramalama, but at the end of the day it's essential in the world today. I hate Docker, but I have to use it - all the time. I, for my life, cannot get myself to remember the shell incantations to operate it. It drives me nuts. Having a unified environment in Proxmox would be an actual killer feature. I want a simple solution, not all of the "enchilada" that @alexskysilk notes.