Container Linux in a KVM

[MimCom]

My first experiment with CoreOS / Container Linux. Spun up a KVM with the current stable ISO as its CDROM. Booted, then created a static.network per https://coreos.com/os/docs/latest/network-config-with-networkd.html and restarted systemd-network. Verified connectivity. Then used sudo coreos-install to install on the disk. Not sure what to do with the localhost-login prompt. I'm guessing there's a much better way to do this. Anyone done this on PVE and have a tip sheet?

thanks~

 

[LnxBil]

Running CoreOS is not different in PVE than in VMware or any other hypervisor. I'm unsure what you expect from PVE at this point.

If CoreOS is compatible with Docker Machine, you may want to wait until the docker machine driver for PVE is ready. Then, you can spin up a docker VM with only one command.

 

[MimCom]

This started with me trying to find a way to run Docker in an LXC while leaving the container (at least mostly) confined. That turned out to be nontrivial.

KVM overhead is significant, so I've been looking at ways to reduce that, along with the hassle of continuously updating VMs. Container Linux seemed to be a good option there, and the Docker-based software we are hosting does support it.

This is really more of a "how do I get from booting to operating on CoreOS?" question. Just not familiar with the paradigm. Seems that some sort of Ignition config might be the right way to do this.

There is a guide to using naked QEMU https://coreos.com/os/docs/latest/booting-with-qemu.html but I'm not sure how that might interact with (or conflict with) PVE's GUI management.

I saw the posts on Docker Machine (and thank you for undertaking that effort) but I'm not entirely certain how that will all work with a simple (non-swarm-like) app deployment.

 

[LnxBil]

This started with me trying to find a way to run Docker in an LXC while leaving the container (at least mostly) confined. That turned out to be nontrivial.

In fact, it is trivial, just add these entries to an LX(C) container configuration:

lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:

install Docker inside of the container and load the modules:

modprobe aufs
modprobe overlay

inside of your PVE host. It worked for me with Debian Stretch, but you will have no security at all. I'm also running at least one container this way, but not on a production machine.

 

[MimCom]

In fact, it is trivial
...
but you will have no security at all.

Right -- hence my (at least mostly) confined qualifier above. I've asked the app developers to craft an appropriately limited AppArmor profile, but they have more important priorities. This really seems like something that LXC and Docker should have worked out by now.

 

[verulian]

Being able to use Dockerized apps within LXC containers is a pretty important function that we need. Just wanted to mention that we'd also like to see this brought into being a first class and supported citizen of the Proxmox environment.

 

[verulian]

These are excellent instructions @LnxBil, but I'm hitting an issue still - could you take a look and advise accordingly?:

/usr/bin/docker: Error response from daemon: Could not check if docker-default AppArmor profile was loaded: open /sys/kernel/security/apparmor/profiles: permission denied.
Your Docker installation is not working correctly

 

[LnxBil]

Being able to use Dockerized apps within LXC containers is a pretty important function that we need. Just wanted to mention that we'd also like to see this brought into being a first class and supported citizen of the Proxmox environment.

People always want to run Docker containers in LXC on one host and it's so sad to see that they do not understand what the real power behind Docker is: the orchestration, autoscaling and HA-features present in e.g. Kubernetes. You don't want to run Docker in Proxmox VE, it's a IaaS solution. Docker wants to run on a PaaS solution, so there will never be an integration of Docker in PVE because it does not make sense at all. I stated this multiple times in all threads about Docker (and the Proxmox staff concurs). The mindset behind LXC and Docker are so profoundly different that they cannot be integrated meaningfully. If you really want to run Docker in LXC, you're good to go, but it breaks a lot of security features and is therefore best run in a VM. I also wrote the docker-machine driver for proxmox which allows to easy spin up docker machines to work with in a almost fully automated way to get the easiest way possible to run Docker on PVE.

Best is to run a IaaS cluster with PVE and on that multiple machines hosting the Kubernetes cluster and the containers on there. You will have instant autoscaling, fault-tolerance, ingress routing and whatever Kubernetes throws also your way and are running an enterprise environment with Docker.

 

[verulian]

@LnxBil so the reason I foresee using Docker in LXC is simply for testing/staging purposes on isolated instances that have no security risks.

However, would it be possible to have a "Create DC" at the top for a Docker container as well in Proxmox via your driver implementation? I could see that being fully acceptable to make the instantiation of Docker instances 1st class citizens...

 

[verulian]

People always want to run Docker containers in LXC on one host and it's so sad to see that they do not understand what the real power behind Docker is: the orchestration, autoscaling and HA-features present in e.g. Kubernetes. You don't want to run Docker in Proxmox VE, it's a IaaS solution. Docker wants to run on a PaaS solution, so there will never be an integration of Docker in PVE because it does not make sense at all. I stated this multiple times in all threads about Docker (and the Proxmox staff concurs). The mindset behind LXC and Docker are so profoundly different that they cannot be integrated meaningfully. If you really want to run Docker in LXC, you're good to go, but it breaks a lot of security features and is therefore best run in a VM. I also wrote the docker-machine driver for proxmox which allows to easy spin up docker machines to work with in a almost fully automated way to get the easiest way possible to run Docker on PVE.

Best is to run a IaaS cluster with PVE and on that multiple machines hosting the Kubernetes cluster and the containers on there. You will have instant autoscaling, fault-tolerance, ingress routing and whatever Kubernetes throws also your way and are running an enterprise environment with Docker.

@LnxBil would you mind giving some more explicit instructions on installation? For example, putting it within $PATH would essentially indicate, on Proxmox, sticking the downloaded bin to any of the defaults: "/root/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

But are those sane? Is there a better location or literally should we just drop it into `/usr/local/bin` for example?

Additionally `docker-machine create --driver proxmox-ve --help | grep -c proxmox 16` fails since `docker-machine` doesn't exist... and so we obviously must install Docker first in a compatible way. Could you please include this step within the README.md as a precursory step?

 

[LnxBil]

@LnxBil would you mind giving some more explicit instructions on installation? For example, putting it within $PATH would essentially indicate, on Proxmox, sticking the downloaded bin to any of the defaults: "/root/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

But are those sane? Is there a better location or literally should we just drop it into `/usr/local/bin` for example?

Additionally `docker-machine create --driver proxmox-ve --help | grep -c proxmox 16` fails since `docker-machine` doesn't exist... and so we obviously must install Docker first in a compatible way. Could you please include this step within the README.md as a precursory step?

The program is supposed to run on a machine with docker-machine installed, e.g. your desktop and it is an additional driver for docker-machine. You have to put it in a path on your machine that is searched by $PATH. Depending on your local environment, the installation may vary. Please refer to the original documentation.

 

[verulian]

Interesting, I clearly have not understood the intention here. So does `docker-machine` run on a workstation your driver facilitates allocation of a docker instance via `docker-machine` on a Proxmox server? I'm still getting some crossed wires... Could you elaborate on the process more fully with a usecase completely delineated @LnxBil?

Just experimenting here, I did try installing `docker-machine` on the Proxmox server as per its Linux installation instructions. I also put `docker-machine-driver-proxmox-ve` into `/usr/local/bin`, but am getting this error:
`Driver "proxmox-ve" not found. Do you have the plugin binary "docker-machine-driver-proxmox-ve" accessible in your PATH?` when I attempt to run `docker-machine create --driver proxmox-ve --help`

 

[LnxBil]

Interesting, I clearly have not understood the intention here. So does `docker-machine` run on a workstation your driver facilitates allocation of a docker instance via `docker-machine` on a Proxmox server?

That is the idea. The big picture is to create Docker machines on demand and add them to a PaaS cluster. RancherOS does this already with Kubernetes, so that you can create whole cluster by just a few steps on Proxmox VE. Docker-machine does this already for all major virtualisation platforms including AWS, VMware, etc. the missing part was Proxmox VE, and therefore I created the driver.

Just experimenting here, I did try installing `docker-machine` on the Proxmox server

The point is to install the VM from a client, not the server. You just installed Docker (as a requirement) on your PVE, so you already have Docker on PVE (not in a VM, but on the hypervisor) and don't need the driver. Maybe it only installed only the docker-cli packages and not the daemon packages.

I also put `docker-machine-driver-proxmox-ve` into `/usr/local/bin`, but am getting this error:

Then please post the output of the following commands (in CODE-tags):

ls -l /usr/local/bin/docker-machine-driver-proxmox-ve
echo $PATH

 

[tstivers]

People always want to run Docker containers in LXC on one host and it's so sad to see that they do not understand what the real power behind Docker is: the orchestration, autoscaling and HA-features present in e.g. Kubernetes. You don't want to run Docker in Proxmox VE, it's a IaaS solution. Docker wants to run on a PaaS solution, so there will never be an integration of Docker in PVE because it does not make sense at all.

This is absolutely not true. Kubernetes provides the autoscaling, HA, etc features. Docker solves a completely different problem. Docker allows developers to run the exact same setup on their local machine as the production server, solving the infamous "works on my machine" problems. It also packages an application and everything it needs to run in a minimal container. Docker doesn't solve problems such as autoscaling, HA, etc. However, it does work really nicely with systems that do implement that.

Many services, such as nginx, mysql, pihole, etc, are available as docker images. LXC simply cannot compete with the sheer amount of images available for docker, often times direct from the developers of the software the image runs.

https://www.infoworld.com/article/3310941/why-you-should-use-docker-and-containers.html

 

[tstivers]

That said, running Docker within LXC *is* pointless. Just run CoreOS as a VM and Portainer within Docker to manage it. It'll be a much better setup than Proxmox will ever have. And CoreOS' footprint is negligible.

 

[LnxBil]

That said, running Docker within LXC *is* pointless. Just run CoreOS as a VM and Portainer within Docker to manage it. It'll be a much better setup than Proxmox will ever have. And CoreOS' footprint is negligible.

That's what I meant with the "behind Docker"-part I wanted to describe that the whole ecosystem is the big part, not just running Docker containers.

Docker allows developers to run the exact same setup on their local machine as the production server, solving the infamous "works on my machine" problems.

Have you ever seen this? I haven't. Most production containers I've ever deployed where stripped down development containers without any test and debugging capabilities.