Docker on LXC or VM?

Discussion in 'Proxmox VE: Installation and configuration' started by n1nj4888, Mar 29, 2019.

Tags:
  1. n1nj4888

    n1nj4888 Member

    Joined:
    Jan 13, 2019
    Messages:
    44
    Likes Received:
    0
    I'm considering running docker (and some docker containers) off of proxmox and wondered whether running docker inside an LXC or a VM would be better? I noticed from the following URL that it mentions that running docker in a VM would be better but gives no reason why this would be the case?

    https://pve.proxmox.com/wiki/Linux_Container

    "If you want to run micro-containers (with docker, rkt, …), it is best to run them inside a VM."

    Also, for LXC more generally and likely a noob question but ...If LXC uses the same kernel as the proxmox host, I'm not sure I understand how there are Ubuntu 18.04.1 (no 18.04.2 template yet?) and Ubuntu 18.10 templates when these use the 4.18.0-16-generic (HWE) kernel? Wouldn't there be issues with the older proxmox kernel (4.15.18-10-pve) running these OS?

    Thanks!
     
  2. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    3,515
    Likes Received:
    318
    docker uses kernel features for encapsulating containers (like lxc) so nesting this is not that easy, but doable when you acitvate the 'nesting' feature (Container->options->features)

    those templates do not load their own kernel but use the host kernel, so you have an ubuntu 18.04 but with the pve kernel

    in general, no there should not be any issues
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. n1nj4888

    n1nj4888 Member

    Joined:
    Jan 13, 2019
    Messages:
    44
    Likes Received:
    0
    Thanks for this - Is there any documentation on this nesting feature and what it does / how? I'm interested since I would have thought it standard to run docker inside an LXC container with the docker instance (and its containers) being transparent to the host i.e. as if the were in a VM?
     
  4. LnxBil

    LnxBil Well-Known Member

    Joined:
    Feb 21, 2015
    Messages:
    3,694
    Likes Received:
    329
    No, LXC is lightweight virtualization. If you're going to run Docker in a production setup, use a VM. Also best do use a Docker-centric distribution like RancherOS for orchestration. You will have a hard time to get Rancher to work in LXC, if at all.

    Docker is a PaaP system that runs best on a IaaS solution like PVE if you use VM, so "real" virtualization.
     
  5. n1nj4888

    n1nj4888 Member

    Joined:
    Jan 13, 2019
    Messages:
    44
    Likes Received:
    0
    Thanks for the replies - Docker on VM it is then!

    Thanks!
     
  6. loiclacombe

    loiclacombe New Member

    Joined:
    Apr 26, 2019
    Messages:
    3
    Likes Received:
    0
    So far in my small scale tests of production setups, RancherOS has been more of a pain than an asset. You would be better off using your favorite Linux OS (Debian or CentOS, for example), then just add your target docker version on top of it. I mostly use Ansible to administrate my VMs and RancherOS is not exactly well equipped for such automated administration. And the main point of Docker is to have a boring and stable OS on which you can install whatever container you want. On Proxmox 5.4 with qemu-kvm, RancherOS does not really work out of the box. Default install has Proxmox cloud-init not working, I had to craft a qemu-guest-agent docker image, and after 6+ hours I was not able to fix RancherOS sluggish startup (+10 minutes) where any debian stretch with Docker boots in seconds. Oh and RancherOS documentation is lacking.
     
    #6 loiclacombe, Apr 26, 2019
    Last edited: Apr 26, 2019
  7. LnxBil

    LnxBil Well-Known Member

    Joined:
    Feb 21, 2015
    Messages:
    3,694
    Likes Received:
    329
    That's totally true, if you're deep into Linux. I also do it this way for myself.

    RancherOS has advantages for deployment with docker-machine, so it is fit for mass deployment for people not so familiar with Linux and Docker, but want to have a good tool integration.
     
  8. alexskysilk

    alexskysilk Active Member

    Joined:
    Oct 16, 2015
    Messages:
    555
    Likes Received:
    58
    I'm interested to hear about your experiences. I've been sandboxing k8s clusters using rancheros for the building blocks; what challenges have you run into? what are the limitations of using ansible on the endnodes? (not at that stage yet but as you can preload the keys at deployment I wasn't expecting any challenges at all)
     
  9. loiclacombe

    loiclacombe New Member

    Joined:
    Apr 26, 2019
    Messages:
    3
    Likes Received:
    0
    Promox/Qemu integration is quite bad. Like I said, to get the IPs of the VM displayed in Proxmox, I had to deploy a docker container with --network host. But then, shutting down the VM became impossible as a docker container cannot trigger a host shutdown. So either I cannot shut down easily my VM, either I have to assign manually fixed IP to the VMs, which is unusable. And frankly, RancherOS documentation is terrible, so I did not try to make a magical, console-like container to run on the system docker (it seems to be possible, but complicated). Then Ansible defaulted to not working (needs python interpreter in the console), unless you switch to Debian or Ubuntu console, or do some crazy Ansible stuff (run ansible command in a container, tried, was awfully complicated). And I had to tweak cloud-init configuration in the installation to get cloud-init to work, but I had to reverse engineer the openstack cow2 image to figure out the proper configuration. Then I wanted to add a second virtual hard drive to the VM, for docker volume storage and I did hit a wall as was is just awful to automate with Ansible.
    And now that K3Os is out, I think that the RancherOS project will soon become obsolete, once that high-availability is fixed.
    But it seems like a widespread issue in docker-first OSes:
    • CentOS Atomic has no real documentation about how you can tweak the OS. Spent 2 hours reading obsolete tutorial about non-existing or renamed or deprecated packages.
    • Free CoreOS had almost zero documentation and I has become even worse since it was bought by Red Hat. Many happy proof of concept around on the web, but it looks like paying customers either keep quiet or do not exist.
    • Flat Car Linux (CoreOS fork), I dunno. I still cannot see how I could integrate with qemu-guest-agent and it looks like I am going to hit the same wall as rancheros
    So, so far, what I did is create a Debian KVM template with qemu-guest-agent. It just works fine, can be easily administrated, updated. It can be tweaked to be light (delete man documentation, for example), like in the debian:stretch-slim docker image. And I could run hundred of these with no real effort or could tighten up security with some more work.
    So far, I still do not see the point in those Docker-first OSes.
     
  10. alexskysilk

    alexskysilk Active Member

    Joined:
    Oct 16, 2015
    Messages:
    555
    Likes Received:
    58
    Ahhh I understand. use this instead of the vanilla image: https://github.com/rancher/os/releases/download/v1.5.1/rancheros-proxmoxve.iso

    it has proxmox compatible qemu tools preinstalled.

    I imagine installing a python binary would be simple enough but you're right that there is no package management integrated. As my intention is to use kubectl and not ansible this doesnt pose a problem for me, but its probably solvable. The base OS for Rancher is debian/ubuntu.

    true enough, I am planning to reroll my lab with it. Since I already had the nodes prejoin a K8s cluster via cloudconfig it will serve the same purpose, just simplify it one step. The biggest challenge was to have unique names assigned at deployment but I gather this will no longer be an issue.
    smaller, faster, effectively stateless, require no upkeep.
     
  11. loiclacombe

    loiclacombe New Member

    Joined:
    Apr 26, 2019
    Messages:
    3
    Likes Received:
    0
    This one link is god-sent, thank you!
    About the upkeep part, as you need to purge old docker images and obsolete volumes, you will need to clean the stateful part, and you will accumulate these, that is not true. As I have yet to use Kubernetes, perhaps this feature is already provided.
    And the bad part is there no real community support in case anything goes wrong. It is a niche OS.

    I will probably evaluate docker-machine-driver-proxmox-ve once I have audited its code; It could be a neat solution on a dedicated LVM volume. Though I do not not know if docker-machine is really production-ready.

    FYI, I do use Docker in my small production servers, both on Openstack and on Proxmox. I did not use Kubernetes because it was too expensive on small scale installations before K3S happened. These are not missing-critical as my beloved VMs hosting various services on Docker, though I like these services to be rock-solid, and easily restored in case anything goes wrong. And there is actually no maintenance as Debian unattended upgrades do just work.
     
    #11 loiclacombe, May 1, 2019
    Last edited: May 1, 2019
  12. alexskysilk

    alexskysilk Active Member

    Joined:
    Oct 16, 2015
    Messages:
    555
    Likes Received:
    58
    I run diskless ;)
     
  13. LnxBil

    LnxBil Well-Known Member

    Joined:
    Feb 21, 2015
    Messages:
    3,694
    Likes Received:
    329
    You can use it to do bare-metal installations, but unfortunately requires RancherOS in the patched version @alexskysilk provided. We also tried the Boot2Docker base image, but that also required patching which was refused upstream. I don't see the point in providing maintenance for stuff I do not use and especially providing install images for third party OSes.

    I have to say, for my K8s test deployments, I also used k8s on Debian Stretch, because of the easy integration in preexisting automation environment. Auto-provisioning via netboot in PVE is also very easy, so most of the benefits of these Docker-first OSes are bogus for a fully-automated Debian environment.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice