LXC ZFS + docker overlay2 driver
- Thread starter Kiwwiaq
- Start date
Best practice to run Docker, according to Wiki, is to use a VM and not a LXC:
But I guess you meant best practice when using a LXC.
Yes, wasn't that what i wrote - docker/zfs/lxc
I know VM's are recommended but they are very resource heavy and feel sluggish, particularly for this situation (with a container environment inside a virtual machine) - but I realize that may just be my subjective impression.While not as efficient, the benefit of a docker VM is, that PVE updates won't break your docker installation all the time. So great if you want stability and availability with less time wasted fixing stuff. Just a bit RAM hungry when you prefer a dedicated docker VM for each container stack to be able to easier migrate your single services between nodes.Yes, wasn't that what i wrote - docker/zfs/lxc
Or as always ... just install Docker on your PVE host itself. Then you WILL have all benefits from ZFS and WILL break everything if you're doing something wrong or got hacked ... just as with LX(C) containers, but with more benefits.
They're great, but not for running Docker and have their special and limited use case. If you have to break every security barrier in order to get things running inside of LX(C) containers you're doing something wrong. In an HA environment, LX(C) containers are currently also very limited ... no live migration.
Actually I don't need live migrations. Which security barrier do you mean are broken for Docker? I run it unprivileged. Nesting is enabled, but I think that just means it will allow running one type of virtualization inside another?
Nesting for example weakens the isolation by letting the LXC access the hosts /proc and /sys filesystems. Similar thing with keyctl and fuse. And editing user/group remapping is lowering isolation too. And all the cgroup stuff that you edit to be able to use the hosts hardware in a LXC.
Last edited:
It looks like fuse-overlayfs will still be needed if you don't use a 'raw' disk or if you are bind mounting the '/var/lib/docker' directory - like I'm doing. I would really rather not lose this ability as I can make my LXC pretty small and just put the docker directory somewhere else. If the LXC ever fails, I don't have to go through the time-consuming effort of recreating all my containers.
As far as weakening security too much, I agree for a business or enterprise scenario, but in a home lab, I find that running lean is much more of a priority, and LXC's, even if they're privileged, nesting, ID remapped, etc., provide just that.
These commits in ZFS :
* https://github.com/openzfs/zfs/commit/86db35c447aa3f4cc848497d78d54ec9c985d1ed
* https://github.com/openzfs/zfs/commit/dbf6108b4df92341eea40d0b41792ac16eabc514
Some more history in the PR : https://github.com/openzfs/zfs/pull/9414
Working thanks to ZFS >= 2.2.0
* Proxmox 8.1 default kernel 6.5 has it.
* Proxmox 7.4 can't tell which exact kernel.
Well the thing is I had this working way back on Proxmox kernel 6.2.6 (https://forum.proxmox.com/threads/u...ocks-up-pve-host-processes.108362/post-547436), so there must be some other factor...? Perhaps the OS running in the LXC? Since I am running Arch Linux which usually has the most recent release for most packages in its repository.