Trouble wrapping my head around containers

[happnatious1]

Background:
Was running ESXI 6.5 and VMWare Workstation for years but a recent ESXI upgrade killed usb audio passthrough. Found Proxmox and was able to export all my
VMs to it and I am impressed. Uptime as of this post is 8 days. Server is 32 cores with 60g of memory.

Problem:
I have no experience with containers. I tried creating a Debian 12 container in proxmox and I added Portainer to it but it seems that I cant add any other containers to it. I would like to have a ddns container, a guacamole container, and a RustDesk container. Do I need to install a full Debian vm with docker and portainer to host multiple containers, or does each docker container need to be on it's own container within proxmox? Am I approaching this all wrong?

Can't seem to wrap my head around containers in Proxmox.

 

[Ramalama]

If your lxc container with docker inside is running on zfs storage, well you need to enable nesting at least.
But even then not all docker container will work. Some will, some won't on zfs as storage.

The easiest way is to create an ext4 dataset...
Means simply add an dataset with a fixed size, like 50gb for example and format that dataset as ext4, then mount it somewhere on your host and add that directory under the datacenter tree as container storage.

Then simply move the drive of your lxc container to that new ext4 storage.

After that, all docker containers should work.

 

[happnatious1]

Thanks Ramalama,

I am indeed on zfs mirror storage.
I found https://github.com/afro-systems/lxc-guac-setup for getting guacamole running on a LXC container unfortunately it requires Debian 10 which proxmox doesn't have a template for.

I'd like to keep my zfs mirrors in case of drive failure so it looks like I'll have to stick with full VMs instead.

Thank you for the insight.

 

[Ramalama]

Thanks Ramalama,

I am indeed on zfs mirror storage.
I found https://github.com/afro-systems/lxc-guac-setup for getting guacamole running on a LXC container unfortunately it requires Debian 10 which proxmox doesn't have a template for.

I'd like to keep my zfs mirrors in case of drive failure so it looks like I'll have to stick with full VMs instead.

Thank you for the insight.

?
You don't need to reformat any drives...
You can keep your zfs mirror.

All you need to do is creating an zfs dataset and format it as ext4.

Imagine it like you create a folder on your zfs storage, but only this folder is ext4. It's ext4 ontop of zfs xD

A lot of people do this, me including.
So nothing you're loosing.

Execute "zfs list" and post the output, i can tell you then the commands simply.

If you for whatever reason don't need that ufs dataset anymore, you can destroy it either, to revert to your current state.

 

[happnatious1]

I did not know you could put ext4 on top of zfs




NAME USED AVAIL REFER MOUNTPOINT
VMs 51.6G 848G 96K /VMs
VMs/vm-100-disk-0 51.6G 856G 42.8G -
rpool 211G 238G 104K /rpool
rpool/ROOT 5.58G 238G 96K /rpool/ROOT
rpool/ROOT/pve-1 5.58G 238G 5.58G /
rpool/data 206G 238G 96K /rpool/data
rpool/data/subvol-105-disk-0 630M 7.38G 630M /rpool/data/subvol-105-disk-0
rpool/data/vm-100-disk-0 42.6G 238G 42.6G -
rpool/data/vm-101-disk-0 26.6G 238G 26.6G -
rpool/data/vm-102-disk-0 16.0G 238G 16.0G -
rpool/data/vm-103-disk-0 25.3G 238G 25.3G -
rpool/data/vm-104-disk-0 94.5G 238G 94.5G -

 

[Ramalama]

- Create the EXT4 Volume on ZFS:

zfs create -V 50GB rpool/docker
mkfs.ext4 /dev/zvol/rpool/docker
mkdir /mnt/docker
mount /dev/zvol/rpool/docker /mnt/docker

- Then go into the GUI:
Datacenter -> Storage -> Add -> Directory:

ID: WhateverYouWant
Directory: /mnt/docker
Content: Container
Nodes: If you have more then one, select the right one
Enabled: yes
Shared: no

- Then in the Container View:

Container -> Ressources -> Tap on Root Disk -> Volume Action at the Top -> Move Storage
Select the new EXT4 Storage and Check delete source

- To make it persistent, you need to add to fstab:
/dev/zvol/rpool/docker /mnt/docker ext4 defaults 0 0

- To destroy the dataset, or ext4 partition, so that you revert back as if nothing happened:

Move your LXC Storage back to your ZFS Storage.
Delete the Storage under "Datacenter -> Storage" again
umount /mnt/docker
rm -rf /mnt/docker
delete the entry in /etc/fstab again
zfs destroy rpool/docker

And thats it.

you can create the Ext4 storage under your "VMs" zpool either instead of "rpool"
zfs create -V 50GB VMs/docker
mkfs.ext4 /dev/zvol/VMs/docker
and so on....

Hopefully it's understandable
Cheers

 

[happnatious1]

Thank you!

 

[louie1961]

I have no experience with containers. I tried creating a Debian 12 container in proxmox and I added Portainer to it but it seems that I cant add any other containers to it. I would like to have a ddns container, a guacamole container, and a RustDesk container. Do I need to install a full Debian vm with docker and portainer to host multiple containers, or does each docker container need to be on it's own container within proxmox? Am I approaching this all wrong?

If you are talking Docker containers and not LXC containers, then you need one or more VMs to run as a docker host. You could use an LXC container to host docker, but I understand this is not a best practice and may be unstable. But docker containers cannot run on Proxmox directly, unlike LXC containers. Spin up a Debian VM, use the instructions found here for installing docker: https://docs.docker.com/engine/install/debian/

Docker operates exeactly the same in a Debian VM as it does on Debian running on bare metal. Just forget that you are running on Proxmox as it really doesn't make a bit of difference for the most part. After you install docker you can install portainer here: https://docs.portainer.io/start/install-ce/server/docker/linux

 

[louie1961]

If your lxc container with docker inside is running on zfs storage, well you need to enable nesting at least.
But even then not all docker container will work. Some will, some won't on zfs as storage.

The easiest way is to create an ext4 dataset...
Means simply add an dataset with a fixed size, like 50gb for example and format that dataset as ext4, then mount it somewhere on your host and add that directory under the datacenter tree as container storage.

Then simply move the drive of your lxc container to that new ext4 storage.

After that, all docker containers should work.

I wouldn't bother with any of this personally. Running docker in an LXC container is not a good practice. I would just spin up a VM, and use normal VM storage for the docker volumes and/or bind mounts, until you get comfortable enough with docker to use an NFS share for your volumes and persistent data. Best practice is to separate your persistent data from the VM running docker, and store it on a NAS or another storage server through NFS, Samba, etc. Docker containers are supposed to be ephemeral.

But for just starting out and learning just add a little extra room to the VM you spin up to host docker.

 

[louie1961]

Here's some great resources for learning Docker basics quickly.

https://www.youtube.com/watch?v=eGz9DS-aIeY&t=73s

https://www.youtube.com/watch?v=WpwtzQq5Er4&list=PLT98CRl2KxKECHltRib03tG8pyKEzwf9t

 

[louie1961]

The hardest part of docker for me was figuring out the networking. https://www.youtube.com/watch?v=bKFMS5C4CG0

 

[happnatious1]

Great info! Thanks Louie1961.

 

[Ramalama]

I run docker inside unprivileged LXC containers since almost forever.
There is no downside running it on ext4 partitions.

Additionally docker will get fully supported once we get zfs 2.2 on Proxmox.

So what's the point of running docker inside an VM other as, "it's not recommended" versus an LXC container?

Lxc containers provided resource benefits, vm's provide isolation benefits.
An unprivileged container provides almost both.

The times we "had to run" docker inside VM's because Docker didn't worked natively in LXC containers, or just with a lot workarounds and downsides is solved since some time already.
I think around 1 year or so.

And in some months with zfs 2.2, it's solved entirely.

Don't forget also, no matter how much isolation you have, especially in terms of unprivileged vs VM, you have still the network connection as the biggest vulnerability.

Otherwise you must put your VM into DMZ.

Just to mention because not everyone knows that:
An ext4 formatted zfs dataset that is 50GB big, won't consume 50GB, only the files/data itself that lays on that dataset.
And you even get snapshotting + most of the other zfs benefits.

Just my 2 cents.
But like we say in Germany, opinions are like a**hol*s, everyone has one

Edit:
About your "bind mounts" recommendation?
Either you will loose entirely the possibility to backup the lxc container or the "bind mount" won't get backupped.
Depends where the bind mount is mounted to.

 

[pottproll]

And in some months with zfs 2.2, it's solved entirely.

Stumbled upon this thread and just checked:

cat /sys/module/zfs/version
2.2.2-pve1

What exactly does this mean for docker in lxc on zfs? Can I use my docker in a debian lxc and if so, do I need some specific settings (i.e. nesting?)?
I did use a vm for my docker when I set it up years ago because it was recommended but since its a not too punchy home server I could use the resources elsewhere...
Thanks for the input anyway. Ill check it if I get the time.

 

[Ramalama]

Stumbled upon this thread and just checked:

cat /sys/module/zfs/version
2.2.2-pve1

What exactly does this mean for docker in lxc on zfs? Can I use my docker in a debian lxc and if so, do I need some specific settings (i.e. nesting?)?
I did use a vm for my docker when I set it up years ago because it was recommended but since its a not too punchy home server I could use the resources elsewhere...
Thanks for the input anyway. Ill check it if I get the time.

Yes docker is an non issue inside lxc containers on zfs anymore.
After Kernel 6.2 and before Before zfs 2.2 docker worked inside lxc containers on everything except zfs.

However there is maybe 1/2 docker images, that might not work, but i didn't camed across against any.
99,9% of all docker images should work.

Privileged/Unprivileged shouldn't matter anymore either.

Cheers