[TUTORIAL] Proxmox with Docker (and Portainer)

[michabbs]

If you want to use Docker with Proxmox you have 3 options: run Docker in VM, run Docker in LXC, or run Docker directly on the PVE node. Here it small tutorial for the 3rd option.
Note: Everything to be run as root.

Enable firewall in PVE GUI (on at least datacenter level). Even if you do not use firewall in Proxmox you must enable it (just set default policy to ACCEPT), because Docker will use netfilter.

Prepare iptables for Docker:

iptables -N DOCKER-USER; iptables -I DOCKER-USER -j ACCEPT

You must make the above setting permanent. For example edit /etc/network/interfaces and add it to the "lo" interface as pre-up script:

auto lo
iface lo inet loopback
    pre-up iptables -N DOCKER-USER; iptables -I DOCKER-USER -j ACCEPT

Note: Possibly there are better ways to achieve it, but the above "works for me"...

Prepare filesystem - more or less this way:

zfs create -o mountpoint=/var/lib/docker rpool/docker-root
zfs create -o mountpoint=/var/lib/docker/volumes rpool/docker-volumes
chmod 700 /var/lib/docker/volumes

If you use zfs-auto-snapshot, you might want to consider this:

zfs set com.sun:auto-snapshot=false rpool/docker-root
zfs set com.sun:auto-snapshot=true rpool/docker-volumes

Create /etc/docker/daemon.json with the following content:

{
  "storage-driver": "zfs"
}

Add /etc/apt/sources.list.d/docker.list with the following content:

deb [arch=amd64] https://download.docker.com/linux/debian buster stable
# deb-src [arch=amd64] https://download.docker.com/linux/debian buster stable

Install Docker:

apt install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
apt update
apt install docker-ce docker-ce-cli containerd.io

You might want this:

usermod -aG docker your-user

Voila! Your Docker should be ready! Test it:

docker run hello-world

Option: Install Portainer

zfs create rpool/docker-volumes/portainer_data
docker volume create portainer_data
docker run -d -p 8000:8000 -p 9000:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce

You might omit the 1st line if you do not want to have separate dataset for the docker volume (bad idea).

Go to http://yournode:9000 and configure.

 

[progl]

How safe is it run portainer on Proxmox?
In lxc or vm u can make backup

 

[michabbs]

Well, you always should make backup, no matter where docker is actually running... ;-)

When you run docker directly on pve node you get potentially higher performance and direct access to zfs (good). You might also notice some problems with network communication between docker and lxc containers on the same node. (bad)
I had no time to investigate and tweak it... :-(

 

[progl]

and how much performance do u win if compare host, lxc, vm? are network problems actual on last version?

 

[michabbs]

and how much performance do u win if compare host, lxc, vm?

No idea, I made no tests. Actually for me zfs integration was the point. I suppose there should be no big difference between lxc and bare metal. Anyway they both use the same kernel in the same PC.

are network problems actual on last version?

Yes. I think netfilter policies are too strict, but as I wrote - I had no time for deeper tests... I suspect adding macvlan network might help.

 

[michabbs]

I made some basic tests: Connetions from lxc's or from qemu-vm's to docker (via forwarded port on the host ip) do not work. It seems packets are lost when they reach DNAT rule (auto-generated by docker) - they go through prerouting and never appear in forward chain.

Solution: create macvlan network in Docker and assign "real" IP to a container. (This of course might have some disadvantages...)

Possible better solution: It seems the rp_filter setting in kernel is somehow involved. When I do this: echo "0" > /proc/sys/net/ipv4/conf/all/rp_filter - the packets go further via forward chain and seem to reach the destination, but there are still problems with reply. More investigation is needed... :-(

 

[Sanman99]

If you want to use Docker with Proxmox you have 3 options: run Docker in VM, run Docker in LXC, or run Docker directly on the PVE node. Here it small tutorial for the 3rd option.
Note: Everything to be run as root.

Enable firewall in PVE GUI (on at least datacenter level). Even if you do not use firewall in Proxmox you must enable it (just set default policy to ACCEPT), because Docker will use netfilter.

Prepare iptables for Docker:

iptables -N DOCKER-USER; iptables -I DOCKER-USER -j ACCEPT

You must make the above setting permanent. For example edit /etc/network/interfaces and add it to the "lo" interface as pre-up script:

auto lo
iface lo inet loopback
    pre-up iptables -N DOCKER-USER; iptables -I DOCKER-USER -j ACCEPT

Note: Possibly there are better ways to achieve it, but the above "works for me"...

Prepare filesystem - more or less this way:

zfs create -o mountpoint=/var/lib/docker rpool/docker-root
zfs create -o mountpoint=/var/lib/docker/volumes rpool/docker-volumes
chmod 700 /var/lib/docker/volumes

If you use zfs-auto-snapshot, you might want to consider this:

zfs set com.sun:auto-snapshot=false rpool/docker-root
zfs set com.sun:auto-snapshot=true rpool/docker-volumes

Create /etc/docker/daemon.json with the following content:

{
  "storage-driver": "zfs"
}

Add /etc/apt/sources.list.d/docker.list with the following content:

deb [arch=amd64] https://download.docker.com/linux/debian buster stable
# deb-src [arch=amd64] https://download.docker.com/linux/debian buster stable

Install Docker:

apt install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
apt update
apt install docker-ce docker-ce-cli containerd.io

You might want this:

usermod -aG docker your-user

Voila! Your Docker should be ready! Test it:

docker run hello-world

Option: Install Portainer

zfs create rpool/docker-volumes/portainer_data
docker volume create portainer_data
docker run -d -p 8000:8000 -p 9000:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce

You might omit the 1st line if you do not want to have separate dataset for the docker volume (bad idea).

Go to http://yournode:9000 and configure.

(is this too old to bump the thread?)

do i need to re-partition anything or add a pool somehow first?

zfs create -o mountpoint=/var/lib/docker rpool/docker-root
cannot create 'rpool/docker-root': no such pool 'rpool'

i made a thread
https://forum.proxmox.com/threads/trying-to-get-started-wont-connect.139171/

still haven't been able to get Docker installed
just hoping to get started soon