Docker on Proxmox VM (not LXC) with ZFS Storage

stringpark

New Member
Dec 19, 2022
9
0
1
I figure this is an elementary question for most, but I've scoured the forum (and internet at large) and am still struggling to put two and two together. I'm trying to understand the proper way to setup Docker on a Proxmox VM using ZFS storage. I have two zpools on my machine and, though I'm still a bit undecided on this part, I think I'd like to run the guest OS / Docker on one zpool and store the Docker volume data on the other.

So far, I've created the VM from scratch (with a single virtual disk on one of the zpools), installed Ubuntu Server and Docker. The "docker info" command shows that the current Storage Driver setting is "overlay2" and the Backing Filesystem is "extfs". Should I change the storage driver to zfs?

Also, how would I best go about setting up storage of the docker volume data on the other zpool? Would this be by attaching another virtual disk (from the other zpool) to the VM? If so, where would I go from there?

Lastly, in this post, @LnxBil makes mention of a ZFS Volume Driver for Docker. Seems like a good idea, but I can't seem to find any info on how to implement this.

Sorry to say, but I guess I'm hoping for a step-by-step type guide in simple language for a noob like me. The one thing that I think I understand from searching the forum is that there are ZFS related advantages to running Docker on a VM (as opposed to an LXC). I'm just hoping to find the best practice configurations needed to capitalize on those advantages.

Thanks to anyone who can help!
 
Also, how would I best go about setting up storage of the docker volume data on the other zpool?
Just use the one pool for both. Does not make sense to have more than one pool. You would need however multiple datasets. I can recommend to use
  • /var/lib/docker
  • /var/lib/docker/volumes
as separate datasets BEFORE installing official docker. Don't install the old packages from Ubuntu.

Should I change the storage driver to zfs?
Yes, this is not the norm anymore. You need to change that manually. This is my most-up-to-date configuration (/etc/docker/daemon.json) for my hosts:

Code:
{
    "storage-driver": "zfs",
    "live-restore": true,
    "log-driver": "syslog",
    "log-opts": {
        "syslog-address": "unixgram:///dev/log",
        "tag" : "docker/{{.Name}}"
    }
}

If you just want the plain-old json logging ... skip the entries with the log-prefix.

Lastly, in this post, @LnxBil makes mention of a ZFS Volume Driver for Docker. Seems like a good idea, but I can't seem to find any info on how to implement this.
Here. Install instructions are also there.

I'm just hoping to find the best practice configurations needed to capitalize on those advantages.
Very good. I thought this turns out to be just another Docker+LXC+ZFS post, but it's refreshingly not ;)
 
  • Like
Reactions: stringpark
I thought "zfs" storage driver is only useful when running ZFS directly? As far as I understand you are using ZFS on the host, using a zvol for your VM and then format it with something like LVM and ext4 from the guestOS? Or do you run ZFS on top of ZFS?
 
Yeah, I'm afraid I'm still a bit lost. I tried to create the recommended datasets from within the guest OS (fresh Ubuntu Server 22.04 install), but it didn't seem to know anything about zfs. This actually made sense to me, since I had only handed the VM a virtual disk created from a zpool managed by PVE, and not created a zpool from within the guest OS itself. I then supposed that perhaps I just needed to create those directories along with the daemon.json file (with storage-drive: zfs), then install docker and maybe some magic would happen, but Docker doesn't start when I do it that way. Is it perhaps that the docker zfs driver is only used if I were passing through the physical drives to the VM and creating the zpool from within the guest OS (rather than just feeding the VM a virtual disk)? I'm afraid I'm still not quite grasping the concept. Any help would be appreciated. Thx for your patience.
 
Last edited:
Oh I missunderstood you. I thought you created the ZFS pool inside of your guest on your second disk (this is what I always do). There is currently no way to get ZFS + Docker to work besides running on your PVE host with ZFS or in a VM with ZFS (inside your guest). Maybe we get it with LXC-ZFS-support in the new upcoming ZFS release.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!