Proxmox 7.4.X to 8.1.4, broke LXC container and docker? (even backup)

[Str00pwafel]

Hi all,

Yesterday I updated Proxmox from 7.4.16(I believe it was .16) to the latest 8.1.4 and that all went fine.
However, after a reboot, I noticed that my LXC container containing my docker is not working anymore.

I can see all containers still exist in /var/lib/docker/containers as the hexadecimal names but docker ps -a or docker images commands show empty.

Before I updated I made a backup of the complete LXC container but restoring that doesn't change anything either.
I can't seem to just do docker run/docker start with these hexadecimal names.

Any tips on what I can do? I'd really not want to start over.

 

[LnxBil]

First and most importantly: running Docker in LX(C) container and specially updating PVE can and will break your system, has been discussed plenty of times on the forum. If you want a stable setup, use KVM/QEMU VMs, not LX(C) containers. This is the official way to go and you're on your own otherwise.

Any tips on what I can do?

Analyze why the docker socket is not started if it's not started. Most of the times it's a missing kernel module on the hypervisor or some permission problem.
Depending on the used Docker version inside of your guest, you will probably have a switch of the backendend storage, that is kernel version dependend and you therefore ran into the problem after upgrading the underlying hypervisor kernel. I had this problem on multiple occasions and got only empty Docker images and Docker container lists. I manually needed to reconfigure the storage in /etc/docker/daemon.json back to the previously implicit configured one. You may check on that via docker system info.

 

[Folke]

Hi,
there has been a discussion on the docker forum about this, which sounds very similar to what you are experiencing [0].

[0] https://forums.docker.com/t/after-reboot-container-disappear/134298/2

 

[Str00pwafel]

It has always been very stable inside an LXC, could update versions of proxmox without any issues until the bump to 8.X.
But I wasn't aware that it was 'bad habit' or anything else, I'm not sure why one way would be better than the other. Specifically since I've been running it for over 3 years without issues.

I will check on the daemon.json and the link Folke shared, if I can't get it up again I guess I'm recreating it all in a VM.

 

[Str00pwafel]

Indeed, this was the fix: https://docs.docker.com/storage/storagedriver/vfs-driver/

Thanks Folke

 

[LnxBil]

But I wasn't aware that it was 'bad habit' or anything else, I'm not sure why one way would be better than the other.

security, isolation, encapsulation ... and now updates from the hypervisor kernel.