Curiousities with the lxc templates for debian and ubuntu.

[Arto Diito]

In my testing I have only used LXC templates provided with pveam.

I have found oddities in the filesystem layout for some LXC templates from proxmox.
I have not tried all other templates but the ones where I have found the oddities are theese.
debian-10-standard_10.7-1_amd64.tar.gz
ubuntu-20.04-standard_20.04-1_amd64.tar.gz
ubuntu-20.10-standard_20.10-1_amd64.tar.gz

In all three of them the filesystem does not have symbolic links in the root folder.

root@testlxc:~# ls -ald /lib* /*bin
drwxr-xr-x  2 root root 4096 Apr 25  2020 /bin
drwxr-xr-x 15 root root 4096 Apr 25  2020 /lib
drwxr-xr-x  2 root root 4096 Apr 25  2020 /lib64
drwxr-xr-x  2 root root 4096 Apr 25  2020 /sbin

But if I install a debian-10 from an iso to a VM the filesystem looks like this.

root@testvm:~$ ls -ald /*bin /lib*
lrwxrwxrwx 1 root root  7 Jan 18 10:00 /bin -> usr/bin
lrwxrwxrwx 1 root root  7 Jan 18 10:00 /lib -> usr/lib
lrwxrwxrwx 1 root root  9 Jan 18 10:00 /lib32 -> usr/lib32
lrwxrwxrwx 1 root root  9 Jan 18 10:00 /lib64 -> usr/lib64
lrwxrwxrwx 1 root root 10 Jan 18 10:00 /libx32 -> usr/libx32
lrwxrwxrwx 1 root root  8 Jan 18 10:00 /sbin -> usr/sbin

If I create a VM and install CentOS-8 from an iso and compare it with a LXC container from centos-8-default_20201210_amd64.tar.xz then there is no difference in the filesystem layout in this regard.
When I created a LXC container from debian-10-turnkey-syncthing_16.0-1_amd64.tar.gz the filesystem layout for /*bin and /lib* inside the container is the same as for a standard debian-10 installation.

My experience is that this oddity in the debian based LXC containers seem to breake things when packages are installed to the containers.
For instance:
In a standard debian installation the folders /lib/systemd and /usr/lib/systemd have the same content and using either path will get the same result in both read and write operations. But that is not true in the odd filesystems.
The examples below is from a debian-10-standard_10.7-1_amd64.tar.gz container.
If a .deb package wants to create files in /usr/lib/systemd/system that catalog does not exist because it can only be found in /lib/systemd/system. And /lib/systemd/user is also missing but can be found as /usr/lib/systemd/user.

So, are theese oddities intentional?
If yes. It would be very interesting if someone could explain the reason why the proxmox provided lxc-templates for debian should differ from a standard debian/ubuntu installation and why there is no such difference in the case of centos8 and debian-10 from turnkey?
If no. Maybe it would be easier to align the templates with the standard installations instead of fixing the .deb packages that does not handle this.

Best regards
Arto Diito

 

[Arto Diito]

BTW. Perhaps I should mention that I have a somewhat crude solution that seems to work for me.
What is especially important is to not accidentaly add / in front of paths. Doing that as root at the wrong moment will definitely do bad stuff with your pvehost.
Use it at your own risk or improve.

Stop container and mount the troublesome filesystem

root@pvehost:~# pct stop 103
root@pvehost:~# losetup /dev/loop11 /var/lib/vz/images/103/vm-103-disk-0.raw
root@pvehost:~# mount /dev/loop11 /media/loop/
root@pvehost:~# cd /media/loop/

There are some catalogs that exist in both lib and usr/lib so these need som special treatment

root@pvehost:/media/loop# mv  lib/systemd/* usr/lib/systemd
root@pvehost:/media/loop# rmdir  lib/systemd/
root@pvehost:/media/loop# mv lib/x86_64-linux-gnu/* usr/lib/x86_64-linux-gnu/
root@pvehost:/media/loop# rmdir   lib/x86_64-linux-gnu/

Then I move everything else in lib

root@pvehost:/media/loop# mv  lib/* usr/lib

usr/lib64 does not exist so I can move the catalog

root@pvehost:/media/loop# ls usr/lib64
ls: cannot access 'usr/lib64': No such file or directory
root@pvehost:/media/loop# mv lib64 usr

Now the empty folders in ./ can be removed

root@pvehost:/media/loop# rmdir bin sbin lib

Now I have a 'clean' root. Everything has been moved down to usr so lets create some needed catalogs and then the missing links

root@pvehost:/media/loop# install -d -m 0755 -o 100000 -g 100000 usr/libx32
root@pvehost:/media/loop# install -d -m 0755 -o 100000 -g 100000 usr/lib32
root@pvehost:/media/loop# ln -s usr/bin bin
root@pvehost:/media/loop# ln -s usr/sbin sbin
root@pvehost:/media/loop# ln -s usr/lib lib
root@pvehost:/media/loop# ln -s usr/lib32 lib32
root@pvehost:/media/loop# ln -s usr/libx32 libx32
root@pvehost:/media/loop# ln -s usr/lib64 lib64
root@pvehost:/media/loop# chown -h 100000.100000 bin sbin lib lib32 libx32 lib64

Done. Umount lcx filesystem and check it.

root@pvehost:/media/loop# cd
root@pvehost:~# umount /media/loop
root@pvehost:~# fsck /dev/loop11
fsck from util-linux 2.33.1
e2fsck 1.44.5 (15-Dec-2018)
/dev/loop11: clean, 24374/524288 files, 231949/2097152 blocks

Detach the loop11 device and check the filesystem again.

root@pvehost:~# losetup -d /dev/loop11
root@pvehost:~# pct fsck 103
fsck from util-linux 2.33.1
/var/lib/vz/images/103/vm-103-disk-0.raw: clean, 24374/524288 files, 231949/2097152 blocks

Start the container again.

root@pvehost:~# pct start 103

Now the filesystem at the root level in the container looks like a standard debian 10.

root@testlxc:/# ls -al /*bin /lib*
lrwxrwxrwx 1 root root  7 Jan 21 10:36 /bin -> usr/bin
lrwxrwxrwx 1 root root  7 Jan 21 10:36 /lib -> usr/lib
lrwxrwxrwx 1 root root  9 Jan 21 10:41 /lib32 -> usr/lib32
lrwxrwxrwx 1 root root  9 Jan 21 10:36 /lib64 -> usr/lib64
lrwxrwxrwx 1 root root 10 Jan 21 10:41 /libx32 -> usr/libx32
lrwxrwxrwx 1 root root  8 Jan 21 10:36 /sbin -> usr/sbin

Regards
Arto Diito