Can't start container restored from backup - unable to detect OS distribution

[Davidoff]

My single node Proxmox recently got very messed up. Gave up on it, killed it and reinstalled Proxmox from scratch (a couple of days ago - v. 5.4). I then restored containers from backups. However, one container (originally CT 100 and restoring to CT 100) does not start. That container had been one of the first I had created when I originally installed Proxmox (which I think was then 4.4). I had kept the old instance of Proxmox up to date (and believe it was 5.4 when it died). The container had started without issue on the old instance. The container was originally installed with Ubuntu 17.10 but then later upgraded to 18.04 without any issues.

After digging through other issues (e.g. /sys/fs/cgroup/systemd/lxc/100 files were not removed after aborted start through the GUI), it seems the root cause is this entry that I found in the debug log when trying to start the container in the foreground (i.e. "lxc-start -n 100 -F -l DEBUG -o lxc100dbg.log":

lxc-start 100 20190421043428.136 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start with output: unable to detect OS distribution

I've reviewed all the threads I could find referencing this message and it almost always points to a missing or corrupted file in /etc of the container. I mounted the container filesystem (pct mount 100) and checked that debian_version and lsb-release exist, and they do. debian_version shows:

buster/sid

While lsb-release is:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"

This is where I run into a dead end. I've looked through various posts that reference the same error and have looked at config files, tried restoring to a different zpool, checked the storage setup for PVE, etc. according to those posts, but didn't seem to have the same problems as cited in those threads.

Tried to keep the initial post brief but have a ton of other logs, etc. so if any further info is desirable please advise. Any suggestions on how I could get the container up and running again would be most appreciated.

 

[Davidoff]

Actually I just noticed something else that might or might not be a bit worrisome. The backup file for the container is 257 GB, but when I check the size of the mounted filesystem, it seems to use only 6GB.

root@fava2:/var/lib/lxc/100/rootfs/lib# df -h /var/lib/lxc/100/
Filesystem        Size  Used Avail Use% Mounted on
rpool/ROOT/pve-1  431G  6.0G  425G   2% /

The ZFS pool to which I restored the root filesystem for the container also shows only 2.4GB being used:

root@fava2:/var/lib/lxc/100/rootfs/lib# df -h /rpool2/subvol-100-disk-2/
Filesystem                Size  Used Avail Use% Mounted on
rpool2/subvol-100-disk-2  400G  2.4G  398G   1% /rpool2/subvol-100-disk-2

This seems somewhat odd. The restore completed OK. Might this be related to the issues I'm having in starting the container?

 

[Davidoff]

A further development: I had restored a bunch of other containers yesterday and started each of them and they seemed to run just fine yesterday, except for the one I was having problems with.

I tried starting them today and am now getting the same error with every single one of them. The debug log shows the same "unable to detect OS distribution" message.

I've not changed configuration or settings for anything in Proxmox. It seems somewhat odd that containers that ran fine yesterday now won't start. Again, any suggestions or thoughts would be most appreciated.