4.0.36 Upgrade and lost LXC containers

[gob]

Hi

I have just upgraded my PVE 4.0.24 to 4.0.36.
I shut down all CTs and VMs first, upgraded then rebooted. When the host came back up I noticed some of the LXC CTs were missing.
Reviewing the SysLog there are lots of errors concerning the file system of the Loop devices. I can browse the file structure in /car/lib/vz and can see the CT images OK.
I am just unsure how to recover from this and get the CT's back online.

This is a single DELL server node with local storage and also Multipath iSCSI to an DELL MD3000i storage array. MY VM's are on the MD3000i and the CT's are on the local storage.
I do have backups of the CT's so I can probably rebuild from scratch but it took a long time to get the multipath working so I would rather not have to rebuild if possible.

Package Versions:

proxmox-ve: 4.0-10 (running kernel: 4.2.0-1-pve)
pve-manager: 4.0-36 (running version: 4.0-36/9815097f)
pve-kernel-3.19.8-1-pve: 3.19.8-3
pve-kernel-4.1.3-1-pve: 4.1.3-7
pve-kernel-4.2.0-1-pve: 4.2.0-10
lvm2: 2.02.116-pve1
corosync-pve: 2.3.4-2
libqb0: 0.17.1-3
pve-cluster: 4.0-17
qemu-server: 4.0-23
pve-firmware: 1.1-7
libpve-common-perl: 4.0-20
libpve-access-control: 4.0-8
libpve-storage-perl: 4.0-21
pve-libspice-server1: 0.12.5-1
vncterm: 1.2-1
pve-qemu-kvm: 2.4-5
pve-container: 0.9-18
pve-firewall: 2.0-11
pve-ha-manager: 1.0-5
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.3-1
lxcfs: 0.9-pve2
cgmanager: 0.37-pve2
criu: 1.6.0-1
zfsutils: 0.6.4-pve3~jessie

I have attached the syslog from when I rebooted the host.

I am not 100% sure the faults are due to the upgrade or perhaps a power outage. The disks are not reporting any errors.
Any suggestions appreciated.

Thanks

 

[wbumiller]

What happens if you run /usr/sbin/pve-update-lxc-config

 

[gob]

I get

# /usr/sbin/pve-update-lxc-config
converting /etc/pve/lxc/311/config to /etc/pve/lxc/311.conf
failed to create required config key 'ostype' at /usr/sbin/pve-update-lxc-config line 586.

 

[gob]

It appears there is only the one LXC offline.
I thought there was more as I have more image folders in /var/lib/vz/images

 

[wbumiller]

Is there an lxc.include line in this particular config? The ostype is extracted from there (as all containers come configured with an include directive based on distribution)
eg

lxc.include = /usr/share/lxc/config/debian.common.conf

If not, try adding one and rerunning pve-update-lxc-config

 

[gob]

Great - that has brough the CT back and it appears to be running OK.

What do you suggest I do regarding the various file system errors?

Many thanks.

 

[wbumiller]

You could fsck the affected (or all) ct images, but they'd need to be offline for that. (And we don't have an automated way to do this (yet)).

 

[gob]

ah, so can I run fsck just on /var/lib/vz/images without having to do the entire volume in single user mode?
It's not a problem shutting down the CT's to do this.

 

[wbumiller]

Yes, a small advantage of using raw images ;-)

 

[gob]

hmmm....

#fsck /var/lib/vz/imagesfsck from util-linux 2.25.2
e2fsck 1.42.12 (29-Aug-2014)
fsck.ext2: Is a directory while trying to open /var/lib/vz/images


The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

Is it really broken or should I try what it suggests?

 

[wbumiller]

Oh, I didn't mean running it on the images/ directory. But on the containers' image files.
Ie on /var/lib/vz/images/1234/vm-1234-disk-1.raw etc. (obviously only for IDs which belong to containers).

 

[gob]

raw files checked and repaired two files. Fingers crossed all looks good now.
Many thanks for you help Wolfgang