[SOLVED] LXC starts but nothing is console and no services

[lifeboy]

I have a new LXC (104) which I created yesterday in a 3 node cluster and it was running perfectly for about 18 hours. This morning I migrated the running container to another node and it stopped working, but proxmox still says it's running. I have tried restarting it, restarted the node, migrated back to original node, checked the container with

pct fsck 104

and tried connecting with

pct console 104

, but nothing is displayed at all.

Here's is the bootup log file attached.

What is going on here and how do I get this container to start again?

System details:

# pveversion -v
proxmox-ve: 5.2-2 (running kernel: 4.15.18-7-pve)
pve-manager: 5.2-10 (running version: 5.2-10/6f892b40)
pve-kernel-4.15: 5.2-10
pve-kernel-4.15.18-7-pve: 4.15.18-27
pve-kernel-4.15.18-2-pve: 4.15.18-21
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph: 12.2.8-pve1
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-40
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-30
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-3
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-20
pve-cluster: 5.0-30
pve-container: 2.0-29
pve-docs: 5.2-8
pve-firewall: 3.0-14
pve-firmware: 2.0-5
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-38
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.11-pve1~bpo1

 

[lifeboy]

Since this happened when I moved the container, I followed this post to try to fix it, but it isn't quite the same issue.
https://forum.proxmox.com/threads/cant-remove-image-when-migrated-with-ceph-krbd.28249/

 

[rholighaus]

Instead of console, have you tried

pct enter 104

?

 

[lifeboy]

Could it be related to the fact that the image cannot be released when the machine is stopped? I can't even clone the machine/image when it is off and the image has been manually unloaded

~# rbd showmapped
id pool    image         snap device  
0  general vm-101-disk-0 -    /dev/rbd0
1  general vm-102-disk-0 -    /dev/rbd1
2  fast    vm-104-disk-0 -    /dev/rbd2
3  general vm-100-disk-0 -    /dev/rbd3
~# lxc-stop -n 104
~# rbd showmapped
id pool    image         snap device  
0  general vm-101-disk-0 -    /dev/rbd0
1  general vm-102-disk-0 -    /dev/rbd1
2  fast    vm-104-disk-0 -    /dev/rbd2
3  general vm-100-disk-0 -    /dev/rbd3
~# rbd unmap /dev/rbd2
~# rbd showmapped
id pool    image         snap device  
0  general vm-101-disk-0 -    /dev/rbd0
1  general vm-102-disk-0 -    /dev/rbd1
3  general vm-100-disk-0 -    /dev/rbd3

It's clear to me now that the lxc hangs right before actually booting the image.

UPDATE: I can actually clone the container, but the clone acts in exactly the same way.

 

[lifeboy]

Instead of console, have you tried

pct enter 104

?

Yes, the container hangs before actually booting the image.

 

[lifeboy]

In the end I recreated the machine container and moved it again and all is well. Since I did a proxmox update inbetween, I assume there was some bug that got sorted in the process.