[SOLVED] Proxmox showing question marks

[PyrexxSys]

Hi,

since today one of our lxc container failed. After that Proxmox showed grey question marks in the web interface and all entries for the lxc containers show a timeout. I tried to kill the lxc container but pct stop vmid hangs indefinitely.
I also tried to kill all relates things to the lxc container. After restarting pvestatd Proxmox shows all things in Webinterface again but after clicking on the specific LXC it all begins anew with showing question marks.
Can someone pls enlighten me what to do?

Regards,
Serkan Siller

 

[oguz]

hi,

please post the output of:

pveversion -v

pct config CTID

and also debug logs from the affected container: lxc-start -n CTID -lDEBUG --logfile /tmp/lxc-CTID.log

and attach the log file here.

 

[PyrexxSys]

Hi,

thank you for your fast response.

pveversion -v

proxmox-ve: 5.4-2 (running kernel: 4.15.18-26-pve)
pve-manager: 5.4-15 (running version: 5.4-15/d0ec33c6)
pve-kernel-4.15: 5.4-19
pve-kernel-4.15.18-30-pve: 4.15.18-58
pve-kernel-4.15.18-29-pve: 4.15.18-57
pve-kernel-4.15.18-28-pve: 4.15.18-56
pve-kernel-4.15.18-27-pve: 4.15.18-55
pve-kernel-4.15.18-26-pve: 4.15.18-54
pve-kernel-4.15.18-25-pve: 4.15.18-53
pve-kernel-4.15.18-24-pve: 4.15.18-52
pve-kernel-4.15.18-23-pve: 4.15.18-51
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.15.18-17-pve: 4.15.18-43
pve-kernel-4.15.18-16-pve: 4.15.18-41
pve-kernel-4.15.18-15-pve: 4.15.18-40
pve-kernel-4.15.18-14-pve: 4.15.18-39
pve-kernel-4.15.18-13-pve: 4.15.18-37
pve-kernel-4.15.18-12-pve: 4.15.18-36
pve-kernel-4.15.18-11-pve: 4.15.18-34
pve-kernel-4.15.18-10-pve: 4.15.18-32
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-8-pve: 4.15.18-28
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.4-1-pve: 4.13.4-26
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: not correctly installed
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-12
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-56
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-14
libpve-storage-perl: 5.0-44
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-7
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-38
pve-container: 2.0-42
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-7
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-4
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-56
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3

pct config ctid

arch: amd64
cmode: console
cores: 8
hostname: cthostname
memory: 6144
nameserver: 1.1.1.1 8.8.8.8 8.8.4.4
net0: name=eth0,bridge=vmbr1,gw=10.1.0.1,hwaddr=26:F2:4E:39:6D:B2,ip=10.1.0.170/24,type=veth
onboot: 1
ostype: debian
protection: 1
rootfs: Proxmox:110/vm-110-disk-0.raw,size=128G
searchdomain: example.com
swap: 2048

and the command for the debug start just hangs and does nothing since 10 minutes.

 

[oguz]

when was the last time you rebooted this machine?

you're running an older kernel, but you have newer kernels installed. maybe you did an update at some point but didn't reboot?

also i notice you're running PVE 5.x which will be EOL by the end of this month. you should look into upgrading your system to get the latest bugfixes and features [0]

[0]: https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0#In-place_upgrade

 

[PyrexxSys]

Last reboot was 89 days ago. There are some critical services running on this machine so a reboot is quite hard to do. Do we have to do a reboot after every update?

 

[oguz]

you need to reboot after kernel updates for system stability.

 

[PyrexxSys]

Thank you for the information, then we will need to do that. One more questions, do you know how we can find out why the container failed? Looking with journalctl and /var/log/lxc/ctid.log don't show much.

 

[oguz]

and also debug logs from the affected container: lxc-start -n CTID -lDEBUG --logfile /tmp/lxc-CTID.log

the debug logs can provide detailed information about the reason.

but most likely after you reboot it will start working.