[SOLVED] LXCs are showing question mark. Node is showing Green, KVMs are showing green

[EmilioMoreno]

Hi
I have a 2 node cluster with pve6.2, it is working ok, both nodes are showing the green tick and i can connect to both via ssh or via webUI. Nodes can see each other in the networking.

In one node all LXC containers are showing a question mark under a grey circle. However the containers are up, and working. In the same node, a KVM is running and it is showing in green. In the other node both containers and KVMs are in green

I have read that rebooting the node should fix the issue but i'd rather not do it if there is another solution, any ideas?

[ADDED]
lxc-ls and lxc-info show the containers as running, e.g.
root@proxmox-4:~# lxc-info 103
Name: 103
State: RUNNING
PID: 9960
IP: 172.16.101.37
CPU use: 8.21 seconds
BlkIO use: 0 bytes
Memory use: 2.28 GiB
KMem use: 16.70 MiB
Link: veth103i0
TX bytes: 9.96 KiB
RX bytes: 166.65 KiB
Total bytes: 176.61 KiB

[ADDED2]
I have updated the other node, but not restarted the node, and now both nodes are showing lxc as grey

 

[Moayad]

Hello,

- There are any error in journalctl -f after restart the node?
- Please check services if running - to check that go to Datacenter -> Node -> System
- Post pveversion -v as well

 

[EmilioMoreno]

This cluster is my TEST env with two nodes.

CPU(s)
48 x Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (2 Sockets)

Kernel Version
Linux 5.3.13-3-pve #1 SMP PVE 5.3.13-3 (Fri, 31 Jan 2020 08:17:11 +0100)

PVE Manager Version
pve-manager/6.2-4/9824574a


I am trying to avoid restarting the node.
Suddenly node2 has shown all containers in green as expected. No action done on my side besides restarting specific lxc container

Node 1 is showing in the log the following:
Jun 05 13:03:01 proxmox-1 systemd[1]: Started Proxmox VE replication runner.
Jun 05 13:03:02 proxmox-1 pvestatd[1501]: lxc status update error: short read on command socket (16 != 0)
Jun 05 13:03:02 proxmox-1 pvestatd[1501]: lxc console cleanup error: short read on command socket (16 != 0)
Jun 05 13:03:12 proxmox-1 pvestatd[1501]: lxc status update error: short read on command socket (16 != 0)
Jun 05 13:03:13 proxmox-1 pvestatd[1501]: lxc console cleanup error: short read on command socket (16 != 0)
Jun 05 13:03:22 proxmox-1 pvestatd[1501]: lxc status update error: short read on command socket (16 != 0)
Jun 05 13:03:22 proxmox-1 pvestatd[1501]: lxc console cleanup error: short read on command socket (16 != 0)
Jun 05 13:03:32 proxmox-1 pvestatd[1501]: lxc status update error: short read on command socket (16 != 0)
Jun 05 13:03:33 proxmox-1 pvestatd[1501]: lxc console cleanup error: short read on command socket (16 != 0)
Jun 05 13:03:42 proxmox-1 pvestatd[1501]: lxc status update error: short read on command socket (16 != 0)
Jun 05 13:03:42 proxmox-1 pvestatd[1501]: lxc console cleanup error: short read on command socket (16 != 0)
Jun 05 13:03:52 proxmox-1 pvestatd[1501]: lxc status update error: short read on command socket (16 != 0)
Jun 05 13:03:53 proxmox-1 pvestatd[1501]: lxc console cleanup error: short read on command socket (16 != 0)
Jun 05 13:04:00 proxmox-1 systemd[1]: Starting Proxmox VE replication runner...
Jun 05 13:04:01 proxmox-1 systemd[1]: pvesr.service: Succeeded.
Jun 05 13:04:01 proxmox-1 systemd[1]: Started Proxmox VE replication runner.
Jun 05 13:04:02 proxmox-1 pvestatd[1501]: lxc status update error: short read on command socket (16 != 0)
Jun 05 13:04:03 proxmox-1 pvestatd[1501]: lxc console cleanup error: short read on command socket (16 != 0)
Jun 05 13:04:12 proxmox-1 pvestatd[1501]: lxc status update error: short read on command socket (16 != 0)
Jun 05 13:04:12 proxmox-1 pvestatd[1501]: lxc console cleanup error: short read on command socket (16 != 0)
Jun 05 13:04:22 proxmox-1 pvestatd[1501]: lxc status update error: short read on command socket (16 != 0)
Jun 05 13:04:23 proxmox-1 pvestatd[1501]: lxc console cleanup error: short read on command socket (16 != 0)
Jun 05 13:04:32 proxmox-1 pvestatd[1501]: lxc status update error: short read on command socket (16 != 0)
Jun 05 13:04:32 proxmox-1 pvestatd[1501]: lxc console cleanup error: short read on command socket (16 != 0)
Jun 05 13:04:42 proxmox-1 pvestatd[1501]: lxc status update error: short read on command socket (16 != 0)
Jun 05 13:04:43 proxmox-1 pvestatd[1501]: lxc console cleanup error: short read on command socket (16 != 0)
Jun 05 13:04:52 proxmox-1 pvestatd[1501]: lxc status update error: short read on command socket (16 != 0)
Jun 05 13:04:52 proxmox-1 pvestatd[1501]: lxc console cleanup error: short read on command socket (16 != 0)
Jun 05 13:05:00 proxmox-1 systemd[1]: Starting Proxmox VE replication runner...
Jun 05 13:05:01 proxmox-1 systemd[1]: pvesr.service: Succeeded.
Jun 05 13:05:01 proxmox-1 systemd[1]: Started Proxmox VE replication runner.

 

[Moayad]

I am trying to avoid restarting the node.

If you don't want restart the node, you can try to restart lxc service following this command:
systemctl restart lxc.service

That happen if not restart the node after upgrade complete, especially if there is an update of the kernel

 

[EmilioMoreno]

Worked like a charm. Thanks!

 

[Moayad]

great, then you can mark the thread as [SOLVED] so others in the same situation can know what to expect

 

[Andre Frossard]

I have the same problem and when I restart the lxc.service the error "short read on..." stops and everything appear to be good, but then I start the container it appears to be running on console but when I click on "Summary" the the error is there again.