node red and vm's as black but running ok

[embb]

Hi

I have a proxmox runnning on online.net and today a very strange thing started happening. All the sudden all the vm's keep working but the web UI shows the node as red and then all vm's as black without any reason.
As I said they all keep working but the web ui is completely useless and then a server reboot is the only option.
However if I wait 10-20 minutes sometimes it comes back to life...

Has anyone experience this issue ?

Thanks
Eduardo

 

[wolfgang]

Hi,
is this node in a cluster?
if yes it can be that the cluster communication is not reliable.

 

[embb]

Hi,

No cluster. The odd it's that is a single node and I can ssh to it without any issue when this happens.

Here is all packages versions:
proxmox-ve-2.6.32: 3.4-160 (running kernel: 2.6.32-40-pve)pve-manager: 3.4-9 (running version: 3.4-9/4b51d87a)pve-kernel-2.6.32-40-pve: 2.6.32-160lvm2: 2.02.98-pve4clvm: 2.02.98-pve4corosync-pve: 1.4.7-1openais-pve: 1.1.4-3libqb0: 0.11.1-2redhat-cluster-pve: 3.2.0-2resource-agents-pve: 3.9.2-4fence-agents-pve: 4.0.10-3pve-cluster: 3.0-18qemu-server: 3.4-6pve-firmware: 1.1-4libpve-common-perl: 3.0-24libpve-access-control: 3.0-16libpve-storage-perl: 3.0-33pve-libspice-server1: 0.12.4-3vncterm: 1.1-8vzctl: 4.0-1pve6vzprocps: 2.0.11-2vzquota: 3.1-2pve-qemu-kvm: 2.2-11ksm-control-daemon: 1.1-1glusterfs-client: 3.5.2-1

Regards,
Eduardo

 

[wolfgang]

do you find some thing relevant in your syslog?

 

[embb]

Forget to mention that the time the communication seems to be down is shown on the charts. Below is the node cpu:

 

[embb]

Seems that a few events do match the begin and end of downtime cross matching with the chart:
Begin:
Sep 1 11:21:40 sd-84132 pveproxy[176655]: worker exit
Sep 1 11:21:40 sd-84132 pveproxy[3639]: worker 176655 finished
Sep 1 11:21:40 sd-84132 pveproxy[3639]: starting 1 worker(s)
Sep 1 11:21:40 sd-84132 pveproxy[3639]: worker 185144 started
Sep 1 11:21:43 sd-84132 pvedaemon[185155]: starting vnc proxy UPID:sd-84132:0002D343:0074DFA7:55E57C37:vncproxy:800:root@pam:
Sep 1 11:21:43 sd-84132 pvedaemon[176014]: <root@pam> starting task UPID:sd-84132:0002D343:0074DFA7:55E57C37:vncproxy:800:root@pam:


End:
Sep 1 11:42:07 sd-84132 kernel: ct0 nfs: server 10.21.21.16 OK
Sep 1 11:42:07 sd-84132 pvestatd[3634]: status update time (925.675 seconds)

Regards
Eduardo

 

[wolfgang]

It looks like there is something wrong with your pve-manager try to reinstall it.

 

[embb]

What are the steps to make that reinstalation or it's just a simple apt-get remove follow by apt-get install of pve-manager?

 

[wolfgang]

Apt-get install --reinstall pve-manager