GUI on 3 Node-Cluster greyed out

[bwg]

I have a cluster with 3 Nodes running. Version on all nodes is 6.2-12.
This cluster has been running all year without any problems and today suddenly the GUI is not working anymore.
- the names für the VMS & LXCs are not displayed, just their ID, all marked with '?'
- all storages are greyed out and marked with '?'
- The summay page for the cluster displays no information.
- all VMs & LXCs are running
- consoles are working
- contents of storages are displayed
- all Performance displays (CPU, Network etc.) are empty

I have tried the following:
- rebooted alle 3 machines => noch change
- executed on all machines
systemctl restart pvedaemon
systemctl restart pveproxy
systemctl restart pvestatd
=> for a few minutes, VM/LXC Info is beeing displayed, then all goes grey again. Storages remain grey.

I have searched for help, but cannot find a solution.

I need help quite urgent, any advise is very welcome.

 

[Moayad]

Hi,

can you explain what you did before?

did you found any error in syslog/journalctl

 

[bwg]

Hi,
yesterday I installed 2 additional disks in on server and started to build a software Raid 1. That is the only change,
In the Syslog I find appr. every minute
Oct 16 16:40:01 pve111 systemd[1]: pvesr.service: Succeeded.
Oct 16 16:40:01 pve111 systemd[1]: Started Proxmox VE replication runner.
Oct 16 16:41:00 pve111 systemd[1]: Starting Proxmox VE replication runner...
Oct 16 16:41:01 pve111 systemd[1]: pvesr.service: Succeeded.
Oct 16 16:41:01 pve111 systemd[1]: Started Proxmox VE replication runner.
Oct 16 16:42:00 pve111 systemd[1]: Starting Proxmox VE replication runner...
Oct 16 16:42:01 pve111 systemd[1]: pvesr.service: Succeeded.

and sometimes
Oct 16 16:42:01 pve111 systemd[1]: Started Proxmox VE replication runner.
Oct 16 16:42:18 pve111 pmxcfs[25198]: [status] notice: received log
Oct 16 16:42:48 pve111 pveproxy[30276]: proxy detected vanished client connection
Oct 16 16:42:48 pve111 pveproxy[13711]: proxy detected vanished client connection
Oct 16 16:43:00 pve111 systemd[1]: Starting Proxmox VE replication runner...
Oct 16 16:43:01 pve111 systemd[1]: pvesr.service: Succeeded.

 

[bwg]

now contents of storages are not displayed anymore, instead I get a "communication failure(0) in the GUI

 

[bwg]

Problem solved

 

[Filip Francis]

Problem solved

Hi,
I am having the same issue that 3 out of 4 nodes that i rebooted are grayed out.
when i do the following command
systemctl restart pvedaemon
systemctl restart pveproxy
systemctl restart pvestatd

It stays green again for a couple of minutes but at the end it goes again to grey and i did not found a solution
How did you solved it
thanks
Filip