Node in grey mode (question mark) up to 7 days

[Ivan Gersi]

I have 5 nodes cluster.
Pve1 was upgraded from 6.0.1 to 6.4-15, Pve4 has fresh install from 6.4-4, rest pves have 6.0.1.
Pve1 working properly.
But pve4 has a strange problem...during 7 days goes to "grey mode". I`m able to connect to pve4 via web shell, I`m able to connect via ssh. I can see all nodes on via pvecm status, all VM are running on Pve4 and I can connect them remotly.
When I restart pve-cluster service node and VM are green via web again.
I can`t see any ssuspiciously record in logs.
Any ideas?

 

[Neobin]

https://forum.proxmox.com/threads/upgraded-nod-has-several-problems-after-11days-uptime.116369

Pve1 was upgraded from 6.0.1 to 6.4-15, Pve4 has fresh install from 6.4-4, rest pves have 6.0.1.

So the cluster is still not running the same recent version...

 

[Ivan Gersi]

https://forum.proxmox.com/threads/upgraded-nod-has-several-problems-after-11days-uptime.116369



So the cluster is still not running the same recent version...

No, because I have not enough resources for destabilizing another node. This is 24/7 traffic and I have to have 100% safety I`ll have a stable node afer upgrade.
Pve4 is extremly unstable from fresh install and I`m not able to fix it.
This is part of log from yesterday restarting.

Nov 13 11:21:00 pve4 systemd[1]: Starting Proxmox VE replication runner...
Nov 13 11:21:01 pve4 systemd[1]: pvesr.service: Succeeded.
Nov 13 11:21:01 pve4 systemd[1]: Started Proxmox VE replication runner.
.............................................................................................................................................................................................................................................................................................................................
.............................................................................................................................................................................................................................................................................................................................
.............................................................................................................................................................................................................................................................................................................................
.............................................................................................................................................................................................................................................................................................................................
.............................................................................................................................................................................................................................................................................................................................
.............................................................................................................................................................................................................................................................................................................................
.............................................................................................................................................................................................................................................................................................................................
.............................................................................................................................................................................................................................................................................................................................
.............................................................................................................................................................................................................................................................................................................................
.............................................................................................................................................................................................................................................................................................................................
.............................................................................................................................................................................................................................................................................................................................
...........................................................................................................................................................................................................................................................................................Nov 13 15:01:54 pve4 kernel: [
0.000000] Linux version 5.4.106-1-pve (build@pve) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP PVE 5.4.106-1 (Fri, 19 Mar 2021 11:08:47 +0100) ()
Nov 13 15:01:54 pve4 systemd-modules-load[597]: Inserted module 'iscsi_tcp'

 

[dale]

May be
chown -R www-data:www-data /var/log/pveproxy
I've seen it before.

 

[pilchu]

I for one have found that the problem may be improperly mounted resources. Unmounting the resources from the node level helped.

umount -f /mnt/pve/storage
or
umount -l /mnt/pve/storage