[SOLVED] after upgrade have issues with cluster

[RobFantini]

just upgraded our 7 node cluster. after upgrade at pve web page, health shows OK. yet most nodes and vm's are shaded out and not accessible. so far i have rebooted 4 of 7 nodes and still have the issue.
vm's are operating OK, however not sure that will continue.

>>>> pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.55-1-pve)
pve-manager: 6.2-11 (running version: 6.2-11/22fb4983)
pve-kernel-5.4: 6.2-5
pve-kernel-helper: 6.2-5
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.55-1-pve: 5.4.55-1
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.44-1-pve: 5.4.44-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
ceph: 14.2.10-pve1
ceph-fuse: 14.2.10-pve1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-1
libpve-guest-common-perl: 3.1-2
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-6
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-10
pve-cluster: 6.1-8
pve-container: 3.1-13
pve-docs: 6.2-5
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-2
pve-ha-manager: 3.1-1
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-13
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-14
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve1

please advise how to fix or what other info may be needed

 

[RobFantini]

shortly after I wrote that - pve screen returned to normal. however it was an issue for an hour. perhaps it took the majority of the nodes to restart then some time for cluster to get sorted out.

 

[RobFantini]

a week ago I added a NFS V4 server directory mount in /etc/fstab to each node, then added that as directory storage.

# mount|grep nfs
<IP hiden>:/nfs-shares on /media/share type nfs4 (rw,nosuid,nodev,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=<IP hidden>,local_lock=none,addr=10.1.10.80,user)

now the nfs server was rebooted recently and that could have caused stale file handles . those did happen at 2 vm's . i had to umount -l and remount.

I have not used NFS in awhile and our mount or export options could need refinement.
So I assume my NFS incorrect configuration caused the issue.