Cluster unusable due to NAS disruption

[jbaas]

Just as we're about to move all virtual machines to our three-machine Proxmox cluster, a NAS with SMB shares failed. This seemed to lock up pvestatd, making it unkillable.

I found an old thread: https://forum.proxmox.com/threads/pvestatd-locks-up-if-network-share-missbehaves.58537/ which describes the same issue.

At this time, I can't restart the services and can't migrate VMs (at least, via the GUI). Is my only option to fully restart the servers?

 

[bbgeek17]

Is my only option to fully restart the servers?

Have you restored your NAS services? If you have similar Kernel messages in your log as the other poster, and you have restored NAS services, then it seems that Kernel is unable to unwedge itself. Reboot is likely the only option in this case.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[jbaas]

Yes, the NAS has been restored and can be accessed again.

I'm not a Linux expert, but I thought kill -9 would always kill a process. In this case, pvestatd won't budge. Starting a new instance also doesn't work.

Any suggestions on how to move the VMs, while the GUI is down?

 

[bbgeek17]

I'm not a Linux expert, but I thought kill -9 would always kill a process.

In the thread you referenced the poster shows evidence of Kernel lock. While "pvestatd" is a userland process, it interacts with Linux Kernel, and specifically SMB/NFS infrastructure. If the Kernel is wedged, "kill -9" will not help.
If you had HA properly configured and your other nodes are healthy, powering off the failed node will initiate an HA event and the VMs will be migrated.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[jbaas]

Gotcha, thanks. Restart it is.