[SOLVED] Server has a grey question mark

[helis]

One of the servers in the cluser, and all VMs on it, all of a sudden got grey question marks. All the external storage on this server is marked the same way too.
The VMs are running fine (but migrating them is impossible because target storages don't show up), the web GUI is available, the copper and fiber links seem to be online.
Restarting pvestatd resolves the issue for several minutes, then the question marks return. The restarting itself takes about a minute too.
Restarting pvedaemon doesn't seem to do anything.

How do I go further with diagnosing this?

 

[bubbadk]

i have the exact same thing. must be an update gone wrong. but yes the pvestatd restart solves it for me. let's hope they solves it

 

[helis]

Doubt it, I haven't made any updates lately.

UPD
This post helped me with a half of the problem: https://forum.proxmox.com/threads/o...i-after-upgrading-to-8-0-4.133873/post-615181
There was indeed one hung vgs process due to unfinished migration. I killed it, restarted the services, now the server and the VMs a green. The storage on the server is still grey though. Are there any other hung processes to look for?

UPD2 Just after I posted it, the server is grey again

UPD3 While I was doing all this (killing vgs process and restarting services), another server in the cluster briefly went grey a couple times. Now it's green again, I didn't touch it. Pretty sure it's the one where that failed migration was supposed to go to. Maybe this will give somebody some insight into whatever is going on.

UPD4 Turns out, after killing the vgs process I also had to killl a vgscan process. Now restarting pvestatd is fast (used to take a minute), but other than that the "question mark" problem persists.

 

[helis]

Aaand turns out there were a bunch of lvchange processes hanging too. Killed them all, everything seems to work now.
Who's good, I'm good.