pveproxy dead; can't restart

[welbo97]

Linux vm 5.0.21-5-pve #1 SMP PVE 5.0.21-10 (Wed, 13 Nov 2019 08:27:10 +0100) x86_64 GNU/Linux

No LSB modules are available.
Distributor ID:    Debian
Description:    Debian GNU/Linux 10 (buster)
Release:    10
Codename:    buster

I left a spice window open and connected and it appears to have crashed pveproxy as I no longer have webgui access. I have tried these steps(as root):

systemctl restart pveproxy
systemctl restart pvedaemon
systemctl stop pveproxy
systemctl stop pvedaemon
kill [pveproxy processid]
kill [pvedaemon processid]
killall pveproxy
killall pvedaemon
killall perl
kill -9 [pveproxy processids]

But the pveproxy start and stop processes remain:

root      7442  0.0  0.1 294144 88316 ?        Ds   13:33   0:00 /usr/bin/perl -T /usr/bin/pveproxy start
root     12888  0.0  0.1 294232 88588 ?        Ds   13:48   0:00 /usr/bin/perl -T /usr/bin/pveproxy start
root     25932  0.0  0.1 294220 88416 ?        Ds   13:40   0:00 /usr/bin/perl -T /usr/bin/pveproxy start
root     28654  0.0  0.1 294232 88532 ?        Ds   13:25   0:00 /usr/bin/perl -T /usr/bin/pveproxy stop
root     25526  0.0  0.1 294072 88556 ?        Ds   13:55   0:00 /usr/bin/perl -T /usr/bin/pvedaemon stop

I apologize if this is a simple linux question. Is there a way to fix the webgui without having to reboot the hardware?

 

[alexskysilk]

check your hosts file. your system name must equal your host name, and must resolve to an ip on your system.

 

[welbo97]

OK. Thank you for the reply. I have confirmed the hosts file on the proxmox system name matches the name in my hosts file and that the ip resolves to it.

What is baffling me is that a kill -9 does not stop the processes. I always thought that was the nuclear option.

 

[alexskysilk]

there are cases where a process turns zombie and cannot be killed. for pveproxy, thats probably storage that disappeared (probably nfs share) or a vm/ct thats a zombie.

 

[welbo97]

So what kind of bullet do you use on a zombie?
I just tried to 'pct list' and 'qm list' to start shutting down the containers and vm's and both of those processes hung too. At this point, it is too unstable for me to live with so I'll go ahead and shut it down.
Thank you for the input.

 

[alexskysilk]

if you can cure the underlying cause, it should come back to life on its own.
If you cant... hard reset (it will not shut down quietly as long as processes are running.)

 

[welbo97]

I don't know what the underlying issue was. I noticed in the logs the spice terminal I left open was hammering pveproxy pretty hard until 4am then stopped. I killed the terminal on the client later but couldn't get pveproxy to come back. The reboot didn't work (as you said) and also locked me out of the terminal (I do not allow root logins over ssh). On hard reset, the system didn't come back up directly because of a usb data drive holding my templates that wouldn't mount. I found it to be corrupt and got the system back up after removing it's mount directive from fstab. Testdisk looks like it was able to fix the usb drive.
So I suppose it will remain a mystery as it is all working properly now.