Proxmox web gui crashes

[Mwhatshisface]

When rebooting my proxmox machine, the gui is accessible for a few seconds, enough for me to login, then crashes. I am not able to ssh into the machine either.
Despite this, the VMs are running and I am able to connect to them over my local network. Oddly enough, I am able to ssh into my proxmox while ssh-ed into one of the VMs. It's like proxmox has built a wall around itself. I've looked through various logs i've found via searches and hven't found any errors. I've tried restarting pve services and used different machines on the network all with the same results. One machine was able to log into ssh but then quickly got disconnected and wasn't able to reconnect.

My router sees proxmox and all VMs on the proper local IP addresses.

I believe the issue is with my router but I'm not sure what has changed.

 

[Moayad]

Hi,

After you are not able to ssh or through GUI could you ping your Proxmox host? if so, please let the ping running and try ssh or log in through the GUI.

please provide us with the following:

ip r
ip a
cat /etc/network/interfaces

 

[fabian]

did you maybe enable the built-in firewall? if you haven't defined the local / management network (correctly), you could definitely see this behaviour (firewall not yet applied on bootup -> connection works, once it's applied it doesn't anymore)

 

[Mwhatshisface]

I believe i figured it out. Not 100% sure on the issue but I think it had to do with my router resolving dchp addresses. I restarted my router and my proxmox machine and haven't had an issue since.

 

[area51x]

I still have this problem from the very start of my installation. The GUI will just freeze and say "connection problem." Then I'm unable to access the GUI for seconds to minutes and then magically it will just come back up. Even during these times when I can't access the proxmox web gui, I can still ping the host and VMs that have their own web interfaces are all still up and working. I have TrueNAS scale as a VM and that is always up and running and accessible through it's own GUI even when I can't get into the proxmox gui. So I know it's a problem with the proxmox management interface. Maybe as fabian stated earlier, it has to do with the built-in firewall. Should I turn that off? I already have a firewall in my physical router. Do I need the proxmox firewall?

 

[area51x]

As an update this is what I see in proxmox right now. Do you think this may be my problem?

 

[Dunuin]

As the warning already suggests...enabling the node firewall won't do anything until you also enable the datacenter firewall. And there are hidden anti-lockout rules in place, so you can't lockout yourself when enabling firewall unless you create a custom rule to explicitly block one of the management ports.

What I would do is...
1.) check the syslog for hints at the time of the unavailable webUI
2.) check the CPU utilization and IO delay graphs at the time of the unavailable webUI if it is high
3.) check if all storages are available via pvesm status
4.) check if SSH is still working when the webUI isn't available and use that to dig deeper and for example run 3 again or check htop or iotop if 2 is high
5.) verify that your network is working properly and for example check that your PVE doesn't got two IPs in the same subnet and that an IP isn't used twice (for example because you gave PVE a static IP that is inside the P range of your routers DHCP server)

 

[area51x]

Ok I will run those checks next time. However, I already have observed that SSH is also down when the webui is down. Another observation is that I attached a VGA monitor and a keyboard directly to the server and am able to use the cli directly when the webui goes down and I am able to ping hosts on my LAN and externally.

In terms of giving the PVE a static IP.. yes I did give it a DHCP reservation from inside by physical router. Is that what you're referring to by static? I'm not setting it statically from within PVE.

pvesm status currently returns this I can try to run it next time it goes down.

pvesm status
Name Type Status Total Used Available %
local dir active 100597760 10769868 89827892 10.71%
local-lvm lvmthin active 1793077248 20441080 1772636167 1.14%
nvme dir active 976284608 69361924 906922684 7.10%

Well guess what... while I was writing this, the webui went down. Can't SSH and pvsem status gives the exact same results with all drives active. I can still ping the PVE from my laptop. I can still ping hosts when typing directly on the server.

Also, yes I have nginx proxy manager running and set up to access PVE but for the purposes of this post I've been accessing it using the actual IP and 8006 port from within the LAN. When the webui is down can't access via IP or via the FQDN.

 

[area51x]

When this was happening early on in my server build, I was wondering whether it was due to some conflict with the IPMI web management built into my motherboard. That was set up to use the IPMI LAN port as well as the eno1 10gb LAN port. I was able to move the webui for the asrock to only the IPMI port which currently has nothing plugged into it and thus I am not even able to access the asrock web-ui. I double checked whether another host had the same IP for my client and there isn't but the host is listed under the name Asrock:

I'm still suspicious this has something to do with these disconnections.

 

[area51x]

Some more info attached in pics (outputs of cat /etc/network/interfaces and ethtool). BTW the webui just came back up so that was a down time of about 15 minutes.

 

[area51x]

Something I just noticed is that I was now able to connect but only using my external subdomain via nginx. I still can't connect using the local IP:8006. Just to be clear neither was working the previous 15-20 minutes.

 

[area51x]

Sorry to use this thread to think out loud. Anyways, I logged on through a different laptop on my LAN was able to connect to the webui. I still can't access the PVE webui from this laptop but I can access a Portainer LXC ui and a TrueNAS vm ui from this laptop which are both running on PVE.

This screenshot is from the other laptop that is currently able to access the webui. Noticed that it says not secure.. Could this be a certificate problem?

 

[area51x]

I ran curl -v https://<domain-or-ip>:8006 directly on the server and got this:

 

[area51x]

"service pveproxy restart" allowed me to be able reconnect
I also added "service pvedaemon restart"