Proxmox VE UI Not Working "proxy detected vanished client connection"

[JosueC]

The entire Proxmox VE Web UI is showing grey question marks & unknown status. This has already happened before, but I assumed it was a bug so I just restarted the host and it seemed to have fixed it. However, the issue keeps coming back, and I can't keep restarting the node. Another thing I noticed, is when I log in to PBS, which is running on the same node, everything with the UI seems fine. Also, I RDPd into one of the vms and attempted to login to the PVE UI through the VM and the login would always fail. However, through ssh it would work fine, and through my own pc it works fine but I get those weird grey question marks. I checked the logs and I also found that I was getting a weird "Proxy detected vanished client connection" error, not sure if this is related.



Things I have already tried:
- Restart Node (Fixes it only temporarily)
- systemctl restart pvedaemon
- systemctl restart pvestatd
- systemctl restart pve-manager
- systemctl restart pve-cluster (Shows the node online status, but only temporarily)


I would really appreciate any help!

*UPDATE*
After a couple of hours I am now unable to log in to the web UI. I keep getting "Login failed. Please try again later". But through SSH it works fine. Super weird...


I also found this when running "journalctl -f"

"Proxy detected vanished client connection"


*UPDATE*
It has happened AGAIN and I tried to reboot the host through SSH but I got the following: "Proxmox Failed to set wall message, ignoring: Transport endpoint is not connected Call to Reboot failed: Transport endpoint is not connected".

It is super inconvenient having to physically power cycle the host server to fix this, specially since it is at a remote location. If someone has a permanent fix for this PLEASE let me know lol.

 

[ManFriday]

running into the same issue after updating..

 

[wavesound]

Seeing this a well on one of our hosts.

 

[greenapple]

Same issue here, also after updating

 

[TGHA-SA]

I have the same issues after latest update.

 

[hprott]

same issue at the last update , a short time after reboot you can log in and after them you can do nothing when you log out and in proxy detected vanished client connection regardless of which method Domainlogin or Linux PAM login

 

[hprott]

same issue at the last update , a short time after reboot you can log in and after them you can do nothing when you log out and in proxy detected vanished client connection regardless of which method Domainlogin or Linux PAM login

today i try after a reboot a update and now work all normally

 

[mastermikioe]

i have been fighting this issue for over a week. after 40 mins of runtime or more the gui simply stops responding.
sometimes you can get ssh access other times not. its like it just goes off into nothing and doesnt respond. ugh...

additionally when attempting updates it says..

The repository 'https://enterprise.proxmox.com/debian/pve bookworm InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.

really no signing?

 

[phanover]

I am having the same issue. Running 8.2.4 and fully up to date. After 20 mins or so all my LXCs go to status unknown. Any ideas gratefully received.

 

[JosueC]

i have been fighting this issue for over a week. after 40 mins of runtime or more the gui simply stops responding.
sometimes you can get ssh access other times not. its like it just goes off into nothing and doesnt respond. ugh...

additionally when attempting updates it says..

The repository 'https://enterprise.proxmox.com/debian/pve bookworm InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.

really no signing?

I've been fighting this for over a month lol. It's STILL happening... it typically happens once a week after I reboot the host. Today it happened again and I tried to reboot the host through SSH but I got the following: "Proxmox Failed to set wall message, ignoring: Transport endpoint is not connected Call to Reboot failed: Transport endpoint is not connected". Ended up having to call someone to physically power cycle the host server.

 

[JosueC]

today i try after a reboot a update and now work all normally

Mine also goes to normal after a physical power cycle, but then it randomly happens again within a week or two. It is becoming extremely frustrating.

 

[mastermikioe]

Something that seems to have corrected the issue with the GUI is ensuring your not overcommitting ram. In previous versions i usually left 2mb on the table for the host with no ill effects. However to keep the GUI stable in this version i am now leaving 8mb for host. i have not had a GUI crash since,

 

[JosueC]

Something that seems to have corrected the issue with the GUI is ensuring your not overcommitting ram. In previous versions i usually left 2mb on the table for the host with no ill effects. However to keep the GUI stable in this version i am now leaving 8mb for host. i have not had a GUI crash since,

Interesting, and you were experiencing the exact same crashes as I am correct?

 

[mastermikioe]

My issues were loss of gui access within 5 minutes of startup and sketchy remote connections to the vms. but after increasing the ram for the host i have not lost the gui access.

 

[JosueC]

My issues were loss of gui access within 5 minutes of startup and sketchy remote connections to the vms. but after increasing the ram for the host i have not lost the gui access.

After what you said yesterday, I limited the ZFS Arc ram consumption because I realized that it was taking up to 45GBs of RAM, hopefully this helps. However, is there a way to manually assign or leave ram JUST for the host? Or is it just what you leave spare?

 

[mastermikioe]

When doling out ram I leave 8gb on the table after assigning it out to VMs. again typically in the past 2gb was sufficient for basic operations. but this version seemed to want more. its not a huge deal on a system with a good amount of ram, such as all of my 7 servers have 64gb.

 

[Steveuk]

So glad I've found this as I've been pulling my hair out trying to get mine back up and running.
I'm at the same point as you guys it started with the grey unknown status now i'm unable to login to the UI but ssh is fine.
I have tried restarting loads of different services now with no help.
I say the SSH is working but not all commands seem to work, for instance

df -h

doesn't show anything just a new blank line leaving me to restart the connection.
I've noticed @mastermikioe you said about increasing the host ram could you please explain how i do thing through the command line.
Thanks

 

[mastermikioe]

So glad I've found this as I've been pulling my hair out trying to get mine back up and running.
I'm at the same point as you guys it started with the grey unknown status now i'm unable to login to the UI but ssh is fine.
I have tried restarting loads of different services now with no help.
I say the SSH is working but not all commands seem to work, for instance

df -h

doesn't show anything just a new blank line leaving me to restart the connection.
I've noticed @mastermikioe you said about increasing the host ram could you please explain how i do thing through the command line.
Thanks

the host has minimum requirements based on the setup you are using. typically 2gb for basic operations +1gb for each Tb of zfs storage you have.
The method is very simple, once you add up what you need for the host simply deduct that from available ram. for instance. if i have 64gb of ram and my calculations say 3gb required for the host. that leaves me with 61gb of assignable ram for vms. thats all there is to it. simply leave ram unassigned so the host can use it. i tried several different amounts and it seemed things settled down once i left 8gb on the table for host.

 

[JosueC]

the host has minimum requirements based on the setup you are using. typically 2gb for basic operations +1gb for each Tb of zfs storage you have.
The method is very simple, once you add up what you need for the host simply deduct that from available ram. for instance. if i have 64gb of ram and my calculations say 3gb required for the host. that leaves me with 61gb of assignable ram for vms. thats all there is to it. simply leave ram unassigned so the host can use it. i tried several different amounts and it seemed things settled down once i left 8gb on the table for host.

Also, if you are using ZFS Storage, by default I believe it uses 50% of all FREE memory. So, you may want to limit ZFS ARC ram consumption, I believe this is what was causing my crashes. My ram would always spike to 96%-98% and my Swap would always be maxed out. Ever since I limited the ZFS ARC everything seems to have stabilized. You can limit it by creating a /etc/modprobe.d/zfs.conf and adding:

Code:
options zfs zfs_arc_min=4294967296 (Example Size)
options zfs zfs_arc_max=8589934592 (Example Size)

 

[Steveuk]

Thanks Guys.
That's helpful, @mastermikioe @JosueC
That's probably my issue then lack of ram.
I don't use zfs just some network shares added
I'm looking into getting one of them low powered N100 devices as I'll only use it for about 3 or 4 vm's or containers.
Would a lack of ram cause me to be locked out of the webui like what's happening at the moment?