Since update to 4, GUI error Connection error - Timeout

[vikozo]

Hello
i found out that since the update to 4.0-48
if you select a existing VM and wish to change something like (Options - Start at boot - edit) set to yes
it go a long time and get a error Connection error - Timeout.
But the server is up and running

have a nice day
vincent

 

[Ovidiu]

Very similar experience over here. Not sure what changed, maybe yesterday's updates?i.e. I ma trying to remove a storage, I get: Connection error 596: Connection timed out

 

[wolfgang]

Hi,
did you flush the browser cache and reload the page?

 

[Ovidiu]

yes and all was good, now I am encountering the same issue again, my GUi in the browser is loading and loading as if the server was down, luckily it isn't its just the GUI which is sluggish. Seeing these in my logs:

Nov 29 18:29:03 james pveproxy[11198]: proxy detected vanished client connection

 

[Ovidiu]

tried another browser and can't even login while everything just times out with the browser where I am already logged in. still connected via ssh and wodnerign what is going on here.

 

[Ovidiu]

all I see is connection timed out (596) and after starting pveproxy with: pveproxy start --debug I see lines like these in the log file:

87.156.169.47 - root@pam [29/Nov/2015:19:56:23 +0100] "GET /api2/png/nodes/james/lxc/100/rrd?ds=diskread,diskwrite&timeframe=week&cf=AVERAGE&_dc=2 HTTP/1.1" 596 -87.156.169.47 - root@pam [29/Nov/2015:19:56:23 +0100] "GET /api2/png/nodes/james/lxc/100/rrd?ds=cpu&timeframe=week&cf=AVERAGE&_dc=3 HTTP/1.1" 596 -87.156.169.47 - root@pam [29/Nov/2015:19:56:26 +0100] "GET /api2/png/nodes/james/lxc/100/rrd?ds=mem,maxmem&timeframe=week&cf=AVERAGE&_dc=3 HTTP/1.1" 596 -87.156.169.47 - root@pam [29/Nov/2015:19:56:26 +0100] "GET /api2/json/nodes/james/lxc/100/status/current HTTP/1.1" 500 -87.156.169.47 - root@pam [29/Nov/2015:19:56:38 +0100] "GET /api2/png/nodes/james/lxc/100/rrd?ds=netin,netout&timeframe=week&cf=AVERAGE&_dc=3 HTTP/1.1" 596 -87.156.169.47 - root@pam [29/Nov/2015:19:56:41 +0100] "GET /api2/png/nodes/james/lxc/100/rrd?ds=diskread,diskwrite&timeframe=week&cf=AVERAGE&_dc=3 HTTP/1.1" 596 -87.156.169.47 - root@pam [29/Nov/2015:19:56:53 +0100] "GET /api2/png/nodes/james/lxc/100/rrd?ds=cpu&timeframe=week&cf=AVERAGE&_dc=4 HTTP/1.1" 596 -87.156.169.47 - root@pam [29/Nov/2015:19:56:53 +0100] "GET /api2/png/nodes/james/lxc/100/rrd?ds=mem,maxmem&timeframe=week&cf=AVERAGE&_dc=4 HTTP/1.1" 596 -87.156.169.47 - root@pam [29/Nov/2015:19:56:56 +0100] "GET /api2/png/nodes/james/lxc/100/rrd?ds=netin,netout&timeframe=week&cf=AVERAGE&_dc=4 HTTP/1.1" 596 -87.156.169.47 - root@pam [29/Nov/2015:19:56:56 +0100] "GET /api2/json/nodes/james/lxc/102/status/current HTTP/1.1" 596 -87.156.169.47 - root@pam [29/Nov/2015:19:56:56 +0100] "GET /api2/json/cluster/tasks HTTP/1.1" 200 100887.156.169.47 - root@pam [29/Nov/2015:19:56:56 +0100] "GET /api2/json/cluster/resources HTTP/1.1" 200 54287.156.169.47 - root@pam [29/Nov/2015:19:57:08 +0100] "GET /api2/png/nodes/james/lxc/100/rrd?ds=diskread,diskwrite&timeframe=week&cf=AVERAGE&_dc=4 HTTP/1.1" 596 -87.156.169.47 - root@pam [29/Nov/2015:19:57:22 +0100] "GET /api2/png/nodes/james/lxc/100/rrd?ds=cpu&timeframe=week&cf=AVERAGE&_dc=5 HTTP/1.1" 596 -87.156.169.47 - root@pam [29/Nov/2015:19:57:24 +0100] "GET /api2/png/nodes/james/lxc/100/rrd?ds=mem,maxmem&timeframe=week&cf=AVERAGE&_dc=5 HTTP/1.1" 596 -87.156.169.47 - root@pam [29/Nov/2015:19:57:24 +0100] "GET /api2/png/nodes/james/lxc/100/rrd?ds=netin,netout&timeframe=week&cf=AVERAGE&_dc=5 HTTP/1.1" 596 -87.156.169.47 - root@pam [29/Nov/2015:19:57:26 +0100] "GET /api2/png/nodes/james/lxc/100/rrd?ds=diskread,diskwrite&timeframe=week&cf=AVERAGE&_dc=5 HTTP/1.1" 596 -87.156.169.47 - root@pam [29/Nov/2015:19:57:26 +0100] "GET /api2/json/nodes/james/lxc/103/status/current HTTP/1.1" 596 -

 

[Ovidiu]

this is quite ridiculous, I restarted the host and everything is fine again. absolutely no clue what's happening here :-(

 

[Ovidiu]

And here we go again, GUI isn't reacting anymore, getting Connection timed out (596) In the logs the usual: Nov 30 20:14:18 james pveproxy[20751]: proxy detected vanished client connection Luckily the guests are still up and running but cannot interact with the GUI.Tried different browsers/Pcs, no luck.Help needed to analyze this please.

 

[RobFantini]

when the issue happens, under 'Datacenter' is the system colored red?

 

[Ovidiu]

Yes it is, it looks like this:


I have finally found a thread which seems to deal with the exact same issue: http://forum.proxmox.com/threads/17709-can-t-login-into-website-anymore
The quick fix is:

If all your VMs are running but the GUI shows all offline or GUI having issues, try to run the following and see if GUI starts behaving.
#service pvedaemon restart
#service pvestatd restart
#service pveproxy restart

but I am yet to understand the full thread. My circumstances are similar, after treading that thread I realize that I too had added a NFS share for backups, when this issue first happened, the GUI died and in my logs I could see problems connecting to that share so I managed to delete it and add it again and now days later the same problem again.

Will re-read that thread I linked to trying to understand it all, any more pointers are welcome.

 

[tom]

Yes it is, it looks like this: http://screencast.com/t/kOPSualS9

..

Please do NOT use external links for logs or screenshots, instead, upload all in this forum.

External sources are known to disappear soon.

 

[RobFantini]

the issue may be network related.

check logs to see if a backup or large file transfer caused red system at pve .

something happened before the red. try limiting bandwidth for the backups , file transfers etc.

 

[Ovidiu]

Apologies, I have corrected that.

 

[Ovidiu]

As mentioned above in the last thread I linked to, it was a stale NFS share that caused all this trouble.

mount | grep nfs

showed me a list of nfs shares, I found the stale one and unmounted it:

umount -l /PATH/OF/BUSY-DEVICE
umount -f /PATH/OF/BUSY-NFS(NETWORK-FILE-SYSTEM)

I only figured this out because I got an email from a failed cronjob: run-parts: /etc/cron.daily/mlocate exited with return code 1
Found this thread explaining it: https://gomadtech.wordpress.com/2013/09/16/proxmox-updatedb-mlocate-error-received-via-e-mail/