[SOLVED] GUI and ssh disconnections

dejieres

New Member
Mar 17, 2023
16
2
3
Hello,

I'm using PVE for some years now, and I'm very happy with it. It's great software.

But, for approximately 1 or 2 monthes, I encounter connexion problems with GUI and ssh.
This week, for example, I could connect without any problem through GUI (didn't use ssh). Today, I had some maintenance to do (via ssh), and after a while (maybe 1 hour) connection was dropped, with a message saying something about a 'client loop'.

Bash:
root@pve:~# debug3: recv - from CB ERROR:10054, io:000002435B8AC010
debug3: send packet: type 1
debug3: send - WSASend() ERROR:10054, io:000002435B8AC010
client_loop: send disconnect: Connection reset
debug3: Successfully set console output code page from 65001 to 65001
debug3: Successfully set console input code page from 65001 to 850

I could reconnect once, but connection was closed after maybe 15 mn, with the same error.

Currently, I'm working through a VPN, but I had exactly the same problems when working on location.

Since, no way to connect again :

Bash:
C:\Users\Denis>ssh root@192.168.1.250 -vvv
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
debug3: Failed to open file:C:/Users/Denis/.ssh/config error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2
debug2: resolve_canonicalize: hostname 192.168.1.250 is address
debug2: ssh_connect_direct
debug1: Connecting to 192.168.1.250 [192.168.1.250] port 22.
debug3: finish_connect - ERROR: async io completed with error: 10061, io:00000200078CC420
debug1: connect to address 192.168.1.250 port 22: Connection refused
ssh: connect to host 192.168.1.250 port 22: Connection refused

The only way to get access to this server is to log in from the Linux console. But, since I'm not on site now, I'm completely stuck.

If I wait long enough, ssh may accept connections again, but it seems completely random.

The server is a Dell PowerEdge, no cluster. I did some hardware test, and everything seems fine with this server.

I saw several posts talking about connection problems (GUI or ssh), but nothing helped. I tried to regenerate the certificates, didn't help either.

All VMs are still accessible and work correctly.

I have no idea where to look, if someone has any idea...

Many thanks in advance,

dejieres
 
  • Like
Reactions: juancarlosmx
Hi,


Even the VMs can't access to your PVE server? - if no, can you SSH into your PVE server from a VM then see the (syslog, and the network configuration, firewall)?
 
Hello,

Thanks for you answer :)
Well, I must admit I didn't try from a VM. I'll test it the next time the problem appears, because this situation isn't permanent.

Another thing that is annoying when this happens : communication with the PBS server also fails, so my backups can't run reliably.
Here is part of the syslog :
Code:
Mar 20 07:50:29 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 07:50:29 pve pvestatd[1169]: status update time (7.200 seconds)
Mar 20 07:50:39 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 07:50:39 pve pvestatd[1169]: status update time (7.174 seconds)
Mar 20 07:50:49 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 07:50:49 pve pvestatd[1169]: status update time (7.167 seconds)
Mar 20 07:50:59 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 07:50:59 pve pvestatd[1169]: status update time (7.175 seconds)
Mar 20 07:51:19 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 07:51:19 pve pvestatd[1169]: status update time (7.202 seconds)
Mar 20 07:51:29 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 07:51:29 pve pvestatd[1169]: status update time (7.167 seconds)
Mar 20 07:51:39 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 07:51:39 pve pvestatd[1169]: status update time (7.200 seconds)
Mar 20 07:51:49 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 07:51:49 pve pvestatd[1169]: status update time (7.193 seconds)
Mar 20 07:55:39 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 07:55:39 pve pvestatd[1169]: status update time (7.189 seconds)
Mar 20 07:55:49 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 07:55:49 pve pvestatd[1169]: status update time (7.174 seconds)
Mar 20 07:57:39 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 07:57:39 pve pvestatd[1169]: status update time (7.218 seconds)
Mar 20 07:57:49 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 07:57:50 pve pvestatd[1169]: status update time (7.171 seconds)
Mar 20 08:06:40 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:06:40 pve pvestatd[1169]: status update time (7.188 seconds)
Mar 20 08:06:49 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:06:49 pve pvestatd[1169]: status update time (7.204 seconds)
Mar 20 08:06:59 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:06:59 pve pvestatd[1169]: status update time (7.185 seconds)
Mar 20 08:07:09 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:07:09 pve pvestatd[1169]: status update time (7.192 seconds)
Mar 20 08:07:49 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:07:49 pve pvestatd[1169]: status update time (7.183 seconds)
Mar 20 08:07:59 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:07:59 pve pvestatd[1169]: status update time (7.175 seconds)
Mar 20 08:08:10 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:08:10 pve pvestatd[1169]: status update time (7.176 seconds)
Mar 20 08:08:19 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:08:19 pve pvestatd[1169]: status update time (7.186 seconds)
Mar 20 08:16:29 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:16:29 pve pvestatd[1169]: status update time (7.195 seconds)
Mar 20 08:16:39 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:16:39 pve pvestatd[1169]: status update time (7.235 seconds)
Mar 20 08:16:49 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:16:49 pve pvestatd[1169]: status update time (7.256 seconds)
Mar 20 08:16:59 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:17:00 pve pvestatd[1169]: status update time (7.184 seconds)
...

Mar 20 08:28:19 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:28:19 pve pvestatd[1169]: status update time (7.180 seconds)
Mar 20 08:28:29 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:28:29 pve pvestatd[1169]: status update time (7.178 seconds)
Mar 20 08:38:49 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:38:49 pve pvestatd[1169]: status update time (7.187 seconds)
Mar 20 08:38:59 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:38:59 pve pvestatd[1169]: status update time (7.183 seconds)
Mar 20 08:55:39 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 08:55:39 pve pvestatd[1169]: status update time (7.173 seconds)
Mar 20 09:06:40 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:06:40 pve pvestatd[1169]: status update time (7.183 seconds)
Mar 20 09:06:49 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:06:49 pve pvestatd[1169]: status update time (7.174 seconds)
Mar 20 09:11:09 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:11:09 pve pvestatd[1169]: status update time (7.188 seconds)
Mar 20 09:11:19 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:11:19 pve pvestatd[1169]: status update time (7.189 seconds)
Mar 20 09:11:29 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:11:30 pve pvestatd[1169]: status update time (7.188 seconds)
Mar 20 09:13:20 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:13:20 pve pvestatd[1169]: status update time (7.190 seconds)
Mar 20 09:13:29 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:13:29 pve pvestatd[1169]: status update time (7.187 seconds)
Mar 20 09:13:39 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:13:39 pve pvestatd[1169]: status update time (7.176 seconds)
Mar 20 09:13:49 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:13:49 pve pvestatd[1169]: status update time (7.198 seconds)
Mar 20 09:14:29 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:14:29 pve pvestatd[1169]: status update time (7.181 seconds)

...

Mar 20 09:36:39 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:36:39 pve pvestatd[1169]: status update time (7.181 seconds)
Mar 20 09:42:59 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:42:59 pve pvestatd[1169]: status update time (7.198 seconds)
Mar 20 09:43:09 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:43:09 pve pvestatd[1169]: status update time (7.175 seconds)
Mar 20 09:49:20 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:49:20 pve pvestatd[1169]: status update time (7.193 seconds)
Mar 20 09:49:29 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:49:29 pve pvestatd[1169]: status update time (7.202 seconds)
Mar 20 09:49:39 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:49:39 pve pvestatd[1169]: status update time (7.177 seconds)
Mar 20 09:49:49 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:49:49 pve pvestatd[1169]: status update time (7.202 seconds)
Mar 20 09:49:59 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:49:59 pve pvestatd[1169]: status update time (7.193 seconds)
Mar 20 09:59:20 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:59:20 pve pvestatd[1169]: status update time (7.185 seconds)
Mar 20 09:59:29 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:59:29 pve pvestatd[1169]: status update time (7.172 seconds)
Mar 20 09:59:39 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:59:39 pve pvestatd[1169]: status update time (7.182 seconds)
Mar 20 09:59:49 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 09:59:49 pve pvestatd[1169]: status update time (7.180 seconds)
Mar 20 10:10:39 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 10:10:39 pve pvestatd[1169]: status update time (7.183 seconds)
Mar 20 10:10:49 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 10:10:49 pve pvestatd[1169]: status update time (7.187 seconds)
Mar 20 10:10:59 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 10:10:59 pve pvestatd[1169]: status update time (7.183 seconds)
Mar 20 10:11:10 pve pvestatd[1169]: PBS_sauvegarde2: error fetching datastores - 500 Can't connect to 192.168.1.248:8007 (Connection timed out)
Mar 20 10:11:10 pve pvestatd[1169]: status update time (7.189 seconds)

I have no problem accessing the PBS GUI.

If you need more information, ask me please. I don't know where to search for this weir problem...
 
What says the output of `pvesm status` command when you see the above log?

It seems to be a network problem. When the issue occurred, the pvestatd process on server 'pve' encountered a 500 error while trying to fetch datastores from server 'PBS_sauvegarde2'.
 
Problem just occurred again. I was connected to PVE via ssh, and PuTTY froze suddenly.

I immediately connected via ssh into one of the VM and tried to connect to PVE from there, but connection is rejected :
Bash:
administrateur@DebianData:~$ ssh root@192.168.1.250 -vvv
OpenSSH_8.4p1 Debian-5+deb11u1, OpenSSL 1.1.1n  15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.1.250 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/administrateur/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/administrateur/.ssh/known_hosts2'
debug2: ssh_connect_direct
debug1: Connecting to 192.168.1.250 [192.168.1.250] port 22.
debug1: connect to address 192.168.1.250 port 22: Connection refused
ssh: connect to host 192.168.1.250 port 22: Connection refused
administrateur@DebianData:~$ administrateur@DebianData:~$ ssh root@192.168.1.250 -vvv
OpenSSH_8.4p1 Debian-5+deb11u1, OpenSSL 1.1.1n  15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.1.250 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/administrateur/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/administrateur/.ssh/known_hosts2'
debug2: ssh_connect_direct
debug1: Connecting to 192.168.1.250 [192.168.1.250] port 22.
debug1: connect to address 192.168.1.250 port 22: Connection refused
ssh: connect to host 192.168.1.250 port 22: Connection refused
administrateur@DebianData:~$

This is one constant : PVE can't be accessed, via GUI or ssh, but all VM can always be accessed without any problem.

I can't run pvesm status for now, since I'm locked out from the PVE server. I just have to wait until it's willing to come back...
 
Last edited:
did you change something ?
what is your mtu ?
Hello pille99,

I didn't change anything in the configuration of the server. This problem just appeared approximately 2 months ago.

I keep this server up to date, applying updates at least twice a month.

I have currently no access, so I can't retrieve any information from the machine. Still no problem for the VMs, they are fully functional and I can access them via ssh or RDP
 
GUI is back, but ssh still refused from the VM, but OK if I ssh directly to the server.

I'm loosing my mind...o_O
 
Last edited:
i had a coule of days ago that my complete networ was set offline. only ssh was woking. the reason was the mtu value. was original set to 9000 and was woring just fine, after the last update a couple of days back, the value didnt work anymore. the new value which worked was 8972. try with this value
 
MTU is default (1500), I never changed it.
Code:
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 50:9a:4c:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    altname enp3s0f0
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 50:9a:4c:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    altname enp3s0f1
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:21:c9:a2:8e:d2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.250/24 brd 192.168.1.255 scope global vmbr0
       valid_lft forever preferred_lft forever

What size would you suggest ?
 
Your situation has a few indications of a duplicate IP. You should record and compare ARP tables/MAC addresses before and during a problem.

You can also create an escape hatch for yourself by adding an additional private IP to vmbr0, ie 10.10.10.10/24 and one of your VMs, ie 10.10.10.11/24. If its a duplicate IP then you should be able to reach your PVE through that IP from the VM (or another host on the network with 10.x.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
@bbgeek17
Seems you are close to the right answer :)
Bash:
administrateur@DebianData:~$ sudo arp-scan 192.168.1.0/24
Interface: ens18, type: EN10MB, MAC: 3a:92:a6:78:cd:9b, IPv4: 192.168.1.242
Starting arp-scan 1.9.7 with 256 hosts (https://github.com/royhills/arp-scan)
...
192.168.1.250   7a:21:c9:a2:8e:d2       (Unknown: locally administered)
...
192.168.1.250   84:1b:5e:83:df:c4       NETGEAR (DUP: 2)
...
I just have to find that device, now... At least, I know it's a Netgear.
 
Well, seems my problem is solved. I reconfigured the WiFi repeater that took the server's IP address (it reverted to DHCP, I don't know why. I don't use DHCP on this small LAN.)

Sorry for the noise, since all this was not related to proxmox, and thank you everyone for pointing me in the right direction.
 
Last edited:
  • Like
Reactions: leesteken

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!