Proxmox VNC client disconnecting every few seconds

BrettR · Dec 15, 2018

Hello.

I just upgraded my system to proxmox version 5.3-5/ae681d.

(I performed the upgrade, to try to resolve a separate problem that I was experiencing..when mounting
USB keys into running VMs. Can't get them to mount correctly.)

After the upgrade, I've found a new proxmox server problem?

The VNC console to a running VM ..suddenly disconnects..and then reconnects every few seconds.
This occurs with both an Ubuntu based VM and a Windows 7 based VM.

Steps taken thus far to try to correct/trace this problem:
* Deleted all web certificates in my (Linux Firefox) web browser from the new proxmox server.
(Then, when I first browse to the proxmox server, the proxmox server's certificate is added to
the client web browser's cert cache.)

* Synchronized the time between the client host and the proxmox server (in case it was affecting SSL
sessions.)

* Have seen the following error message in the Firefox browser's 'Console' (debug) window:
Failed when connecting: Connection closed
(Code: 1006)
app.js (8763:21)

* I have a second proxmox server running on this flat network..also recently upgraded to proxmox 5.3-5
and it's client console sessions work correctly. (I tried shutting down the second server..to see if
was interfering with the new proxmox server.. But the problems remained after shutting down the
second server.)

Thanks,

Brett R.

dcsapak · Dec 17, 2018

this sounds like a network issue, are you behind any kind of proxy/firewall that may cause an issue with websockets?

BrettR · Dec 18, 2018

Proxmox 5.3.0 Error Report
==========================
VNC client disconnect errors
----------------------------
(I believe that the following syslog errors are relevant to the VNC client disconnects...)
Entries from the proxmox-malass.aal.gov /var/log/syslog file:
-------------------------------------------------------------
Dec 17 09:34:51 proxmox-malass pvedaemon[13153]: starting vnc proxy UPID

roxmox-malass:00003361:01DBF66E:5C17D02B:vncproxy:700:root@pam:
Dec 17 09:34:52 proxmox-malass pveproxy[12492]: problem with client 192.168.40.131; ssl3_read_bytes: tlsv1 alert unknown ca
Dec 17 09:34:52 proxmox-malass pvedaemon[28238]: <root@pam> end task UPID

roxmox-malass:00003361:01DBF66E:5C17D02B:vncproxy:700:root@pam: OK
Dec 17 09:35:00 proxmox-malass systemd[1]: Starting Proxmox VE replication runner...
Dec 17 09:35:01 proxmox-malass systemd[1]: Started Proxmox VE replication runner.
Dec 17 09:35:02 proxmox-malass pveproxy[25928]: problem with client 192.168.40.131; ssl3_read_bytes: tlsv1 alert unknown ca
Dec 17 09:35:02 proxmox-malass pvedaemon[26888]: <root@pam> starting task UPID

roxmox-malass:00003381:01DBFA9B:5C17D036:vncproxy:700:root@pam:
Dec 17 09:35:02 proxmox-malass pvedaemon[13185]: starting vnc proxy UPID

roxmox-malass:00003381:01DBFA9B:5C17D036:vncproxy:700:root@pam:
Dec 17 09:35:22 proxmox-malass pveproxy[8614]: problem with client 192.168.40.131; ssl3_read_bytes: tlsv1 alert unknown ca
Dec 17 09:35:22 proxmox-malass pvedaemon[26888]: <root@pam> end task UPID

roxmox-malass:00003381:01DBFA9B:5C17D036:vncproxy:700:root@pam: OK
Dec 17 09:35:23 proxmox-malass pvedaemon[28238]: <root@pam> starting task UPID

roxmox-malass:000033AA:01DC02AD:5C17D04B:vncproxy:700:root@pam:
Dec 17 09:35:23 proxmox-malass pvedaemon[13226]: starting vnc proxy UPID

roxmox-malass:000033AA:01DC02AD:5C17D04B:vncproxy:700:root@pam:
Dec 17 09:35:29 proxmox-malass pveproxy[12492]: problem with client 192.168.40.131; ssl3_read_bytes: tlsv1 alert unknown ca

(Testing SSL connections to the SSL server running on the proxmox-malass.aal.gov host.
Seeing "certificate not trusted" error.)

openssl client test (to proxmox-malass.aal.gov:8006)
----------------------------------------------------
[root][~]# openssl s_client -connect proxmox-malass.aal.gov:8006
CONNECTED(00000003)
depth=0 OU = PVE Cluster Node, O = Proxmox Virtual Environment, CN = proxmox-malass.aal.gov
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 OU = PVE Cluster Node, O = Proxmox Virtual Environment, CN = proxmox-malass.aal.gov
verify error:num=27:certificate not trusted
verify return:1
depth=0 OU = PVE Cluster Node, O = Proxmox Virtual Environment, CN = proxmox-malass.aal.gov
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/OU=PVE Cluster Node/O=Proxmox Virtual Environment/CN=proxmox-malass.aal.gov
i:/CN=Proxmox Virtual Environment/OU=d71d42af-870e-4dfe-bbb8-60c217ee2fa4/O=PVE Cluster Manager CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIE0TCCArmgAwIBAgIBATANBgkqhkiG9w0BAQsFADB2MSQwIgYDVQQDDBtQcm94
bW94IFZpcnR1YWwgRW52aXJvbm1lbnQxLTArBgNVBAsMJGQ3MWQ0MmFmLTg3MGUt
NGRmZS1iYmI4LTYwYzIxN2VlMmZhNDEfMB0GA1UECgwWUFZFIENsdXN0ZXIgTWFu
YWdlciBDQTAeFw0xODEyMTMwMDUxMDZaFw0yODEyMTAwMDUxMDZaMGIxGTAXBgNV
BAsTEFBWRSBDbHVzdGVyIE5vZGUxJDAiBgNVBAoTG1Byb3htb3ggVmlydHVhbCBF
bnZpcm9ubWVudDEfMB0GA1UEAxMWcHJveG1veC1tYWxhc3MuYWFsLmdvdjCCASIw
DQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMTiS6qu/iScwzop920C6n6o7ljY
==================================================================================================================
(Testing SSL connections to the SSL server running on a laptop at 192.168.100.2:8006).
Seeing "unable to verify the first certificate" warning...but I don't get the "VNC client disconnect errors"
on this proxmox laptop system. i.e. VNC is working on this proxmox system...)

Second test host - laptop client
--------------------------------
openssl client test (to laptop 192.168.100.2:8006)
--------------------------------------------------
root@bras-laptop:~# openssl s_client -connect 192.168.100.2:8006
CONNECTED(00000003)
depth=0 OU = PVE Cluster Node, O = Proxmox Virtual Environment, CN = proxmox-malass.aal.net
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 OU = PVE Cluster Node, O = Proxmox Virtual Environment, CN = proxmox-malass.aal.net
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/OU=PVE Cluster Node/O=Proxmox Virtual Environment/CN=proxmox-malass.aal.net
i:/CN=Proxmox Virtual Environment/OU=6b3e694f-1cf7-4077-ad7a-2090d18595cb/O=PVE Cluster Manager CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIE0TCCArmgAwIBAgIBATANBgkqhkiG9w0BAQsFADB2MSQwIgYDVQQDDBtQcm94
bW94IFZpcnR1YWwgRW52aXJvbm1lbnQxLTArBgNVBAsMJDZiM2U2OTRmLTFjZjct
NDA3Ny1hZDdhLTIwOTBkMTg1OTVjYjEfMB0GA1UECgwWUFZFIENsdXN0ZXIgTWFu
YWdlciBDQTAeFw0xODEyMDYxMzA2MDdaFw0yODEyMDMxMzA2MDdaMGIxGTAXBgNV
BAsTEFBWRSBDbHVzdGVyIE5vZGUxJDAiBgNVBAoTG1Byb3htb3ggVmlydHVhbCBF
bnZpcm9ubWVudDEfMB0GA1UEAxMWcHJveG1veC1tYWxhc3MuYWFsLm5ldDCCASIw

*** How can I fix the SSL errors on the proxmox.aal.gov system?
(Is there a problem with the certificate that is generated during the installation
of the proxmox system?)

Thanks.

Brett R

Stoiko Ivanov · Dec 18, 2018

Hi,
* Any chance that your browser is sending a client-certificate to the PVE-node?
* Please try with another browser (without any addons/plugins/certificates installed)

AlexanderR · Jan 21, 2019

I have got the exact same issue.... can I provide any info to get this fixed?

Code:

Jan 21 16:57:04 sh01 pveproxy[6397]: problem with client 192.168.1.15; ssl3_read_bytes: sslv3 alert bad certificate
Jan 21 16:57:04 sh01 pvedaemon[14189]: starting vnc proxy UPID:sh01:0000376D:01031FB9:5C45EBD0:vncproxy:101:root@pam:
Jan 21 16:57:11 sh01 pveproxy[6397]: problem with client 192.168.1.15; ssl3_read_bytes: sslv3 alert bad certificate
Jan 21 16:57:11 sh01 pvedaemon[14204]: starting vnc proxy UPID:sh01:0000377C:0103227F:5C45EBD7:vncproxy:101:root@pam:
Jan 21 16:57:17 sh01 pveproxy[8358]: problem with client 192.168.1.15; ssl3_read_bytes: sslv3 alert bad certificate
Jan 21 16:57:18 sh01 pvedaemon[14234]: starting vnc proxy UPID:sh01:0000379A:010324E6:5C45EBDD:vncproxy:101:root@pam:
Jan 21 16:57:18 sh01 pveproxy[6397]: problem with client 192.168.1.15; ssl3_read_bytes: sslv3 alert bad certificate

Stoiko Ivanov · Jan 22, 2019

Does your Browser send a client certificate?
If yes - disable it for the PVE-site or use a different browser to access it

AlexanderR · Jan 22, 2019

Stoiko Ivanov said:
Does your Browser send a client certificate?
If yes - disable it for the PVE-site or use a different browser to access it

Thank you for your response!

How can I check if my browser is sending a client certificate?

I cant access the novnc console with any pc. I also tried PCs that never visited the webinterface before.. All started when I requested the LE-Certificate..

Stoiko Ivanov · Jan 22, 2019

Sorry - misread the error-message - this looks like a different problem
Did you try restarting the `pveproxy`, `pvedaemon` services?

AlexanderR · Jan 22, 2019

yes, I already tried that... it didnt help to fix the problem..

Stoiko Ivanov · Jan 22, 2019

* Is this a single node or a cluster?
* if it's a cluster - do you connect to a vm on the same node as you're connected to with your browser?
* can you connect to the node with `openssl s_client` - what's the output?

AlexanderR · Jan 22, 2019

It is a single node.

the output of 'openssl s_client':

Code:

CONNECTED(00000003)
---
Certificate chain
 0 s:CN = sh01.pm.dvision.org
   i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
 1 s:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGXjCCBUagAwIBAgISAw0+Hl0w14XoW3xIIzgE/rLvMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xOTAxMjExMDM4NTJaFw0x
OTA0MjExMDM4NTJaMB4xHDAaBgNVBAMTE3NoMDEucG0uZHZpc2lvbi5vcmcwggIi
MA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDE+5FRdx2ujtW5ki04fcGDP5dN
/ez+2b6ZtYV2sFxJrt6EkWVWgQIb9jgeBlSqE2yBaaJhXmrarjyGo1xJmKh20iKH
ML2IX2ppdaNmE9d/Xll0yrVU59VAAjo72qB/yJtoO5/J0MfbEOS+xDjaiu+A/pZJ
NGR2ma63/ODEv5vkijeJ4k5Yp6/TfVz2T3LDF5BxULR443tszJALIIwtZJeOZT4J
7IigOQOm5yMsAF77XpGJQKrB8+uWWcMLFDAwIf8YlFjDp+doJElhzdrz/snyue/R
FYSu3Vwskxj1mYoA7QH0mJmujY6mqemohkVE3pt1RpUmKukAMKxgsAHEbKaXc6/0
aPssoW3/SkrOcG8oYyq01bVcSPRhwdqP7JsD0QCDfZMfBTNV+RtUwZJHqPj3Ix18
f1x/CPYMSgjCDUupEFMt463pPvxb6cbPXzKbLDVzoXd6txX36qd5hGMxAYYtt7F+
PDsF2EAdwSIBNCD+9WK0K7aVXYh5PsFhzSQW7MTd+lsrBMrAtPS9LLGiASJMkekF
CpvFGzuFXHw+mGaN90E+ealkGY4ckNAFeC36du/0L3UPhNPod/5aof/veSnMYd/4
wV5H4+xpZ8kp8COpH16MKx7VDSCfbv5//B/k0CqPU1whajBwE3/nfv9V7+iXSK3A
30umlyx4zdbLcUo4wQIDAQABo4ICaDCCAmQwDgYDVR0PAQH/BAQDAgWgMB0GA1Ud
JQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQW
BBQ4ynGzIpiJ7r37aGZ8TifY80fdYDAfBgNVHSMEGDAWgBSoSmpjBH3duubRObem
RWXv86jsoTBvBggrBgEFBQcBAQRjMGEwLgYIKwYBBQUHMAGGImh0dHA6Ly9vY3Nw
LmludC14My5sZXRzZW5jcnlwdC5vcmcwLwYIKwYBBQUHMAKGI2h0dHA6Ly9jZXJ0
LmludC14My5sZXRzZW5jcnlwdC5vcmcvMB4GA1UdEQQXMBWCE3NoMDEucG0uZHZp
c2lvbi5vcmcwTAYDVR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAm
BggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEEBgorBgEE
AdZ5AgQCBIH1BIHyAPAAdwB0ftqDMa0zEJEhnM4lT0Jwwr/9XkIgCMY3NXnmEHvM
VgAAAWhwNMHNAAAEAwBIMEYCIQD9bmQx9fMwOH2VRo6OyzvbBY6qbbXit8mEO3+Z
UaTcPwIhAPhNdXnJTjrWGUb+5mjL5/gaVnS+NKdr9/F6VkgjvtjCAHUAKTxRllTI
OWW6qlD8WAfUt2+/WHopctykwwz05UVH9HgAAAFocDTCIAAABAMARjBEAiBP50ir
Z3lb/SO8jQeyyFssn4yAxt+2iUNRm07HblknuAIgMdfMr2yD2/0qE0Dfz5f99QdR
DlLrE0PMlMaizDshgyIwDQYJKoZIhvcNAQELBQADggEBABq0dJAfdI585DBwTjY+
gwblTCXAvheekYGunuI1jruwYbnCZCpczUSQGBHbh3+qkkyxPJHQP5IxKPAiz3Oz
MK+60oK9I9hwcvxwN/9taLglciKYXaUoSu5XazB7U9MfH9tOCrCuSeLWshMvH0Fu
SaBpyRsXLtizlAS7vqFamRvmMPBGrYO6iPxzG5DZMX6x5gMv2TL7oMo5YWnreIY4
iELc/fyHsrlcdgHmXKzLDDAfh1/kX9IinlPmGH+cvnt0wajDFg0G2IE+duhm7WQ6
mOOiQySTKZOKbeZdsh8Q2KlDTeaWHp7jkIYGH33ibaklOS4TcN+6lFd903GFL0y7
inU=
-----END CERTIFICATE-----
subject=CN = sh01.pm.dvision.org

issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3708 bytes and written 414 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: E35FEA41EF4796FA0BD48DDE0DA4FB2EFA9F449CFA413726784623769F8E85C1
    Session-ID-ctx:
    Master-Key: E879F0375E02CB6A17C3CA505AACFE90336892AE304CEDBB11EC33DF6ABFD08EB345348E53B83D2450B28066DE28C3D1
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 60 5e ee a0 83 3e 4c 1b-91 37 68 b6 48 38 46 fb   `^...>L..7h.H8F.
    0010 - 23 59 9f 15 e3 57 ab 0e-1e a7 39 48 be 4a b3 c3   #Y...W....9H.J..
    0020 - d9 8f 80 b5 90 8a 47 5b-df 60 94 52 a9 9e b8 fa   ......G[.`.R....
    0030 - 91 e2 7c 12 77 7c 52 6b-3a 5b 36 70 fc 01 5c d5   ..|.w|Rk:[6p..\.
    0040 - 7c fc 8d 2d d2 8e 28 7f-0b 66 d2 0c 60 11 cb 10   |..-..(..f..`...
    0050 - 26 af 6a af 77 6f 85 97-57 60 3d 75 a4 e5 a7 09   &.j.wo..W`=u....
    0060 - ee 74 2b 8b ca 8d 81 6c-34 78 26 5a 32 e8 8f 2c   .t+....l4x&Z2..,
    0070 - e7 f1 c8 65 f5 bf b1 2d-e6 cd 37 52 7b 53 ca 3f   ...e...-..7R{S.?
    0080 - 14 b5 48 31 2e c2 40 fa-a9 d7 a2 7c fe 9c 05 49   ..H1..@....|...I
    0090 - d6 4e f8 2e 0a d7 94 24-86 7e 8d 12 cf d3 1d 81   .N.....$.~......
    00a0 - ed c0 50 19 c8 be 75 02-7c 34 4f b5 3c b0 a9 7c   ..P...u.|4O.<..|

    Start Time: 1548165733
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

Stoiko Ivanov · Jan 22, 2019

Strange ...

* are there any middle-boxes between you and the PVE-node? possibly mangling the traffic?
* maybe check with the browser development-tools what's happening between - web-access to the PVE-GUI, and when you click on the noVNC-button
* also try starting noVNC in a window of its own, instead of within the main browser window

AlexanderR · Jan 22, 2019

no, the problem persists regardless what is in between or not...

Opening the noVNC in a window also doesnt help.

Here is a screenshot from the developer tools of my browser:

AlexanderR · Jan 23, 2019

@Stoiko Ivanov any ideas / updates about the bug?

Stoiko Ivanov · Jan 23, 2019

Hi,

hmm - sadly nothing sticks from your description, and I cannot reproduce the problem here -not even on a machine with LE-Certificate.

It seems that the connection has a problem with the UPGRADE request to switch to websocket.
Which browser version are you using?

* make sure to use a browser without any plugins installed

* I would also check the Network-tab of the developer utils instead of the js-console
* you could try to use something like mitmproxy to sniff the traffic after the TLS-encryption to gain some insights

AlexanderR · Jan 25, 2019

Good News: After restarting the server a few days later, the issue is gone... kind of strange, because I restarted the server already several times before... @Stoiko Ivanov thank you for your help

Stoiko Ivanov · Jan 25, 2019

You're welcome - always hard to catch those hard-to-reproduce errors! Glad it's working again. Please mark the Thread as Solved, so that others know what to expect - Thanks!

plastilin · May 17, 2019

Hello. i have the same issue on fresh install of proxmox. But i try to add ACME cert by LetsEncrypt after install. Now i fix in logs

May 17 11:22:22 alpha pveproxy[44603]: problem with client 192.168.27.1; ssl3_read_bytes: tlsv1 alert unknown ca

BrettR · Dec 6, 2019

My Proxmox Node is located in a DMZ. There is a firewall between my VNC client and the proxmox node.

Which network ports need to be open..to prevent the random VNC disconnects? (e.g. 80, 443)

Which network protocols need to be allowed through the firewall (e.g. TCP, UDP, etc)

Thanks,
Brett

Stoiko Ivanov · Dec 9, 2019

BrettR said:
Which network ports need to be open..to prevent the random VNC disconnects? (e.g. 80, 443)

Which network protocols need to be allowed through the firewall (e.g. TCP, UDP, etc)

Check our reference documentation:
https://pve.proxmox.com/pve-docs/chapter-pve-firewall.html#_ports_used_by_proxmox_ve

I hope this helps!

Proxmox VNC client disconnecting every few seconds

Member

Proxmox Staff Member

Member

Proxmox Staff Member

Well-Known Member

Proxmox Staff Member

Well-Known Member

Proxmox Staff Member

Well-Known Member

Proxmox Staff Member

Well-Known Member

Proxmox Staff Member

Well-Known Member

Well-Known Member

Proxmox Staff Member

Well-Known Member

Proxmox Staff Member

Renowned Member

Member

Proxmox Staff Member