Getting a '502 - Bad gateway' when proxying the vncwebsocket.

michael-h

New Member
Apr 18, 2023
1
1
1
Hello,

I am currently using Nginx to embed a console onto an external webpage. To accomplish this, I am proxying the content to the Proxmox host. Please refer to the Nginx configuration below:

Nginx configuration:
Code:
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    return 301 https://example.com$request_uri;
}
server {
    listen 443 ssl;
    listen [::]:443 ssl;
    ssl_certificate /etc/ssl/certs/example.com.pem;
    ssl_certificate_key /etc/ssl/private/example.com.key;
    ssl_session_timeout 1d;
    root /var/www/example.com;
    index index.html;
     location /console/ {
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Host proxmox-host.com;
        proxy_pass https://proxmox-host.com/;
    }

    location /api2/json {
        proxy_pass https://proxmox-host.com/api2/json;
        proxy_http_version 1.1;
        proxy_set_header Host proxmox-host.com;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Authorization  "PVEAPIToken=TOKEN";    
    }
    location /novnc/ {
        proxy_pass https://proxmox-host.com/novnc/;
    }
}


The problem I am experiencing:
With the above Nginx configuration, I can now access the console via an external web page. All calls made by the novnc client work. However, as soon as I make the call to establish the websocket connection, I usually get a 502 bad gateway error after 6 seconds. Occasionally, it is possible to connect to the console, and then the connection is established after 2 seconds. But this occurs at most 2 times out of 10 attempts. Most of the time, the websocket connection is simply not established.

Scherm­afbeelding 2023-04-28 om 11.03.00.png


What I have already tried:
  • Tried setting different headers, such as keep-alive, in the proxy_set_header.
  • Checked if Nginx sets a timeout after 6 seconds, but this doesn't seem to be the case.
  • Checked the Nginx logs, which indicate that the handshake has failed.


My specific question:
Does anyone know why the websocket connection is sometimes established but mostly not? And why it times out after 6 seconds when it fails?
 
  • Like
Reactions: Kolovatoff1
@yswery, personally I didn't figure this out myself, but if I remember correctly, it had something to do with routing within Proxmox. The issue was related to IPv4 and IPv6, but I'm not exactly sure. It was a problem with correctly resolving the address or something similar. Hopefully, this information can help you make some progress.
 
May I ask how it was ultimately resolved? I also encountered this problem.
Sometimes it can work for lxc vncwebsocket, but more often it raise 502.
I always get 502 for vm vncwebsocket.:(

I also tried to replace nginx with apache, but the problem still persists.

PVE 8.2.4, nginx 1.26, apache 2.4.62

Code:
apache Error:
10.0.0.103 - - [03/Sep/2024:03:21:40 +0000] "GET /api2/json/nodes/www/qemu/101/vncwebsocket?port=5901&vncticket=PVEVNC%3A66D680A2%3A%3AbbmHwo5wF7yPtheqYHupnIXip9KmhO428MsTdABqPcD9g79t4qHKoFlv%2BbAc0fANQ5Nz4CFIrbzVJwO1lkMhUpnaRSVMDDpjAXy81uWWdxpC8DswOP6yIjypyMu1r6hiOKG%2Fb%2Bd5vZ4TuJWgN%2BKxiC%2Ffjmv5MHrIJUX%2FpdT14i4%2B7q8rYHHVTpyeCC3DyWazzMErb5d1J2geiSWc%2B%2Bq3A%2FqU0NZvNjUvkYaObdFQKt%2F4Xe5izA90rUIQYRwsZcirmtdstLUx71bPZ5woAd7gDRqS3O%2Fu7%2F9i56DjeLK3HEnXVqgzZzp8Fq1EKkYVvwSGA%2FyxXqCdNRGX3ouZqhGbbw%3D%3D HTTP/1.1" 502 341
[Tue Sep 03 03:22:48.312491 2024] [proxy_http:error] [pid 41:tid 41] (70014)End of file found: [client 10.0.0.103:35760] AH01102: error reading status line from remote server 10.0.0.1:8006
[Tue Sep 03 03:22:48.312519 2024] [proxy:error] [pid 41:tid 41] [client 10.0.0.103:35760] AH00898: Error reading from remote server returned by /api2/json/nodes/www/qemu/101/vncwebsocket
10.0.0.103 - - [03/Sep/2024:03:22:43 +0000] "GET /api2/json/nodes/www/qemu/101/vncwebsocket?port=5903&vncticket=PVEVNC%3A66D680E5%3A%3AgmMgUw0fGNkbsbT6xERshETfMZ%2BosdU%2FXAc1kqK%2BnOOoprY6IXBL%2FWeKvtIIEGqxWnJDvxKkOFdCeVjldTaMjvK%2Bu5yVEMyoGxYxELjLNaN41Kg%2FLMNXDJMVtUUTvb9Qlj1Rt5yO2E4uNOe%2F3CNAYF0YxQ%2B%2Ba%2F9AbT1GFqqOFGGxT7VuoHImoGbAZsuwpyqfLjWewczx3eGQvcLvaIPSfakH%2B2Bb%2BzMtUTQQR6TuxhCMM8DE5E5TeA0YzYVg76Ddo3OZ%2BVVIHmILh4BI9wq%2BvuElVHCwNvfUfqNGDpi3DGebTO1qQHZEO%2FD2xvO94Jz%2Bu4RLeSdX4T6N4G6IqmNX1A%3D%3D HTTP/1.1" 502 341

NGINX:
nginx Error:
==> /var/log/nginx/error.log <==
2024/09/03 13:48:35 [error] 3725891#3725891: *7 upstream prematurely closed connection while reading response header from upstream, client: 10.0.0.103, server: www.industrial-sandbox.com, request: "GET /api2/json/nodes/www/lxc/113/vncwebsocket?port=5902&vncticket=PVEVNC%3A66D6A318%3A%3AkKRGtCWWwd1Fo8ULsazMVmK2DXyoJevWC5Ye6Fk46LQpI9AaxZplTVBppMfHw%2BHAjxvrQ9xpCxMFF%2FbxzJTQJ2jOLXYkDkjIN3yEZj8PNXNOKumeVv7rd46zabhcZ%2Be%2FzF%2FqXrve6oj4A%2BV1qfAHaw%2FzzmBMjm9c7WatqCz878iqJF3zxcE%2FBm04Rht1d4z4prIfNyjwso1iqZW2jb6ha0jyk6KVHTGY9mTFIIOL75bU3lvcj3LY%2FA%2BKvP3dCLKD0FZWjSeJrarHFhZJDf%2Fc%2BTpia65Vd%2Brm12euLKOykXvxplfzvyWuyrc5foflLbL4H3zYtNzTeGwPh62qLZI%2FYA%3D%3D HTTP/1.1", upstream: "https://10.0.0.1:8006/api2/json/nodes/www/lxc/113/vncwebsocket?port=5902&vncticket=PVEVNC%3A66D6A318%3A%3AkKRGtCWWwd1Fo8ULsazMVmK2DXyoJevWC5Ye6Fk46LQpI9AaxZplTVBppMfHw%2BHAjxvrQ9xpCxMFF%2FbxzJTQJ2jOLXYkDkjIN3yEZj8PNXNOKumeVv7rd46zabhcZ%2Be%2FzF%2FqXrve6oj4A%2BV1qfAHaw%2FzzmBMjm9c7WatqCz878iqJF3zxcE%2FBm04Rht1d4z4prIfNyjwso1iqZW2jb6ha0jyk6KVHTGY9mTFIIOL75bU3lvcj3LY%2FA%2BKvP3dCLKD0FZWjSeJrarHFhZJDf%2Fc%2BTpia65Vd%2Brm12euLKOykXvxplfzvyWuyrc5foflLbL4H3zYtNzTeGwPh62qLZI%2FYA%3D%3D", host: "10.0.0.103:4443"

==> /var/log/nginx/access.log <==
10.0.0.103 - - [03/Sep/2024:13:48:35 +0800] "GET /api2/json/nodes/www/lxc/113/vncwebsocket?port=5902&vncticket=PVEVNC%3A66D6A318%3A%3AkKRGtCWWwd1Fo8ULsazMVmK2DXyoJevWC5Ye6Fk46LQpI9AaxZplTVBppMfHw%2BHAjxvrQ9xpCxMFF%2FbxzJTQJ2jOLXYkDkjIN3yEZj8PNXNOKumeVv7rd46zabhcZ%2Be%2FzF%2FqXrve6oj4A%2BV1qfAHaw%2FzzmBMjm9c7WatqCz878iqJF3zxcE%2FBm04Rht1d4z4prIfNyjwso1iqZW2jb6ha0jyk6KVHTGY9mTFIIOL75bU3lvcj3LY%2FA%2BKvP3dCLKD0FZWjSeJrarHFhZJDf%2Fc%2BTpia65Vd%2Brm12euLKOykXvxplfzvyWuyrc5foflLbL4H3zYtNzTeGwPh62qLZI%2FYA%3D%3D HTTP/1.1" 502 150 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" "-"
 
Last edited: