Getting a '502 - Bad gateway' when proxying the vncwebsocket.

michael-h

New Member
Apr 18, 2023
1
1
1
Hello,

I am currently using Nginx to embed a console onto an external webpage. To accomplish this, I am proxying the content to the Proxmox host. Please refer to the Nginx configuration below:

Nginx configuration:
Code:
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    return 301 https://example.com$request_uri;
}
server {
    listen 443 ssl;
    listen [::]:443 ssl;
    ssl_certificate /etc/ssl/certs/example.com.pem;
    ssl_certificate_key /etc/ssl/private/example.com.key;
    ssl_session_timeout 1d;
    root /var/www/example.com;
    index index.html;
     location /console/ {
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Host proxmox-host.com;
        proxy_pass https://proxmox-host.com/;
    }

    location /api2/json {
        proxy_pass https://proxmox-host.com/api2/json;
        proxy_http_version 1.1;
        proxy_set_header Host proxmox-host.com;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Authorization  "PVEAPIToken=TOKEN";    
    }
    location /novnc/ {
        proxy_pass https://proxmox-host.com/novnc/;
    }
}


The problem I am experiencing:
With the above Nginx configuration, I can now access the console via an external web page. All calls made by the novnc client work. However, as soon as I make the call to establish the websocket connection, I usually get a 502 bad gateway error after 6 seconds. Occasionally, it is possible to connect to the console, and then the connection is established after 2 seconds. But this occurs at most 2 times out of 10 attempts. Most of the time, the websocket connection is simply not established.

Scherm­afbeelding 2023-04-28 om 11.03.00.png


What I have already tried:
  • Tried setting different headers, such as keep-alive, in the proxy_set_header.
  • Checked if Nginx sets a timeout after 6 seconds, but this doesn't seem to be the case.
  • Checked the Nginx logs, which indicate that the handshake has failed.


My specific question:
Does anyone know why the websocket connection is sometimes established but mostly not? And why it times out after 6 seconds when it fails?
 
  • Like
Reactions: Kolovatoff1
@yswery, personally I didn't figure this out myself, but if I remember correctly, it had something to do with routing within Proxmox. The issue was related to IPv4 and IPv6, but I'm not exactly sure. It was a problem with correctly resolving the address or something similar. Hopefully, this information can help you make some progress.
 
May I ask how it was ultimately resolved? I also encountered this problem.
Sometimes it can work for lxc vncwebsocket, but more often it raise 502.
I always get 502 for vm vncwebsocket.:(

I also tried to replace nginx with apache, but the problem still persists.

PVE 8.2.4, nginx 1.26, apache 2.4.62

Code:
apache Error:
10.0.0.103 - - [03/Sep/2024:03:21:40 +0000] "GET /api2/json/nodes/www/qemu/101/vncwebsocket?port=5901&vncticket=PVEVNC%3A66D680A2%3A%3AbbmHwo5wF7yPtheqYHupnIXip9KmhO428MsTdABqPcD9g79t4qHKoFlv%2BbAc0fANQ5Nz4CFIrbzVJwO1lkMhUpnaRSVMDDpjAXy81uWWdxpC8DswOP6yIjypyMu1r6hiOKG%2Fb%2Bd5vZ4TuJWgN%2BKxiC%2Ffjmv5MHrIJUX%2FpdT14i4%2B7q8rYHHVTpyeCC3DyWazzMErb5d1J2geiSWc%2B%2Bq3A%2FqU0NZvNjUvkYaObdFQKt%2F4Xe5izA90rUIQYRwsZcirmtdstLUx71bPZ5woAd7gDRqS3O%2Fu7%2F9i56DjeLK3HEnXVqgzZzp8Fq1EKkYVvwSGA%2FyxXqCdNRGX3ouZqhGbbw%3D%3D HTTP/1.1" 502 341
[Tue Sep 03 03:22:48.312491 2024] [proxy_http:error] [pid 41:tid 41] (70014)End of file found: [client 10.0.0.103:35760] AH01102: error reading status line from remote server 10.0.0.1:8006
[Tue Sep 03 03:22:48.312519 2024] [proxy:error] [pid 41:tid 41] [client 10.0.0.103:35760] AH00898: Error reading from remote server returned by /api2/json/nodes/www/qemu/101/vncwebsocket
10.0.0.103 - - [03/Sep/2024:03:22:43 +0000] "GET /api2/json/nodes/www/qemu/101/vncwebsocket?port=5903&vncticket=PVEVNC%3A66D680E5%3A%3AgmMgUw0fGNkbsbT6xERshETfMZ%2BosdU%2FXAc1kqK%2BnOOoprY6IXBL%2FWeKvtIIEGqxWnJDvxKkOFdCeVjldTaMjvK%2Bu5yVEMyoGxYxELjLNaN41Kg%2FLMNXDJMVtUUTvb9Qlj1Rt5yO2E4uNOe%2F3CNAYF0YxQ%2B%2Ba%2F9AbT1GFqqOFGGxT7VuoHImoGbAZsuwpyqfLjWewczx3eGQvcLvaIPSfakH%2B2Bb%2BzMtUTQQR6TuxhCMM8DE5E5TeA0YzYVg76Ddo3OZ%2BVVIHmILh4BI9wq%2BvuElVHCwNvfUfqNGDpi3DGebTO1qQHZEO%2FD2xvO94Jz%2Bu4RLeSdX4T6N4G6IqmNX1A%3D%3D HTTP/1.1" 502 341

NGINX:
nginx Error:
==> /var/log/nginx/error.log <==
2024/09/03 13:48:35 [error] 3725891#3725891: *7 upstream prematurely closed connection while reading response header from upstream, client: 10.0.0.103, server: www.industrial-sandbox.com, request: "GET /api2/json/nodes/www/lxc/113/vncwebsocket?port=5902&vncticket=PVEVNC%3A66D6A318%3A%3AkKRGtCWWwd1Fo8ULsazMVmK2DXyoJevWC5Ye6Fk46LQpI9AaxZplTVBppMfHw%2BHAjxvrQ9xpCxMFF%2FbxzJTQJ2jOLXYkDkjIN3yEZj8PNXNOKumeVv7rd46zabhcZ%2Be%2FzF%2FqXrve6oj4A%2BV1qfAHaw%2FzzmBMjm9c7WatqCz878iqJF3zxcE%2FBm04Rht1d4z4prIfNyjwso1iqZW2jb6ha0jyk6KVHTGY9mTFIIOL75bU3lvcj3LY%2FA%2BKvP3dCLKD0FZWjSeJrarHFhZJDf%2Fc%2BTpia65Vd%2Brm12euLKOykXvxplfzvyWuyrc5foflLbL4H3zYtNzTeGwPh62qLZI%2FYA%3D%3D HTTP/1.1", upstream: "https://10.0.0.1:8006/api2/json/nodes/www/lxc/113/vncwebsocket?port=5902&vncticket=PVEVNC%3A66D6A318%3A%3AkKRGtCWWwd1Fo8ULsazMVmK2DXyoJevWC5Ye6Fk46LQpI9AaxZplTVBppMfHw%2BHAjxvrQ9xpCxMFF%2FbxzJTQJ2jOLXYkDkjIN3yEZj8PNXNOKumeVv7rd46zabhcZ%2Be%2FzF%2FqXrve6oj4A%2BV1qfAHaw%2FzzmBMjm9c7WatqCz878iqJF3zxcE%2FBm04Rht1d4z4prIfNyjwso1iqZW2jb6ha0jyk6KVHTGY9mTFIIOL75bU3lvcj3LY%2FA%2BKvP3dCLKD0FZWjSeJrarHFhZJDf%2Fc%2BTpia65Vd%2Brm12euLKOykXvxplfzvyWuyrc5foflLbL4H3zYtNzTeGwPh62qLZI%2FYA%3D%3D", host: "10.0.0.103:4443"

==> /var/log/nginx/access.log <==
10.0.0.103 - - [03/Sep/2024:13:48:35 +0800] "GET /api2/json/nodes/www/lxc/113/vncwebsocket?port=5902&vncticket=PVEVNC%3A66D6A318%3A%3AkKRGtCWWwd1Fo8ULsazMVmK2DXyoJevWC5Ye6Fk46LQpI9AaxZplTVBppMfHw%2BHAjxvrQ9xpCxMFF%2FbxzJTQJ2jOLXYkDkjIN3yEZj8PNXNOKumeVv7rd46zabhcZ%2Be%2FzF%2FqXrve6oj4A%2BV1qfAHaw%2FzzmBMjm9c7WatqCz878iqJF3zxcE%2FBm04Rht1d4z4prIfNyjwso1iqZW2jb6ha0jyk6KVHTGY9mTFIIOL75bU3lvcj3LY%2FA%2BKvP3dCLKD0FZWjSeJrarHFhZJDf%2Fc%2BTpia65Vd%2Brm12euLKOykXvxplfzvyWuyrc5foflLbL4H3zYtNzTeGwPh62qLZI%2FYA%3D%3D HTTP/1.1" 502 150 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" "-"
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!