PXVE, Nginx & Websocket proxy

hades666

New Member
Jan 30, 2015
5
0
1
Hi All,

Simply Put, We have a PX Cluster of 20+ Servers. I am trying setup a Reverse Proxy to Manage 1 common URL Loadbalancing PVE Managers in the Backend.

This VM is running RHEL6 with NGINX on the actual PX Cluster in HA Mode.

Site is loading fine and all standard actions work. What doesn't is the NoVNC console.

What am I missing?

px_error.jpg
Versions
- nginx.x86_64 1.6.2-1.el6.ngx
- PVE 3.3-1

Nginx Config
upstream proxmox {
server 10.0.0.161:8006;
server 10.0.0.162:8006;
server 10.0.0.163:8006;
server 10.0.0.164:8006;
...
}

server {
listen 80 default_server;
server_name domain.company.com;
rewrite ^(.*) https://domain.company.com permanent;
}
server {
listen 443;
server_name domain.company.com;
ssl on;
ssl_certificate /etc/nginx/ssl/SSL-cert-domain.company.com.pem;
ssl_certificate_key /etc/nginx/ssl/SSL-domain.company.com.key;
proxy_redirect off;
location / {
proxy_pass https://proxmox;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
 
Anyone? Really, no one has tried this or do I need to explain it better? Just trying to proxy pvemanager via proxy and access noVNC.
 
Anyone? Really, no one has tried this or do I need to explain it better? Just trying to proxy pvemanager via proxy and access noVNC.

This works for me:
in the http section:
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

In the server scection:
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_pass https://proxmox:8006/;
}
 
Hi Vrang, any chance you could share you entire nginx config if you got this working.?
 
Hi Vrang, any chance you could share you entire nginx config if you got this working.?

What Vrang posted works for me, here is my entire config (/etc/nginx/sites-enabled/default):

Code:
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}


server {
        listen 443 ssl;
        ssl on;
        ssl_certificate /etc/ssl/nginx/bundle.crt;
        ssl_certificate_key /etc/ssl/nginx/ssl.key;


        server_name nginx.domain;


        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;
                proxy_pass https://x.x.x.x:8006;
        }
}
 
No luck.

This is a simple RHEL VM with NGINX install to act as a the proxy.

If I try your config, the only change is the proxy_pass info. In this case, the host server has no listening port on 8006 so it fails.

Starting nginx: nginx: [warn] upstream "proxmox" may not have port 8006 in /etc/nginx/conf.d/proxmox-gui.conf:39 [FAILED]



upstream proxmox {
server 10.0.0.4:8006;
server 10.0.0.5:8006;
server 10.0.0.6:8006;
server 10.0.0.7:8006;
server 10.0.0.8:8006;
}

map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen 80 default_server;
server_name server.domain.com;
rewrite ^(.*) https://server.domain.com permanent;
}
server {
listen 443;
server_name server.domain.com;
ssl on;
ssl_certificate /etc/nginx/ssl/SSL.pem;
ssl_certificate_key /etc/nginx/ssl/SSL.key;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_pass https://proxmox;
}
}

Any other ideas?
 
No luck.
Starting nginx: nginx: [warn] upstream "proxmox" may not have port 8006 in /etc/nginx/conf.d/proxmox-gui.conf:39 [FAILED]

Any other ideas?

Did you ever get it working? I'm having the same problem here.
 
Install haproxy on a VM and use this config file as /etc/haproxy/haproxy.cfg

Example below is running and working on Debian Jessie.

Code:
global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3
    tune.ssl.default-dh-param 2048

defaults
    log    global
    mode    tcp
    option    tcplog
    option    dontlognull
    option    redispatch
    option    contstats
        timeout connect 5000
        timeout client  50000
        timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http
#    option forwardfor
#    option http-server-close

frontend http_front
    bind *:443 ssl crt /path/to/ssl.pem
    reqadd X-Forwarded-Proto:\ https
    default_backend http_back

backend http_back
    redirect scheme https if !{ ssl_fc }
    balance roundrobin
    server pve1 IP_1:8006 check ssl verify none
    server pve2 IP_2:8006 check ssl verify none
    server pve3 IP_3:8006 check ssl verify none

listen proxmox_spice
    bind *:3128
    option tcpka
    balance roundrobin
        server pve1 IP_1:3128 check
        server pve2 IP_2:3128 check
        server pve3 IP_3:3128 check

listen stats
    bind *:8080
    log global
    mode http
    stats enable
    stats refresh 30s
    stats hide-version
    stats show-node
    stats uri /
    stats admin if TRUE
 
  • Like
Reactions: morph027
The same thing happens to me. I'm using proxmox behind nginx reverse proxy and the nginx is behind a Cisco ASA. Everything goes well exception to noVNC console that freezes in seconds of use.
 
I'm not sure this will ever work.
I've tried the haproxy + SPICE but it randomly disconnects.
With noVNC it's even worse. Sometimes it won't connect at all. Sometimes it disconnects.
Nginx + noVNC the same. I didn't get managed to run Nginx + SPICE but i think it will be the same again.
I think it's obvious why this is happening.

This whole solution needs some workaround.
Has anyone solved this?
 
Confirming that vrang->TJN combined with op hades666 solution now works on Proxmox 6.1-11 with proxied over nginx 1.17.7 on openwrt 19.07.2

If you are using multiple backend solution as hades666 is proposing then either weight=xxx should be added to nodes or evenbetter ip_hash; parameter so that sessions are stickied. Otherwise vnc will still be broken the same way as before because nginx is bouncing around websocket connections between nodes and this does not play ball.

For simplicity dumping the final code once again:

NGINX:
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

upstream proxmox {
    ip_hash;
    server 10.15.0.5:8006;
    server 10.15.0.6:8006;
    server 10.15.0.7:8006;
    server 10.15.0.8:8006;
}

server {
    server_name proxmox.mydomain.com;
    access_log /var/log/nginx/proxmox.mydomain.com.log;
    error_log /var/log/nginx/proxmox.mydomain.com_err.log;

    listen 8006 ssl;
    ssl_certificate      /etc/acme/proxmox.mydomain/fullchain.cer;
    ssl_certificate_key  /etc/acme/proxmox.mydomain/proxmox.mydomain.key;

    location / {
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_pass https://proxmox;
    }  
}
 
Last edited:
Confirming that vrang->TJN combined with op hades666 solution now works on Proxmox 6.1-11 with proxied over nginx 1.17.7 on openwrt 19.07.2

How about SPICE proxy is it working for you? I have tried to proxy it over nginx but without success:
NGINX:
upstream spice {ip_hash; server n1:3128; server n2:3128; server n3:3128; }
server {
  listen <server_IP>:3128;
  include mime.types;
  server_name <FQDN>;
  location / {
    proxy_pass https://spice;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Forwarded-Port 443;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Origin http://$host;
 
Last edited:
I would assume that you have a problem that there is a hash for 8006 port and hash for 3128 , but they work independently, so having a sticky on 8006 port on one server could give you spice sticky to another. Have you tried having just one ip in upstream for 3128 and 8006 ports to see if it works?
 
I would assume that you have a problem that there is a hash for 8006 port and hash for 3128 , but they work independently, so having a sticky on 8006 port on one server could give you spice sticky to another. Have you tried having just one ip in upstream for 3128 and 8006 ports to see if it works?
This doesn't seem to be the case. I've set proxy_pass to single host. The 8006 works okay but SPICE proxy on 3128 doesn not:
NGINX:
server {
  listen <server_IP>:3128;
  include mime.types;
  server_name <FQDN>;
  location / {
    proxy_pass https://n1:3128;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Forwarded-Port 443;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Origin http://$host;
 
i was just faced with the same problem,

you just have to set up a stream in nginx (nginx.conf).



JSON:
stream {

       upstream spice {
          server server1:3128;
          server server2:3128;
          server server3:3128;
        }


server {

listen 3128;
proxy_pass spice;
}

}
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!