PXVE, Nginx & Websocket proxy

hades666 · Jan 30, 2015

Hi All,

Simply Put, We have a PX Cluster of 20+ Servers. I am trying setup a Reverse Proxy to Manage 1 common URL Loadbalancing PVE Managers in the Backend.

This VM is running RHEL6 with NGINX on the actual PX Cluster in HA Mode.

Site is loading fine and all standard actions work. What doesn't is the NoVNC console.

What am I missing?

Versions
- nginx.x86_64 1.6.2-1.el6.ngx
- PVE 3.3-1

Nginx Config
upstream proxmox {
server 10.0.0.161:8006;
server 10.0.0.162:8006;
server 10.0.0.163:8006;
server 10.0.0.164:8006;
...
}

server {
listen 80 default_server;
server_name domain.company.com;
rewrite ^(.*) https://domain.company.com permanent;
}
server {
listen 443;
server_name domain.company.com;
ssl on;
ssl_certificate /etc/nginx/ssl/SSL-cert-domain.company.com.pem;
ssl_certificate_key /etc/nginx/ssl/SSL-domain.company.com.key;
proxy_redirect off;
location / {
proxy_pass https://proxmox;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}

hades666 · Feb 1, 2015

Anyone? Really, no one has tried this or do I need to explain it better? Just trying to proxy pvemanager via proxy and access noVNC.

vrang · Feb 3, 2015

hades666 said:
Anyone? Really, no one has tried this or do I need to explain it better? Just trying to proxy pvemanager via proxy and access noVNC.

This works for me:
in the http section:
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
In the server scection:
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_pass https://proxmox:8006/;
}

hades666 · Feb 4, 2015

hm.. not working for me. what is your full .conf should you don't mind sharing.

hades666 · Feb 5, 2015

Hi Vrang, any chance you could share you entire nginx config if you got this working.?

TJN · Feb 17, 2015

hades666 said:
Hi Vrang, any chance you could share you entire nginx config if you got this working.?

What Vrang posted works for me, here is my entire config (/etc/nginx/sites-enabled/default):

Code:

map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}


server {
        listen 443 ssl;
        ssl on;
        ssl_certificate /etc/ssl/nginx/bundle.crt;
        ssl_certificate_key /etc/ssl/nginx/ssl.key;


        server_name nginx.domain;


        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;
                proxy_pass https://x.x.x.x:8006;
        }
}

hades666 · Feb 18, 2015

No luck.

This is a simple RHEL VM with NGINX install to act as a the proxy.

If I try your config, the only change is the proxy_pass info. In this case, the host server has no listening port on 8006 so it fails.

Starting nginx: nginx: [warn] upstream "proxmox" may not have port 8006 in /etc/nginx/conf.d/proxmox-gui.conf:39 [FAILED]

upstream proxmox {
server 10.0.0.4:8006;
server 10.0.0.5:8006;
server 10.0.0.6:8006;
server 10.0.0.7:8006;
server 10.0.0.8:8006;
}

map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen 80 default_server;
server_name server.domain.com;
rewrite ^(.*) https://server.domain.com permanent;
}
server {
listen 443;
server_name server.domain.com;
ssl on;
ssl_certificate /etc/nginx/ssl/SSL.pem;
ssl_certificate_key /etc/nginx/ssl/SSL.key;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_pass https://proxmox;
}
}

Any other ideas?

ffrank · Jun 5, 2016

hades666 said:
No luck.
Starting nginx: nginx: [warn] upstream "proxmox" may not have port 8006 in /etc/nginx/conf.d/proxmox-gui.conf:39 [FAILED]

Any other ideas?

Did you ever get it working? I'm having the same problem here.

mir · Jun 5, 2016

Install haproxy on a VM and use this config file as /etc/haproxy/haproxy.cfg

Example below is running and working on Debian Jessie.

Code:

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3
    tune.ssl.default-dh-param 2048

defaults
    log    global
    mode    tcp
    option    tcplog
    option    dontlognull
    option    redispatch
    option    contstats
        timeout connect 5000
        timeout client  50000
        timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http
#    option forwardfor
#    option http-server-close

frontend http_front
    bind *:443 ssl crt /path/to/ssl.pem
    reqadd X-Forwarded-Proto:\ https
    default_backend http_back

backend http_back
    redirect scheme https if !{ ssl_fc }
    balance roundrobin
    server pve1 IP_1:8006 check ssl verify none
    server pve2 IP_2:8006 check ssl verify none
    server pve3 IP_3:8006 check ssl verify none

listen proxmox_spice
    bind *:3128
    option tcpka
    balance roundrobin
        server pve1 IP_1:3128 check
        server pve2 IP_2:3128 check
        server pve3 IP_3:3128 check

listen stats
    bind *:8080
    log global
    mode http
    stats enable
    stats refresh 30s
    stats hide-version
    stats show-node
    stats uri /
    stats admin if TRUE

Edson Junior · Sep 29, 2016

The same thing happens to me. I'm using proxmox behind nginx reverse proxy and the nginx is behind a Cisco ASA. Everything goes well exception to noVNC console that freezes in seconds of use.

davidisko · Dec 23, 2016

I'm not sure this will ever work.
I've tried the haproxy + SPICE but it randomly disconnects.
With noVNC it's even worse. Sometimes it won't connect at all. Sometimes it disconnects.
Nginx + noVNC the same. I didn't get managed to run Nginx + SPICE but i think it will be the same again.
I think it's obvious why this is happening.

This whole solution needs some workaround.
Has anyone solved this?

Jannoke · Oct 29, 2019

Is this still broken in v6 version of proxmox?

Jannoke · Jun 10, 2020

Confirming that vrang->TJN combined with op hades666 solution now works on Proxmox 6.1-11 with proxied over nginx 1.17.7 on openwrt 19.07.2

If you are using multiple backend solution as hades666 is proposing then either weight=xxx should be added to nodes or evenbetter ip_hash; parameter so that sessions are stickied. Otherwise vnc will still be broken the same way as before because nginx is bouncing around websocket connections between nodes and this does not play ball.

For simplicity dumping the final code once again:

NGINX:

map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

upstream proxmox {
    ip_hash;
    server 10.15.0.5:8006;
    server 10.15.0.6:8006;
    server 10.15.0.7:8006;
    server 10.15.0.8:8006;
}

server {
    server_name proxmox.mydomain.com;
    access_log /var/log/nginx/proxmox.mydomain.com.log;
    error_log /var/log/nginx/proxmox.mydomain.com_err.log;

    listen 8006 ssl;
    ssl_certificate      /etc/acme/proxmox.mydomain/fullchain.cer;
    ssl_certificate_key  /etc/acme/proxmox.mydomain/proxmox.mydomain.key;

    location / {
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_pass https://proxmox;
    }  
}

aasami · Sep 24, 2020

Jannoke said:
Confirming that vrang->TJN combined with op hades666 solution now works on Proxmox 6.1-11 with proxied over nginx 1.17.7 on openwrt 19.07.2

How about SPICE proxy is it working for you? I have tried to proxy it over nginx but without success:

NGINX:

upstream spice {ip_hash; server n1:3128; server n2:3128; server n3:3128; }
server {
  listen <server_IP>:3128;
  include mime.types;
  server_name <FQDN>;
  location / {
    proxy_pass https://spice;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Forwarded-Port 443;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Origin http://$host;

Jannoke · Sep 24, 2020

I would assume that you have a problem that there is a hash for 8006 port and hash for 3128 , but they work independently, so having a sticky on 8006 port on one server could give you spice sticky to another. Have you tried having just one ip in upstream for 3128 and 8006 ports to see if it works?

aasami · Sep 25, 2020

Jannoke said:
I would assume that you have a problem that there is a hash for 8006 port and hash for 3128 , but they work independently, so having a sticky on 8006 port on one server could give you spice sticky to another. Have you tried having just one ip in upstream for 3128 and 8006 ports to see if it works?

This doesn't seem to be the case. I've set proxy_pass to single host. The 8006 works okay but SPICE proxy on 3128 doesn not:

NGINX:

server {
  listen <server_IP>:3128;
  include mime.types;
  server_name <FQDN>;
  location / {
    proxy_pass https://n1:3128;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Forwarded-Port 443;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Origin http://$host;

badsmoke · Jan 22, 2021

i was just faced with the same problem,

you just have to set up a stream in nginx (nginx.conf).

JSON:

stream {

       upstream spice {
          server server1:3128;
          server server2:3128;
          server server3:3128;
        }


server {

listen 3128;
proxy_pass spice;
}

}

Search

Search

PXVE, Nginx & Websocket proxy

hades666

New Member

hades666

New Member

vrang

New Member

hades666

New Member

hades666

New Member

TJN

New Member

hades666

New Member

ffrank

Renowned Member

mir

Famous Member

Edson Junior

New Member

davidisko

New Member

Jannoke

Renowned Member

Jannoke

Renowned Member

aasami

Renowned Member

Jannoke

Renowned Member

aasami

Renowned Member

badsmoke

Member