NGINX reverse-proxy for SPICE

number5 · May 12, 2021

I have NGINX working as a reverse-proxy for the web-based GUI and for VNC, but I am having trouble getting SPICE consoles to work.

Internally, I have a reverse-proxy set up directly on the PVE host and it's fine: SPICE works and so does the web GUI. For this, however, I did not have to set up a separate stream, and SPICE worked right away. Externally, I have a similar setup, but SPICE does not work. Below is my NGINX config for the external server:

Code:

upstream proxmox {
    server server1:8006;
}

upstream spice {

server server1:3128;

}

server {
    listen 80;

    server_name my.domain.com;
    return 301 https://$host$request_uri;

}

server {
    listen 443 ssl http2;
    server_name my.domain.com;
    ssl on;
    ssl_certificate /etc/letsencrypt/live/my.domain.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/my.domain.com/privkey.pem; # managed by Certbot
    proxy_redirect off;

    location / {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass https://proxmox;
        proxy_buffering off;
        client_max_body_size 0;
        proxy_connect_timeout  3600s;
        proxy_read_timeout  3600s;
        proxy_send_timeout  3600s;
        send_timeout  3600s;
    }


}

server {
    listen 3128;
    server_name my.domain.com;

    location / {
        proxy_pass https://spice;
        proxy_buffering off;
        client_max_body_size 0;
        proxy_connect_timeout  3600s;
        proxy_read_timeout  3600s;
        proxy_send_timeout  3600s;
        send_timeout  3600s;
    }

}

on the external host, I do see a connection attempt in the NGINX access.log, which I do not see in the internal NGINX host. This is what the line looks like:

Code:

"CONNECT pvespiceproxy:609c40d6:212:server1::0feeb24e84a57ba7418452bb4233f729c6319012:61033 HTTP/1.0" 400 182 "-" "-"

The connection is dropped immediately and virt-viewer states "Unable to connect to the graphic server filename.vv"

Any help with figuring out where the issue is would be appreciated. Thank you.

BuzzKillingtonne · Dec 19, 2021

Here is the solution:

I went down a rabbit hole for this....
Your above configuration is running in layer 7 (application layer) and we need to run in layer 4 (transport layer)

I'm using debian, so some things may vary with things like sites-enabled/sites-available package names etc, but the meat and potatoes are the same:

Make sure that the stream module is installed. On Debian the package is libnginx-mod-stream, i think CentOS it is nginx-mod-stream.
Debian:$ apt list --installed libnginx-mod-stream
Make sure the module is under /usr/lib/nginx/modules/ with the name ngx_stream_module.so
Make sure you have the http stream nginx module in /etc/nginx/modules-enabled/50-mod-stream.conf
Make sure that /etc/nginx/modules-enabled/50-mod-stream.conf contains "load_module modules/ngx_stream_module.so;" without the quotes.
You may have seen people ask you to run nginx -V to see if it is loaded, mine did not show here, I believe it is loaded by default now, but i'm not sure.

In your /etc/nginx/nginx.conf you need to add

Code:

stream {
    include /etc/nginx/sites-available/proxmox-spice.conf;
}

This must be in the top level on that file and cannot be under the http block. so it must look like this:

Code:

http {
    All the stuff in here (mine has "include /etc/nginx/sites-enabled/*;")
}
   
stream {
   include /etc/nginx/sites-available/proxmox-spice.conf;
}

Next you must create a new config file in the location of /etc/nginx/sites-available/proxmox-spice.conf put the location where ever you like, just make sure to adjust the include directive under stream in the first step.

The config file is simple and looks like this:

Code:

server {
    listen    3128;
    proxy_pass X.X.X.X:3128;
}

Make sure to replace X.X.X.X with your server's IP. and remember that this is a layer 4 rule, so you do not add the usual stuff, we are just routing the port 3128 tcp to the back end server. Notice that I did not put http or https, that is again becasue this is layer 4, http and https are layer 7. only put the server IP and port number.

Make sure to run nginx -t and fix any syntax errors before reloading nginx with systemctl reload nginx.

micky1067 · Mar 17, 2022

Hello...

I have do it like your tutorial. But I can not connect with spice from outside.
Maybe you have a solution for me ?

my spice.conf in /etc/nginx/sites-available/spice.conf


server {
    listen    3128;
    proxy_pass 192.168.6.3:3128;
}

my nginx.conf


user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections 768;
    # multi_accept on;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    client_max_body_size 1G;
    # server_tokens off;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # SSL Settings
    ##

    ssl_protocols TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
    ssl_prefer_server_ciphers on;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;

    # gzip_vary on;
    # gzip_proxied any;
    # gzip_comp_level 6;
    # gzip_buffers 16 8k;
    # gzip_http_version 1.1;
    # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

stream {
    include /etc/nginx/sites-available/spice.conf;
}

And here my proxmox nginx conf in sites-available


map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

server {
  listen 80;
  server_name proxmox.my.domain;
  return 301 https://$host$request_uri;
}

# SSL configuration
server {
  listen 443 ssl;
  server_name proxmox.my.domain;
  ssl_certificate      /etc/letsencrypt/live/proxmox.my.domain/fullchain.pem;
  ssl_certificate_key  /etc/letsencrypt/live/proxmox.my.domain/privkey.pem;

  # Improve HTTPS performance with session resumption
  ssl_session_cache shared:SSL:10m;
  ssl_session_timeout 5m;

  # Enable server-side protection against BEAST attacks
  ssl_prefer_server_ciphers on;
  ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5;

  # Disable SSLv3
  #ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_protocols TLSv1.2 TLSv1.3;

  # Diffie-Hellman parameter for DHE ciphersuites
  ssl_dhparam /etc/ssl/certs/dhparam.pem;

  # Enable HSTS (https://developer.mozilla.org/en-US/docs/Security/HTTP_Strict_Transport_Security)
  add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";

  # Enable OCSP stapling (http://blog.mozilla.org/security/2013/07/29/ocsp-stapling-in-firefox)
  ssl_stapling on;
  ssl_stapling_verify on;
  ssl_trusted_certificate /etc/letsencrypt/live/proxmox.my.domain/fullchain.pem;
  resolver 8.8.8.8 8.8.4.4 valid=300s;
  resolver_timeout 5s;

  proxy_redirect off;

  location / {
    proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade"; 
        proxy_pass https://192.168.6.3:8006;
       proxy_buffering off;
       client_max_body_size 0;
       proxy_connect_timeout  3600s;
        proxy_read_timeout  3600s;
        proxy_send_timeout  3600s;
        send_timeout  3600s;
  }

BuzzKillingtonne · Mar 17, 2022

micky1067 said:
Hello...

I have do it like your tutorial. But I can not connect with spice from outside.
Maybe you have a solution for me ?

my spice.conf in /etc/nginx/sites-available/spice.conf

Code:

server { listen 3128; proxy_pass 192.168.6.3:3128; }

my nginx.conf

Code:

stream { include /etc/nginx/sites-available/spice.conf; }

That all looks correct to me, I don't see any config issues. If you're trying to get it to work outside of your network, have you made sure to port forward TCP port 3128 to your nginx reverse proxy server?

Your proxmox config file in Nginx will be irrelevant to getting spice to pass through the reverse proxy as the reverse proxy config for the proxmox webUI is a Layer 7 proxy and the spice proxy is a Layer 4 proxy. Basically we are just passing the port number and all of it's traffic through to proxmox.

if you try connecting to the spice console without a firewall between you and the nginx server (such as on a local LAN and using split scope DNS to point your DNS address to your reverse proxy server) does it work or does it fail? to me this looks like its probably a firewall issue.

Something to also be sure of is if there is anything blocked 3128 outbound from the location you are tryting to connect from. I ask this becasue my firewall at my office blocks 3128 outbound as it is the default port used by squid proxy.

I use OPNsense as my firewall and had to create a NAT port forward for port 3128 pointing to my Nginx reverse proxy server for this to work externally.

micky1067 · Mar 17, 2022

Hello...

You are right. That was my fault. Need Forward to my Proxy Server Not to my proxmox Server. Now it runs perfekt.
Thank you so much....

Greetings
Michael

thehitmonkey24 · Jan 29, 2024

So in this case is NPM running on debian directly or via docker?

micky1067 · Jan 29, 2024

I use nginx direct. Not NPM. I have not testet NPM till now.

Search

Search

NGINX reverse-proxy for SPICE

number5

New Member

BuzzKillingtonne

Member

micky1067

Member

BuzzKillingtonne

Member

micky1067

Member

thehitmonkey24

New Member

micky1067

Member