[SOLVED] Upgraded and domain SSL not working

[robetus]

I just upgraded to the latest version of proxmox 6 (6.4-13) and my SSL domain is no longer working on nginx. I've double checked everything included the A record propagation and let's encrypt certs all valid. I can't seem to find the issue after the upgrade. Is there something that changed with 6.4-13 that I can check? Nginx is running with no errors.

 

[oguz]

hi,

my SSL domain is no longer working on nginx.

what is the exact issue you're experiencing?
can you tell us about your setup?

 

[robetus]

I followed the directions exactly as stated here: https://pve.proxmox.com/wiki/Web_Interface_Via_Nginx_Proxy

I doubled checked to make sure nginx had no errors running or in the configuration and it does not. I'm currently getting a bad gateway error when I visit the domain.

I can also verify the Let's Encrypt Certs are valid in the GUI which I can access by IP address only.

The directions here https://pve.proxmox.com/wiki/Web_Interface_Via_Nginx_Proxy should allow me to visit the GUI without the need for the port :8006 correct. I know that's how a nginx proxy works but I want to confirm that is the way with proxmox.

 

[oguz]

I followed the directions exactly as stated here: https://pve.proxmox.com/wiki/Web_Interface_Via_Nginx_Proxy

okay

I doubled checked to make sure nginx had no errors running or in the configuration and it does not. I'm currently getting a bad gateway error when I visit the domain.

can you post the nginx configuration here? (please use [code] tags when pasting it)

The directions here https://pve.proxmox.com/wiki/Web_Interface_Via_Nginx_Proxy should allow me to visit the GUI without the need for the port :8006 correct. I know that's how a nginx proxy works but I want to confirm that is the way with proxmox.

yes that's correct, you should be able to access it so: https://your.domain (where your.domain has a record pointing to your PVE IP)

 

[robetus]

Here is my nginx configuration:

upstream proxmox {
    server "proxmox.mydomain.com";
}
 
server {
    listen 80 default_server;
    rewrite ^(.*) https://$host$1 permanent;
}
 
server {
    listen 443;
    server_name _;
    ssl on;
    ssl_certificate /etc/letsencrypt/live/proxmox.mydomain.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/proxmox.mydomain.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
    proxy_redirect off;
    location / {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass https://localhost:8006;
    proxy_buffering off;
    client_max_body_size 0;
    proxy_connect_timeout  3600s;
        proxy_read_timeout  3600s;
        proxy_send_timeout  3600s;
        send_timeout  3600s;
    }
}

The A record propagation for proxmox.mydomain.com has been verified to my promox install IP. I pretty much copy and pasted the configuration over from the link I have and only changed the domain and the ssl certs which are all valid. Nginx is running with no errors an no errors are found in any of the nginx config files.

 

[oguz]

Here is my nginx configuration:

thanks, config looks okay at first glance.

The A record propagation for proxmox.mydomain.com has been verified to my promox install IP. I pretty much copy and pasted the configuration over from the link I have and only changed the domain and the ssl certs which are all valid. Nginx is running with no errors an no errors are found in any of the nginx config files.

can you test if it works when you use the exact config from the wiki (with the pve certs)?
just to make sure things are working as expected without let's encrypt.
after swapping the config, restart nginx and try accessing your panel. if that works we can narrow the issue down

 

[robetus]

Bad gateway with default config but it's not secure. Same result.

 

[oguz]

Bad gateway with default config but it's not secure. Same result.

default configuration shouldn't give you bad gateway, maybe something else on your setup is causing it?

could you post the output from all your nginx configuration files:

find /etc/nginx -type f -exec echo {} ';' -exec cat {} ';'

and your nginx log:

cat /var/log/nginx/error.log
cat /var/log/nginx/access.log # you can mask your IP addresses if you wish

 

[robetus]

I purged nginx and reinstalled and still the same gateway error.

cat /var/log/nginx/error.log:

[error] 14755#14755: *1 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream, client: 201.84.22.108, server: _, request: "GET / HTTP/1.1", upstream: "https://127.0.0.1:8006/", host: "proxmox.mysite.com"

find /etc/nginx -type f -exec echo {} ';' -exec cat {} ';'

produces an output too long to post inside one reply. Do you want to see a specific section of it?

 

[oguz]

produces an output too long to post inside one reply. Do you want to see a specific section of it?

do you have a file /etc/nginx/sites-enabled/default? if so please delete that

also is your server configured with the correct domain name in /etc/hosts?

I purged nginx and reinstalled and still the same gateway error.

not sure what's wrong, works fine here with the config from the wiki

 

[robetus]

In /etc/hosts it shows: my.ip.address proxmox.mysite.com proxmox pvelocalhost

And no there is nothing inside /etc/nginx/sites-enabled/

Bummer, I've been using proxmox for 6 years now after many upgrades and this is a first for me. It always worked until now.

 

[robetus]

I got to the:

Welcome to nginx!​

page using this:

# HTTPS server
server {
   listen 443 ssl;
   server_name proxmox.mysite.com;
    ssl_certificate /etc/letsencrypt/live/proxmox.mysite.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/proxmox.mysite.com/privkey.pem; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
   ssl_session_cache shared:SSL:10m;
   ssl_session_timeout 5m;
   ssl_protocols TLSv1.1 TLSv1.2;
   ssl_prefer_server_ciphers on;
   ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256;
   add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
}

# HTTP redirect
server {
   listen 80;
   server_name proxmox.mysite.com;
   return 301 https://$server_name$request_uri;
}

but when I add the proxy info form the wiki:

    proxy_redirect off;
    location / {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass https://localhost:8006;
    proxy_buffering off;
    client_max_body_size 0;
    proxy_connect_timeout  3600s;
        proxy_read_timeout  3600s;
        proxy_send_timeout  3600s;
        send_timeout  3600s;
    }

it goes back to the bad gateway. I tried accessing the domain with :8006 as well which I wouldn't care to do, I just want SSL I don't need the proxy and it doesn't work either. It shows the error connection reset and doesn't even load the page.

A simpler conf used same result as above (Welcome to Nginx! page):

# HTTPS server
server {
   listen 443 ssl;
   server_name proxmox.mysite.com;
   ssl_certificate /etc/letsencrypt/live/proxmox.mysite.com/fullchain.pem; # managed by Certbot
   ssl_certificate_key /etc/letsencrypt/live/proxmox.mysite.com/privkey.pem; # managed by Certbot
   ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
   ssl_session_cache shared:SSL:10m;
   ssl_session_timeout 5m;
   ssl_protocols TLSv1.1 TLSv1.2;
}

# HTTP redirect
server {
   listen 80;
   server_name proxmox.mysite.com;
   return 301 https://$server_name$request_uri;
}

 

[robetus]

I figured out the issue on this. It's not an nginx or SSL issue it's a pveproxy issue. I have my pveproxy secured with specific IPs like so:

ALLOW_FROM="192.168.0.1,192.168.0.3,192.168.0.1,192.168.0.5"
DENY_FROM="all"
POLICY="allow"

Obviously not using my real IPs, but I have all the IPs of the servers listed including the IP of the machine I'm trying to access the GUI from. Am I missing something? Is there something I can check and add to the list?

When I remove all restrictions from the pveproxy file nginx loads normally with the domain and the reserve proxy works fine. I'm trying to think of what IP address I need to add in there, maybe letsencrypt's IPs?

 

[oguz]

When I remove all restrictions from the pveproxy file nginx loads normally with the domain and the reserve proxy works fine. I'm trying to think of what IP address I need to add in there, maybe letsencrypt's IPs?

thanks, i can reproduce that 502 now with your pveproxy settings.

since nginx proxy passes from 127.0.0.1, if you allow 127.0.0.1 in /etc/default/pveproxy it should work,
but then it will work for any IP address (since pveproxy will see all requests coming from 127.0.0.1)

if you want to restrict access to the GUI you should look into that from the nginx level if you're using it as reverse proxy, any IP being allowed/disallowed has to be done there.

 

[robetus]

Thanks @oguz makes total sense. I just went ahead and did a htpassword directly on the login and then 2fa. Got tired of limiting IPs but all solved now, thanks for all the support.

 

[oguz]

Thanks @oguz makes total sense. I just went ahead and did a htpassword directly on the login and then 2fa. Got tired of limiting IPs but all solved now, thanks for all the support.

great, you're welcome! please mark the thread as [SOLVED] so that others can know what to expect

 

[eyildiz]

I solved this problem by deleting /etc/pve/local/pveproxy-ssl.key and /etc/pve/local/pveproxy-ssl.pem files and it worked.