ProxmoxVE Doesn't Support Letsencrypt CA?

nabor · Feb 10, 2021

I have both Poxmox VE and PBS running. Recently, I set up Letsencrypt on PBS, however, VE is still not trusting the SSL. I had to manually add the Fingerprint in the storage options.

Am I missing something?

Moayad · Feb 11, 2021

Hi,

Have you see our wiki guide about Let's Encrypt [0] in PBS and PVE [1]

nabor said:
Am I missing something?

May you check the firewall?

[0] https://pbs.proxmox.com/wiki/index.php/HTTPS_Certificate_Configuration#Let.27s_Encrypt_using_acme.sh
[1] https://pve.proxmox.com/wiki/Certificate_Management#sysadmin_certs_get_trusted_acme_cert

nabor · Feb 15, 2021

Moayad said:
Hi,

Have you see our wiki guide about Let's Encrypt [0] in PBS and PVE [1]

May you check the firewall?

[0] https://pbs.proxmox.com/wiki/index.php/HTTPS_Certificate_Configuration#Let.27s_Encrypt_using_acme.sh
[1] https://pve.proxmox.com/wiki/Certificate_Management#sysadmin_certs_get_trusted_acme_cert

Hi Moayad,

I'm using certbot for getting Letsencrypt certs and created symbolic links to /etc/proxmox-backup/. Added acl so that the backup user can read the certs. No access issues with any browsers.

ACME has already registered on VE. However, it doesn't help because VE is using an internal DNS, not reachable from the outside.

I think this issue is because of the VE is not trusting Letsencrypt CA. If I issue curl on my VE against the PBS, I got this error:

Code:

root@proxmom_ve # curl https://pbx.example.com:8007
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

fabian · Feb 15, 2021

does it work in the browser? if so, you might have setup the wrong file (just the certificate instead of certificate+CA certificate) with PBS. browser fetch intermediate certificates if referenced in the cert itself, but most other clients including PVE don't do that.

nabor · Feb 16, 2021

fabian said:
does it work in the browser? if so, you might have setup the wrong file (just the certificate instead of certificate+CA certificate) with PBS. browser fetch intermediate certificates if referenced in the cert itself, but most other clients including PVE don't do that.

The browser works fine. I'm using the "cert.pem" file which is generated by Letsencrypt and I believe that's the only cert file I can use and the cert+ca bundle I think it's an intermediate cert?

Code:

# ll /etc/proxmox-backup/proxy.*
-rw-r----- 1 root backup 3243 Feb  5 15:29 /etc/proxmox-backup/proxy.key.bak
-rw-r----- 1 root backup 2163 Feb  5 15:29 /etc/proxmox-backup/proxy.pem.bak
lrwxrwxrwx 1 root root     55 Feb 10 15:07 /etc/proxmox-backup/proxy.key -> /etc/letsencrypt/live/pbs.example.com/privkey.pem
lrwxrwxrwx 1 root root     52 Feb 16 15:48 /etc/proxmox-backup/proxy.pem -> /etc/letsencrypt/live/pbs.example.com/cert.pem

t.lamprecht · Feb 16, 2021

If browser works but not curl (and other CLI tools not side loading the rest of the chain) with such an error then it's really using just the leaf cert instead of the fullchain one, so adapt and try:

Bash:

ln -sf /etc/letsencrypt/live/pbs.example.com/fullchain.pem /etc/proxmox-backup/proxy.pem

FYI: we have some documentation for Let's Encrypt and PBS with acme.sh as ACME client:
https://pbs.proxmox.com/index.php/HTTPS_Certificate_Configuration#Let.27s_Encrypt_using_acme.sh

nabor · Feb 16, 2021

t.lamprecht said:
If browser works but not curl (and other CLI tools not side loading the rest of the chain) with such an error then it's really using just the leaf cert instead of the fullchain one, so adapt and try:

Bash:

ln -sf /etc/letsencrypt/live/pbs.example.com/fullchain.pem /etc/proxmox-backup/proxy.pem

FYI: we have some documentation for Let's Encrypt and PBS with acme.sh as ACME client:
https://pbs.proxmox.com/index.php/HTTPS_Certificate_Configuration#Let.27s_Encrypt_using_acme.sh

Thanks Thomas! This fixed the VE connection issue. However, one of my bare mental servers start complaining SSL issue:

Code:

#  proxmox-backup-client backup root.pxar:/ --repository backup@pbs@pbs.example.com:backups
Password for "backup@pbs": ****************
Starting backup: host/web1/2021-02-16T08:09:42Z
Client name: web1
Starting backup protocol: Tue Feb 16 21:09:58 2021
Error: error trying to connect: the handshake failed: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:: unable to get local issuer certificate

t.lamprecht · Feb 16, 2021

Stupid question, but just to be sure: did you reload the API daemons after replacing the link?

Bash:

systemctl reload proxmox-backup-proxy

(or over web interface)

nabor · Feb 16, 2021

t.lamprecht said:
Stupid question, but just to be sure: did you reload the API daemons after replacing the link?

Bash:

systemctl reload proxmox-backup-proxy

(or over web interface)

Surely I did it on the PBS. If I didn't reload the service, VE wouldn't work.

t.lamprecht · Feb 16, 2021

Yeah thought so, that's why I added the "stupid question" prefix

It's a bit weird though, as Proxmox VE uses the proxmox-backup-client itself for most stuff, and the client works here with Let's Encrypt provided certs just fine since a while.

Is there any proxy or cache in between? Also, is this the exact same PVE server you added the storage too or another machine?

nabor · Feb 16, 2021

t.lamprecht said:
Yeah thought so, that's why I added the "stupid question" prefix

It's a bit weird though, as Proxmox VE uses the proxmox-backup-client itself for most stuff, and the client works here with Let's Encrypt provided certs just fine since a while.

Is there any proxy or cache in between? Also, is this the exact same PVE server you added the storage too or another machine?

Exactly the same PBS server and same datastore. It works on another machine (web2). If I switch the cert back on PBS, the web1 works again, BUT web2 stopped working and showing a similar error.

Does proxmox client has cache?

t.lamprecht · Feb 17, 2021

nabor said:
Does proxmox client has cache?

Not really for connections, albeit there's a fingerprint cache in $HOME/.config/proxmox-backup/fingerprints - you could try to clear that, albeit the unable to get local issuer certificate error is before fingerprint checks even can happen, so not really sure if that helps.

Is the ca-certificates package installed?

What does curl think?

Bash:

curl https://pbsdev1.pmx.lamprecht.org:8007

nabor · Feb 17, 2021

t.lamprecht said:
Not really for connections, albeit there's a fingerprint cache in $HOME/.config/proxmox-backup/fingerprints - you could try to clear that, albeit the unable to get local issuer certificate error is before fingerprint checks even can happen, so not really sure if that helps.

Is the ca-certificates package installed?

What does curl think?

Bash:

curl https://pbsdev1.pmx.lamprecht.org:8007

You're right, I emptied the fingerprint file and still no luck.

The ca bundle is installed and curl looks happy:

Code:

# rpm -qa|grep ca-certificates
ca-certificates-2020.2.41-70.0.el7_8.noarch
# curl https://pbs.example.com:8007
<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
    <title>pbsbackup - Proxmox Backup Server</title>
    <link rel="icon" sizes="128x128" href="/images/logo-128.png" />
    <link rel="apple-touch-icon" sizes="128x128" href="/pve2/images/logo-128.png" />
    <link rel="stylesheet" type="text/css" href="/extjs/theme-crisp/resources/theme-crisp-all.css" />
    <link rel="stylesheet" type="text/css" href="/extjs/crisp/resources/charts-all.css" />
    <link rel="stylesheet" type="text/css" href="/fontawesome/css/font-awesome.css" />
    <link rel="stylesheet" type="text/css" href="/widgettoolkit/css/ext6-pmx.css" />
    <link rel="stylesheet" type="text/css" href="/css/ext6-pbs.css" />

    <script type='text/javascript'> function gettext(buf) { return buf; } </script>


    <script type="text/javascript" src="/extjs/ext-all.js"></script>
    <script type="text/javascript" src="/extjs/charts.js"></script>

    <script type="text/javascript">
    Proxmox = {
        Setup: { auth_cookie_name: 'PBSAuthCookie' },
        NodeName: "proxbackup1",
        UserName: "",
        CSRFPreventionToken: "",
    };
    </script>
    <script type="text/javascript" src="/widgettoolkit/proxmoxlib.js"></script>
    <script type="text/javascript" src="/extjs/locale/locale-en.js"></script>
    <script type="text/javascript">
      Ext.History.fieldid = 'x-history-field';
    </script>
    <script type="text/javascript" src="/qrcodejs/qrcode.min.js"></script>
    <script type="text/javascript" src="/js/proxmox-backup-gui.js"></script>
  </head>
  <body>
    <!-- Fields required for history management -->
    <form id="history-form" class="x-hidden">
      <input type="hidden" id="x-history-field"/>
    </form>
  </body>
</html>

fabian · Feb 18, 2021

are the client binaries identical? maybe something went wrong at build time that causes it to not pick up ca-certificates?

nabor · Feb 18, 2021

Possibly, the server which is not working is running Centos7 with 1.0.6 while the working one is running Debian 10 with 1.0.8. Those're the only differences.

Search

Search

ProxmoxVE Doesn't Support Letsencrypt CA?

nabor

New Member

Moayad

Proxmox Staff Member

nabor

New Member

fabian

Proxmox Staff Member

nabor

New Member

t.lamprecht

Proxmox Staff Member

nabor

New Member

t.lamprecht

Proxmox Staff Member

nabor

New Member

t.lamprecht

Proxmox Staff Member

nabor

New Member

t.lamprecht

Proxmox Staff Member

nabor

New Member

fabian

Proxmox Staff Member

nabor

New Member