[SOLVED] Failing connection to pbs after setting a custom certificate in it.

resoli

Renowned Member
Mar 9, 2010
147
4
83
Hello,

After having uploaded a custom certificate, made with out internal PKI.

Even if I updated the cert fingerprint on pbs storage configuration on pve side, pbs is no more accessible from our pve clusters, which show this error:

Code:
proxmox-backup-client failed: Error: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1914: (500)

pbs is at v2.2-5
pve
is at v7.2-7

bye,
rob
 
Last edited:
are you sure the fingerprint is correct? can you try curl -vk https://PBSHOST:8007/ and post the output here?
 
Yes, checked multiple times. Checked with wget and curl (before reverting to the autogenereted certificate). Connecting to the pbs web GUI with the custom certificate installed works, only pve -> pbs connection fails.
 
Last edited:
do you maybe have a .config/proxmox-backup/fingerprints file containing the outdated fingerprint?
 
Code:
# cat .config/proxmox-backup/fingerprints
cat: .config/proxmox-backup/fingerprints: No such file or directory
 
without more details about the certificate it will be hard to tell what's going on..
 
I can provide cert and chain, but may be is more productive if I do a new attempt on a test server.
 
Last edited:
  • Like
Reactions: fabian
Hello @fabian , just tried on a test server (pbs 2.2-1, error connecting it as a storage from pve 7.2-4). Same error.

How can I provide you the cert?
 
The pbs storage addition seems succesful at first, but the storage icon has a question mark, and browsing the storage for backup listing gives the error in the first message here. Same message in syslog.
 
I checked the sha-256 fingerprint of my custom cert with openssl, and it is identical to the fingerprint copied from pbs dashboard.
 
Here's the result of a curl connection test (without -k because the root is in my trusted certs list)
Bash:
$ curl -v https://REDACTED_HOST:8007
*   Trying REDACTED_IP:8007...
* Connected to REDACTED_HOST (REDACTED_IP) port 8007 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=IT; ST=Trento; L=Trento; O=Comune di Trento; OU=Innovazione e Transizione Digitale; CN=REDACTED_HOST.intra.comune.trento.it
*  start date: Aug 17 13:46:18 2022 GMT
*  expire date: Aug 16 13:46:18 2024 GMT
*  subjectAltName: host "REDACTED_HOST" matched cert's "REDACTED_HOST"
*  issuer: C=IT; O=Comune di Trento; OU=Servizio Sistema informativo; CN=Comune di Trento Component CA
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/1.1
> Host: REDACTED_HOST:8007
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: text/html
< content-length: 1938
< date: Wed, 17 Aug 2022 14:28:23 GMT
<
<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
    <title>REDACTED_HOST - Proxmox Backup Server</title>
    <link rel="icon" sizes="128x128" href="/images/logo-128.png" />
    <link rel="apple-touch-icon" sizes="128x128" href="/pve2/images/logo-128.png" />
    <link rel="stylesheet" type="text/css" href="/extjs/theme-crisp/resources/theme-crisp-all.css" />
    <link rel="stylesheet" type="text/css" href="/extjs/crisp/resources/charts-all.css" />
    <link rel="stylesheet" type="text/css" href="/fontawesome/css/font-awesome.css" />
    <link rel="stylesheet" type="text/css" href="/widgettoolkit/css/ext6-pmx.css" />
    <link rel="stylesheet" type="text/css" href="/css/ext6-pbs.css" />
    
    <script type='text/javascript'> function gettext(buf) { return buf; } </script>
    
    
    <script type="text/javascript" src="/extjs/ext-all.js"></script>
    <script type="text/javascript" src="/extjs/charts.js"></script>
    
    <script type="text/javascript">
    Proxmox = {
    Setup: { auth_cookie_name: 'PBSAuthCookie' },
    NodeName: "REDACTED_HOST",
    UserName: "",
    CSRFPreventionToken: "",
    };
    </script>
    <script type="text/javascript" src="/widgettoolkit/proxmoxlib.js"></script>
    <script type="text/javascript" src="/extjs/locale/locale-en.js"></script>
    <script type="text/javascript">
      Ext.History.fieldid = 'x-history-field';
    </script>
    <script type="text/javascript" src="/qrcodejs/qrcode.min.js"></script>
    <script type="text/javascript" src="/js/proxmox-backup-gui.js"></script>
  </head>
  <body>
    <!-- Fields required for history management -->
    <form id="history-form" class="x-hidden">
      <input type="hidden" id="x-history-field"/>
    </form>
  </body>
</html>
* Connection #0 to host REDACTED_HOST left intact
 
if the certificate is trusted by the system, you shouldn't even need to provide a fingerprint at all..

How can I provide you the cert?

you can post the output of openssl x509 -in /path/to/cert.pem -noout -text
 
Code:
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 56 (0x38)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = IT, O = Comune di Trento, OU = Servizio Sistema informativo, CN = Comune di Trento Component CA
        Validity
            Not Before: Aug 17 13:46:18 2022 GMT
            Not After : Aug 16 13:46:18 2024 GMT
        Subject: C = IT, ST = Trento, L = Trento, O = Comune di Trento, OU = Innovazione e Transizione Digitale, CN = REDACTED_HOST.intra.comune.trento.it
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:a5:39:ac:a3:81:ac:f4:68:2c:c8:d5:12:4c:81:
                    e3:81:b5:2b:06:9e:6b:35:d2:fe:ca:c3:04:b1:75:
                    5f:59:d2:86:ac:e2:f7:a1:70:02:77:28:d6:cb:dd:
                    28:40:ea:b4:72:40:7c:03:2c:5f:32:63:1d:fb:31:
                    73:78:44:f4:93:c1:73:2f:6d:8a:d5:88:f8:eb:dd:
                    28:e1:8a:51:e8:5d:ef:40:ea:8a:31:e1:22:38:14:
                    b9:77:ed:c0:9c:9d:4b:9b:07:f0:65:64:87:bd:33:
                    d5:5e:d8:9f:91:84:85:d0:13:49:b0:4d:27:6a:8b:
                    2c:cd:27:77:50:d6:69:27:7f:9e:1d:12:ea:94:33:
                    88:84:8d:af:98:01:3b:b4:9f:fa:52:31:f7:64:c4:
                    95:b0:18:ec:dc:14:fc:e1:67:63:5c:f5:a6:2e:47:
                    10:83:4d:3c:c9:7f:43:ac:29:0b:d6:db:2c:29:e3:
                    37:ee:9f:3d:b1:e9:a4:c5:78:fe:1e:32:3e:77:46:
                    cf:24:3e:d8:be:f8:1e:bd:13:12:26:ff:17:e9:d0:
                    2c:cb:8e:2e:2c:05:e3:f5:21:b5:ee:a1:e3:ae:df:
                    9c:00:82:20:af:17:8e:b8:6c:65:7c:13:51:3a:a4:
                    8b:85:dc:2d:d1:87:de:28:b1:c1:ce:d4:bb:32:b6:
                    f6:af
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier:
                D8:30:7A:9A:A4:7B:27:1A:D7:18:40:39:D9:55:A3:BD:2A:85:A8:05
            X509v3 Authority Key Identifier:
                45:83:DF:25:FC:58:E9:6D:67:BA:07:84:01:FA:7B:AE:56:76:C8:F5
            Authority Information Access:
                CA Issuers - URI:https://webapps.comune.trento.it/ca/component-ca.cer
                OCSP - URI:https://webapps.comune.trento.it/ocsp
            X509v3 CRL Distribution Points:
                Full Name:
                  URI:https://webapps.comune.trento.it/ca/component-ca.crl
            X509v3 Certificate Policies:
                Policy: 1.3.6.1.4.1.0.1.7.9
            X509v3 Subject Alternative Name:
                DNS:REDACTED_HOST, DNS:REDACTED_HOST.intra.comune.trento.it
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        65:2d:28:29:1d:d7:19:b6:46:b2:86:89:80:5d:2b:e7:c8:d9:
        4b:18:22:6b:63:1c:5c:cc:66:f4:dd:9d:6f:56:8c:ed:3a:7a:
        f7:14:fd:ff:71:14:e3:6d:a1:fc:c0:34:15:67:23:c8:3a:fa:
        52:93:3f:06:f4:7a:87:3b:2f:b3:f1:59:db:fb:1d:8d:94:26:
        e7:cb:92:fc:0e:ca:be:a6:63:e7:73:d2:14:e4:e4:b7:b5:92:
        32:f1:c9:ea:d1:6f:5a:9a:7f:7d:f8:c4:ec:f2:b0:99:fd:57:
        55:09:2f:2c:ad:5b:33:01:ce:5f:8b:38:cc:34:ed:f5:86:af:
        26:99:59:ad:7e:29:ab:4b:d6:2c:a0:b2:29:4f:60:0e:9a:24:
        81:7e:eb:3c:fd:87:28:a0:6d:12:67:9a:34:c8:89:9b:fa:1d:
        b0:95:4c:f7:f1:99:28:26:39:69:ae:b2:2f:31:61:6d:8d:de:
        83:c7:1e:43:03:b4:86:0b:e9:57:54:b4:40:e5:5f:65:91:24:
        16:a9:cd:d1:87:8b:eb:6f:4a:1d:ae:a0:3e:6e:9e:4f:a3:8f:
        86:f0:f8:c0:fe:26:ef:a5:46:cb:d5:dc:22:ca:fd:47:39:b0:
        c0:45:4e:4c:e6:c3:c3:7b:63:ff:0c:39:84:41:39:cf:6c:d2:
        58:81:ff:13
 
the only thing that looks a bit strange there is the key usage and EKU, but those just contain extra stuff that is not strictly needed but shouldn't cause any issues.

what happens if you just run proxmox-backup-client status --repository ...? does it prompt for a fingerprint, and if so, is the one displayed the expected one?
 
I tried generate a self signed cert (chain with only 1 cert) and it works. May be there is some problem in chain verification?
 
did you upload the chain or just the cert? how many levels are there? if there's more than two (server and CA), which one is trusted by the client-side system?
 
the only thing that looks a bit strange there is the key usage and EKU, but those just contain extra stuff that is not strictly needed but shouldn't cause any issues.

These are quite common for web server certificates ...

what happens if you just run proxmox-backup-client status --repository ...? does it prompt for a fingerprint, and if so, is the one displayed the expected one?

Will try this evening, I have not the privkey at hand now.
 
did you upload the chain or just the cert? how many levels are there? if there's more than two (server and CA), which one is trusted by the client-side system?

Chain. Three levels, excluding root. I tried to include the root as well, but it makes no difference.
 
If you want, send me a csr and i will generate a short lived cert for your test ...

PS: Maybe is worthwhile to open an issue for this?
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!