No connection to Proxmox Backup Server, tls_process_server_certificate

[eagle2020]

I'm running PVE 6.4, PBS 1.1, latest udates!

Since two weeks no connection from PVE to PBS is possible.
The fingerprint of the PBS changed, but I have already updated the fingerprint on the PVE Server.
I get now following error:

proxmox-backup-client failed: Error: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1915:

I use letsencrypt certs for both servers, so actually I don't need the fingerprint if I'right. I can connect to the PBS with and without the fingerprint, but I still get above error.

Anyone an idea whats my issue?


Frank

 

[eagle2020]

I just did a new PBS installation inside proxmox PVE, just for testing.
The new installation works (I can connect to the new PBS from my PVE).
The only difference is: the https connection of the new PBS is not secured by LetsEncrypt certificate! (I'm using fingerprint now)
I suppose, my issue is connected with the LetsEncrypt certificate of my original installation, but I don't no why and how to solve the problem.
Any Ideas?

 

[fabian]

is the time set correctly on both nodes? can you do curl -v https://PBS:8007 from the PVE node (replace PBS with hostname of PBS system)?

 

[eagle2020]

I get a certificate expired error.
But when I connect vie browser to my PBS the certificate is valid 29.November 2021.
So what is the problem? Are there different certificats for the web gui and the pbs server?
The Certificate is a wildcard certificate by letsencrypt (I manually uploaded the cert).

* Expire in 0 ms for 1 (transfer 0x55a144693fb0)
*   Trying 10.0.2.105...
* TCP_NODELAY set
* Expire in 149999 ms for 3 (transfer 0x55a144693fb0)
* Expire in 200 ms for 4 (transfer 0x55a144693fb0)
* Connected to pbs.mydomain.de (10.0.2.105) port 8007 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, certificate expired (557):
* SSL certificate problem: certificate has expired
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server

 

[fabian]

possibly https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ ? although I am not sure why that should affect you on an up-to-date 6.4...

 

[eagle2020]

infact my backups stopped working on 29.September.
but that would mean, pve is not trusting ISRG Root X1, makes no sense.
how can I figure out, if this is my problem?

In a first step I coud try to remove the letsEncrypt Certificate and use the fingerprint with self signd certs.

 

[fabian]

could you please dump the full chain - e.g. with openssl s_client -showcerts PBS:8007 (again, with PBS replaced with your host).

 

[eagle2020]

BTW, thanks for your help so far,
it seams the DST Root CA X3 is the problem.
I probably have to get a new cert for my domain!
I'm just wondering why the web GUI is working fine with the same cert!

CONNECTED(00000003) --- Certificate chain 0 s:CN = *.mydomain.de i:C = US, O = Let's Encrypt, CN = R3 -----BEGIN CERTIFICATE----- MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD EwJSMzAeFw0yMTA4MzEwMDE2MzNaFw0yMTExMjkwMDE2MzJaMBkxFzAVBgNVBAMM DioucGxhc21hZnVuLmRlMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA BQADggEBAIz3xmDptdTQPfxkjDDqPtZupPkR/+Mlph6ZW3m7UvBEqU5sFxuJlRhx ot/9ctaUZmObKocRepfVA6MJD0SUXs8Hva23G7CA3yg0/j3Un8UQ5vFqENP02CUl ahxue75lwUAeousyik2iZ6SuIx4l+es= -----END CERTIFICATE----- 1 s:C = US, O = Let's Encrypt, CN = R3 i:O = Digital Signature Trust Co., CN = DST Root CA X3 -----BEGIN CERTIFICATE----- MIIEZTCCA02gAwIBAgIQQAF1BIMUpMghjISpDBbN3zANBgkqhkiG9w0BAQsFADA/ MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT S8MXjohyc9z9/G2948kLjmE6Flh9dDYrVYA9x2O+hEPGOaEOa1eePynBgPayvUfL qjBstzLhWVQLGAkXXmNs+5ZnPBxzDJOLxhF2JIbeQAcH5H0tZrUlo5ZYyOqA7s9p O5b85o3AM/OJ+CktFBQtfvBhcJVd9wvlwPsk+uyOy2HI7mNxKKgsBTt375teA2Tw UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg== -----END CERTIFICATE----- --- Server certificate subject=CN = *.mydomain.de issuer=C = US, O = Let's Encrypt, CN = R3 --- No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: X25519, 253 bits --- SSL handshake has read 3009 bytes and written 388 bytes Verification error: certificate has expired --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 2048 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 10 (certificate has expired)

 

[eagle2020]

So finally I got my backup working again - but without the LetsEncrypt Certificate.
I created a new Self Signed Certificate on the PBS and used the fingerprint for the PVE.

I also tested a renewed LetsEncrypt certificate from today - same issue.
So by now I'll stick to the self signed certs on the PBS!

 

[fabian]

the web interface works because browsers use their own built-in trust store, and can thus ship exceptions/updates/.. quite fast. you don't really need to get a new or self-signed certificate, you just need to switch to using the alternate chain..

 

[fabian]

could you also run the following commands?

ldd /usr/bin/proxmox-backup-client
dpkg -l "libssl*"

thanks!

 

[eagle2020]

could you also run the following commands?

ldd /usr/bin/proxmox-backup-client
dpkg -l "libssl*"

thanks!

        linux-vdso.so.1 (0x00007fff6b5f6000)
        libacl.so.1 => /lib/x86_64-linux-gnu/libacl.so.1 (0x00007f941f22f000)
        libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f941f190000)
        libfuse3.so.3 => /lib/x86_64-linux-gnu/libfuse3.so.3 (0x00007f941f152000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f941efce000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f941efc4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f941ee03000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f941fbb5000)
        libssl.so.1.1 => /lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f941ed6f000)
        libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f941ea86000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f941ea6c000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f941ea4b000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f941e8c8000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f941e8c3000)
        libattr.so.1 => /lib/x86_64-linux-gnu/libattr.so.1 (0x00007f941e8b9000)

||/ Name            Version          Architecture Description
+++-===============-================-============-===============================================
ii  libssl1.1:amd64 1.1.1d-0+deb10u7 amd64        Secure Sockets Layer toolkit - shared libraries

 

[eagle2020]

the web interface works because browsers use their own built-in trust store, and can thus ship exceptions/updates/.. quite fast. you don't really need to get a new or self-signed certificate, you just need to switch to using the alternate chain..

I thought so already.
But in fact I'm happy that I was able to got the LetsEncrypt certs running at my home servers. But I don't know how to change the chain (I never thought I need to concern about that).
For my nextcloud server I also have a (different) LetsEncrypt cert and after 29.September I encounterd some cert problems (untrusted DST Root CA X3), but I could confirm, so the clients are working.
May be I get rid of the problem if LetsEncrypt changes the root cert? But may be it's some what to complicated for me
Thanks for your help!

 

[fabian]

I am really confused as to how you trigger this issue - we confirmed that the PBS 1.x clients work fine with the default LE chain terminating at the expired root.. all the docs also agree that openssl 1.1 shouldn't be affected, only 1.0.2 which we are not using in PBS at all..

 

[eagle2020]

I'm sorry for that, it's just a hobby. And I don't get your point from your last post.
I think I'm using OpenSSL 1.1, so I should not have any problems but I have. Fortunately, my backups are running fine with the self signed certificate as long as I don't understand my issue.

 

[fabian]

no worries - I was just expressing my confusion I can confirm that it works in general with the setup you describe, so there must be some other factor at play on your system.