[SOLVED] Failing to backup PVE (7.4-3) to my NAS over NFS4 using PBS (2.4-1)

[LooneyTunes]

Hi,

I have setup a PBS server (2.4-1) that uses my NAS over NFS4. I followed this excellent guide.

PBS can see the created datastore, and it is writable. Adding the storage in PVE (7.4-3) seemed first to have gone well, no thrown error at least, but the GUI shows a questionmark on it, and hovering above tells me 'Status: Unknown', which probably isn't a good sign... When I then try to backup, following error is shown

ERROR: Backup of VM 101 failed - VM 101 qmp command 'backup' failed - backup connect failed: command error: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1914:

Looking into the log didn't reveal much more

Apr 18 14:32:35 pve pvestatd[949]: proxmox-backup-client failed: Error: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1914:

Trying to google this turns up all sorts of creative suggestions - some seem to suggest this has to do with the SSL certs, but I'd doubt that as they work fine in every other way. Please help me understand what the issue is, or if it in fact is something with the certs, and what to do about it.

Thanks

 

[shanreich]

This seems indeed to be an issue with the SSL certificate. Are you using a self-signed one, or do you have a proper certificate (e.g. Lets Encrypt)? Has the certificate changed after you added the datastore?

 

[LooneyTunes]

Well, it is indeed a self-signed one, but a bit surprised it is not accepted. I have not changed it since adding the datastore.

 

[shanreich]

On the PBS, what is the output of this command:

proxmox-backup-manager cert info

And on PVE side, how does the storage.cfg look like?

cat /etc/pve/storage.cfg

 

[LooneyTunes]

On the PBS, what is the output of this command:

proxmox-backup-manager cert info

proxmox-backup-manager cert info
Subject: <masked>
    DNS:pbs.home.arpa
Issuer: <masked>
Validity:
    Not Before: Apr 15 14:12:04 2023 GMT
    Not After : May  4 14:12:04 2025 GMT
Fingerprint (sha256): <masked>
Public key type: rsaEncryption
Public key bits: 4096

And on PVE side, how does the storage.cfg look like?

cat /etc/pve/storage.cfg

pbs: PBS
        datastore NFSfromSynology
        server pbs.home.arpa
        content backup
        fingerprint <masked>
        prune-backups keep-all=1
        username client@pbs

Fingerprint in both outputs are identical

I just found this thread with more or less the exact same issue. As I use intermediate cert, should I as explained in post #22 upload that to /usr/local/share/ca-certificates/ on both PBS & PVE ?

 

[LooneyTunes]

Ok, tried the above, both with the intermediate and the CA, but didn't work any better unfortunately. Any suggestions?

 

[shanreich]

Does the certificate, that you uploaded to the PBS, contain the full chain or just the leaf certificate?

You would also need to install the root certificate rather than the intermediate certificate on the PVE (if you haven't already).

 

[LooneyTunes]

Does the certificate, that you uploaded to the PBS, contain the full chain or just the leaf certificate?

You would also need to install the root certificate rather than the intermediate certificate on the PVE (if you haven't already).

It contains the leaf and intermediate cert. Ok, but why would it not accept the intermediate, as that was used to sign the leaf?

I will try again Thanks

 

[shanreich]

Yes, you're right. Of course it should work with the intermediate certificate as well - sorry.

Did you make sure to run update-ca-certificates after adding the intermediate certificates?

 

[LooneyTunes]

Yes, you're right. Of course it should work with the intermediate certificate as well - sorry.

Did you make sure to run update-ca-certificates after adding the intermediate certificates?

Yes I did, with the -f flag too. But no change in read numbers of certs, it was the same regardless if I had added it to /usr/local/share/ca-certificates/ or not. I am thinking it maybe should be somewhere else?

this was the output I got, always showing 129 certs...

update-ca-certificates -f
Clearing symlinks in /etc/ssl/certs...
done.
Updating certificates in /etc/ssl/certs...
129 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

It wouldn't be enough to just copy it into /etc/ssl/certs, would it?

 

[shanreich]

/usr/local/share/ca-certificates should be fine

Did you try with -v flag? That should give more output.
Does the cert file have the proper extension (.crt)?

 

[LooneyTunes]

/usr/local/share/ca-certificates should be fine

Did you try with -v flag? That should give more output.
Does the cert file have the proper extension (.crt)?

One down, I did not use the .crt extention. When using that, it imported the cert at least. But PVE is still not recognizing the datastore. I have not rebooted, will try that next

Edit: reboot did not help. re-adding the datastore in PVE did not do it either...

Edit2: When messing with PVE & it's certs I noted that the pve-ssl.pem is all wrong! It has the wrong domain and also ip... Will look into regenerating this to better match... Ok, updated, no difference regarding the datastore though

Edit3:

• How can we proceed with this?
  • Is it possible to get more information about what makes PBS fail to validate the cert, than the error?
  • Or turn it around, what are the full requirements on the certs for PBS/PVE to work properly?
  • Can it be firewall related? Ports on the LAN that needs opening?

When messing with this yesterday I managed to get both PVE and PBS in a very peculiar mode. They did no longer recognize PAM and had to be shut down/restarted through SSH. All I had done was changing their certs and adding different kind of storage. Strange.

I am willing to reinstall if necessary, and also recreate/modify my certs slightly (even though if not absolutely needed I'd rather not)

 

[shanreich]

You could always use openssl on the PVE host to try and get more debug information:

openssl s_client -connect pbs.home.arpa

Generally, if the certificate is correctly issued & configured on the PBS (hard for me to tell with the info available), there must be some misconfiguration on your PVE. You can attach the certificate of PBS, if you want me to gloss over it, there is no risk involved as long as you keep the private key private.

Additionally, the cert that you added to the PVE certificate storage would be interesting as well, otherwise it is hard for me to tell if there is anything wrong there. Maybe the openssl connect command can already give you the information you need to resolve this.

Or turn it around, what are the full requirements on the certs for PBS/PVE to work properly?

The same as for any other cert you would want to use for e.g. HTTPS connections. Correct SAN, root (or intermediate) certificate properly installed on the client, ...

Can it be firewall related? Ports on the LAN that needs opening?

Unlikely, since otherwise you wouldn't get this far.

When messing with this yesterday I managed to get both PVE and PBS in a very peculiar mode. They did no longer recognize PAM and had to be shut down/restarted through SSH. All I had done was changing their certs and adding different kind of storage. Strange.

That seems strange indeed, and shouldn't be caused by adding a cert to the certificate store alone.

 

[LooneyTunes]

You could always use openssl on the PVE host to try and get more debug information:

openssl s_client -connect pbs.home.arpa

Thanks, I will try that and see if it reveals anything useful

Generally, if the certificate is correctly issued & configured on the PBS (hard for me to tell with the info available), there must be some misconfiguration on your PVE. You can attach the certificate of PBS, if you want me to gloss over it, there is no risk involved as long as you keep the private key private.

I will generate new certs you can examine if you are willing, I would appriciate that a lot really. There is something odd here, it may be PBS/PVE, but perhaps most likely the certs. I then regenerate everything from root to leaf.

Additionally, the cert that you added to the PVE certificate storage would be interesting as well, otherwise it is hard for me to tell if there is anything wrong there. Maybe the openssl connect command can already give you the information you need to resolve this.

I will generate new ones for both PVE & PBS then

 

[LooneyTunes]

Just for the heck of it I will try to manually generate a set of certs for PVE & PBS, just to test if my generating scripts may be part of this mystery. Thanks for your support so far, I'll come back to this when I have investigated more

 

[LooneyTunes]

Got it working finally. Not entirely sure if both intermediate and CA was it, but created a CA for this which did the job apparently. But when mounted PBS in PVE, I directly got another error... But new thread for that I guess. Thanks for the help and suggestions!