SSL issue when using ACME certificate

[Javier Vilarroig]

Hi.

I have found that every time the SSL ACME certificate, from Let's Encrypt, is renewed, all my scripts for automated backup and restore start to fail.

The error message is

certificate validation failed - Certificate fingerprint was not confirmed.

If I run the command manually I get a prompt for accepting the signature change:

Are you sure you want to continue connecting? (y/n)

After that, my scripts work again, until the next certificate change.

Is there a way to avoid that behavior? I mean, to instruct proxmox-backup-client to ignore a change in the signature?

Thanks in advance.
Javier Vilarroig

 

[B.Otto]

Hello,

did you add the Proxmox Backup Server via its fingerprint? I don't use the PBS command-line tool often, but this looks like the same phenomenom as in the case of Backups of a PVE-Cluster on a PBS storage.

That fingerprint depends on the certificate and changes every time the certificate changes. So the verification via fingerprint is only useful if you are using a self-signed certificate.

There are two ways the PBS can verify that it is the right server:

• Fingerprint:
  • No need for valid DNS records, Client only needs the IP-address and fingerprint of the PBS
  • Self-signed certificate is enough on PB
  • BUT Fingerprint will change with new certificates on PBS
• Certificate
  • Client needs to be able to resolve a valid DNS record, like for example pbs.example.com
  • PBS needs to have a valid certificate for pbs.example.com
  • Renewal of the PBS certificate will not cause errors (if the new certificate is valid for pbs.example.com)

You seem to have a valid certificate, so your scripts should be able to connect to your PBS via DNS record, and you can leave the fingerprint empty.

Kind regards,
Benedikt

 

[Javier Vilarroig]

Hi.

Thanks for your answer.

I understand you answer, makes a lot of sense.

In my case, both systems (server and client) are having valid LE certificates.

I don't know how to change from fingerprint to certificate validation. I have found nothing in the documentation.

Can you point me on where I ca find that information?

Thanks a lot!!
Javier Vilarroig

 

[aaron]

all my scripts for automated backup and restore start to fail.

How do they work? By using the Proxmox VE API?

How is the storage to the PBS configured in PVE? If you have a fingerprint in there, remove it. It is needed for self signed certs, but once you have LE certs, they should be trusted and therefore, providing a fingerprint to verify against is not needed, and with fast changing certs, even problematic.

 

[Javier Vilarroig]

Hi Aaron.

Sorry. I realized that I omitted one piece of information.

That backups are taken using proxmox-backup-client in bare metal servers.

We still have some of those waiting for migration to LXC.

Backups work like a charm except for issue with the fingerprint.

Cheers,
Javier Vilarroig

 

[aaron]

That backups are taken using proxmox-backup-client in bare metal servers.

Okay, if they are somewhat up-to-date, they should trust the LE certificates out of the box. Try to run it without the fingerprint.

 

[Javier Vilarroig]

Okay, if they are somewhat up-to-date, they should trust the LE certificates out of the box. Try to run it without the fingerprint.

I already upgraded them from the repos, just in case.

How can I run them without the fingerprint?

 

[aaron]

Well, not knowing your scripts, but somewhere they need to define all the needed data to access the PBS. Either a environment variables or as parameters when calling proxmox-backup-client. Removing/unsetting them for a test should work.

ENV variables might also be set by other means.

 

[Javier Vilarroig]

This is the command line I launch for a restore:

PBS_PASSWORD='${PASSWORD}' proxmox-backup-client restore ${host} --repository ${USERNAME}@${Pbs_Server}:${Pbs_DataStore} --ignore-ownership true ${snapshot} ${destination}

I think variable names are clear, but I can clarify

 

[Dunuin]

Somwhere (user home folder or directly inside the script) should be the ENV variable "PBS_FINGERPRINT". Just keep it empty: PBS_FINGERPRINT="".

https://pbs.proxmox.com/docs/backup-client.html#environment-variables:

PBS_FINGERPRINT
When set, this value is used to verify the server certificate (only used if the system CA certificates cannot validate the certificate).

 

[Javier Vilarroig]

Hi Dunuin.

Thanks for the hint.

PBS_FINGERPRINT is for sure not in the environment.

When I try to set it to empty value it fails with

WARNING: certificate fingerprint does not match expected fingerprint!
expected:
fingerprint: XXX:XXX......

So, it seems that is not disabling the fingerprint check but jest expecting an empty fingerprint and thus failing because it's not empty

I have started to look at pbs source code, but rust is not a language I master.

Thanks

 

[Dunuin]

Forgot, that you need to export that fingerprint. Here, in a bash script, I use this with the proxmox backup client to backup the host configs to a remote PBS with valid but annually changing SSL certificate:

PBS_FINGERPRINT=""
export PBS_FINGERPRINT

 

[Javier Vilarroig]

Hi Duniun.

It's not working for me

If I set PBS_FINGERPRINT to empty proxmox-backup-client is expecting to get empty as the fingerprint and also failing.

For sure it's included in the command environment. To confirm that, I give it a value, like "ignored" and then I get this:

WARNING: certificate fingerprint does not match expected fingerprint!
expected: ignored
fingerprint: XXX:XXX......

What I don't understand is why proxmox-backup client is using the fingerprint approach when there's a good quality SSL certificate available.

Thanks

 

[fabian]

you probably passed it in in the past, and now it is stored in $HOME/.config/proxmox-backup/fingerprints. delete the entry from that file (or the whole file), and it should not bother you anymore as long as you don't pass in a fingerprint via the CLI arguments or environment.

 

[Javier Vilarroig]

Hi Fabian.

That file is not existing.

 

[fabian]

did you check for the correct user? that file or an explicitly passed in fingerprint are the sources for the expected value.. one or the other must be "active"..

 

[Javier Vilarroig]

Yes, I checked in the right user. I even run a find to look for any finger* file in the full system.

I'm going to move to a different investigation line. I will deploy a new container from scratch so see if I can replicate the issue. Maybe there's something wrong in the already existing servers.

I will have you posted.

Thanks a lot to everyone for your help!!!

 

[B.Otto]

Hello,

I feel like the PBS_FINGERPRINT does get set somewhere in the process of your backup, so I think you gotta tell us more about how the backup is made. There are a lot of different ways to execute things and they all have different environments and variables, so sometimes it is hard to keep track of which variables are set....

In order to back up stuff, are you just logging into the server and executing your posted command? Or is this command part of a script? Is there a cronjob executing it?

PBS_PASSWORD='${PASSWORD}' proxmox-backup-client restore ${host} --repository ${USERNAME}@${Pbs_Server}:${Pbs_DataStore} --ignore-ownership true ${snapshot} ${destination}

Did you replace these Variables in order to mask the critical information, or is this command just as-is? In that case these variables have to be set somewhere..

Kind regards,
Benedikt