[SOLVED] Backup failed with an error I am unsure about

[Ovidiu]

I have been backing up to PBS for a few weeks. The backup job contains 3 VMs. THe first one was backed up correctly the next 2 failed with this error:

in this code you can see the end of the backup of the first VM followed by the error.

INFO:  97% (18.6 GiB of 19.1 GiB) in  4m 57s, read: 85.3 MiB/s, write: 50.7 MiB/s
INFO:  98% (18.8 GiB of 19.1 GiB) in  5m  0s, read: 60.0 MiB/s, write: 20.0 MiB/s
INFO: 100% (19.1 GiB of 19.1 GiB) in  5m  3s, read: 113.3 MiB/s, write: 96.0 MiB/s
INFO: backup is sparse: 1004.00 MiB (5%) total zero data
INFO: backup was done incrementally, reused 142.89 GiB (89%)
INFO: transferred 19.12 GiB in 308 seconds (63.6 MiB/s)
INFO: Finished Backup of VM 100 (00:05:14)
INFO: Backup finished at 2021-02-12 01:20:15
Using encryption key from file descriptor
ERROR: Backup job failed - upload log failed: Error: KeyConfig contains wrong fingerprint FINGERPRINTA, contained key has fingerprint FINGERPRINTB

Any ideas what this means?

This is quite baffling a PVE and PBS are both running on the same host.

 

[fabian]

are the two fingerprints completely different? what happens if you do proxmox-backup-client key show <PATH TO KEYFILE> (the key file is in /etc/pve/priv/storage/STORAGEID.enc)

 

[Ovidiu]

well, the error logs looks like this:

ERROR: Backup job failed - upload log failed: Error: KeyConfig contains wrong fingerprint 90:ef:xx:xx:xx:xx:98:5b, contained key has fingerprint 9e:4f:xx:xx:xx:xx:4a:ef

proxmox-backup-client key show ...

results in showing me the fingerprint mentioned in above error log as "wrong fingerprint" => 90:ef:xx:xx:xx:xx:98:5b

Anyway, this is just my local nas, the setup is quite new, I don't rely on any of these backups so what is the easiest way to "get things back to working"

open for any pointers.

 

[fabian]

this is quite strange - did you ever modify the key file manually?

to get it back to working you can either generate a completely new key (WARNING: without the old key you can obviously not access the old encrypted backups), or remove the fingerprint part from the keyconfig. although personally I'd be wary of using that key unless you remember some benign hand-editing of the key file.. the fingerprint stored in the file and the one that is generated by the key don't match. we only ever write the fingerprint to the file on key generation or passphrase changes, and the latter doesn't change the key itself so the fingerprint must remain the same..

 

[Ovidiu]

I don't need the encrypted backups. I had a look at the GUI, can you please specify how to generate a new key and replace the old one?
I'n not 100% sure if this can be done via GUI as the help doesn't really explain these options:


I guess i need to manually delete the old useless backups?

Yes, I once restored the PBs installation and when restoring the saved key, I did not upload the key but only copy / pasted a part of it. Can't remember exactly but I think I made sure the fingerprints match the fingerprint of the PBs and then I pasted the data part of the key? Anyway, it was a test and weirdly enough, since that day, backups have been working daily for about 2 weeks at least. I'm just confused it blew up now.

Its confusing as you can see in the screenshot it shows the wrong fingerprint:

while the key file itself: /etc/pve/priv/storage/storage-id.enc actually contains the correct fingerprint. Anyway, I know it was me who messed up

 

[fabian]

well, if you don't mind starting over the easiest course is to delete the existing backups and create a completely new key.

if you want to keep your existing key and backups, you can also manually remove the fingerprint (or correct it) in the keyconfig file, and then check how far back you can access backups using the client (the show configuration button in PVE would be a quick check that works on the GUI). if you have backups that are encrypted with a different key that you no longer have, just remove them.

 

[Ovidiu]

well, if you don't mind starting over the easiest course is to delete the existing backups and create a completely new key.

if you want to keep your existing key and backups, you can also manually remove the fingerprint (or correct it) in the keyconfig file, and then check how far back you can access backups using the client (the show configuration button in PVE would be a quick check that works on the GUI). if you have backups that are encrypted with a different key that you no longer have, just remove them.

thanks. I already tried correcting the fingerprint in the .enc file but the PVE GUI still shows the old fingerprint as in my last screenshot I posted here so I'm wondering which service I could restart for the .enc file to be reread?

 

[fabian]

ah, that one is stored in /etc/pve/storage.cfg

 

[Ovidiu]

ah, that one is stored in /etc/pve/storage.cfg

ok, that was the problem.

/etc/pve/priv/storage/STORAGEID.enc contains a fingerprint of the PBS


/etc/pve/storage.cfg contains BOTH encyption-key AND fingerprint. Encryption-key is NOT as expected the STORAGEID but rather the fingerprint contained in the encryption-key.

This is all very confusing but I guess now I have corrected the fingerprint in all necessary places and am running my backups again

 

[Ovidiu]

###edit###
all this fiddling around didn't help. I will now delete all backups via PBS GUI, remove the storage from PVE and start from scratch.

 

[fabian]

no, .enc is the keyfile and contains the key fingerprint. the storage.cfg contains both the fingerprint of the key, and the fingerprint of the TLS cert (if using a self-signed one).