[SOLVED] Consolidate 2 Encrypted Datastores into a new Datastore

[AllanM]

I have created a bit of a mess for myself and looking for a way forward that makes sense.

At one time, I had a single datastore on our PBS server and a weekly backup schedule.

I wanted to move some VM's to a daily backup schedule, and for some reason I created a separate datastore on different storage for the daily backups. The contents of these existing datastores are encrypted with different keys. I don't know why I did this but the simple answer is that I should not have done it this way, now I'm trying to clean up my mess. All I needed was 2 separate backup schedules. Looking for any guidance or ideas on how to fix this.

I would like to create a new, much larger datastore, and consolidate (decrypt and re-encrypt) the contents of these 2 existing datastores and newly created backups going forward all under a single encryption key.

Feasible?

I was reading in the manual about a master key solution, but I'm not sure if this would work for existing keys.

Thanks for any help!
-Eric

 

[fabian]

I would do one of the following two options, given storage A with key X and storage B with key Y (which one is which is up to you):
- keep storage A with key X, pull storage B into storage A, destroy storage B but keep a backup of key Y until all snapshots using it in storage A are no longer around
- keep storage A with key X, and either keep storage B as archive until you no longer need it, or delete it now

both variants would use only storage A with key X for new backups going forward.

the first variant has the downside that deduplication doesn't work across encryption keys, so you will have roughly the same usage as when keeping storage B around as archive.

in either case, you could think about enabling master key support so that the next time you want to rotate encryption keys (or add new clients with different encryption keys) you only have to safeguard the master key. enabling the master key has no retro-active effect though, so existing backups are unchanged.

consolidating/re-encrypting is not implemented, although it would be possible technically (you could do it manually - restore one snapshot, back its data up again using the new key to the new storage with --backup-time set, repeat for all snapshots in chronological order starting with the oldest - but the whole process would take long and be error-prone and require space..).

 

[AllanM]

Thanks very much fabian!

That gives me a path forward. I'm basically going to do option 2 there, and choose to archive one that I can delete most of the backups in as soon as a new backup is made in "A"

In a few months, when we have a good recent history established in the new datastore, I'll probably just go snag a few of the oldest and manually do that recovery/re-backup operation to consolidate it all later on.

 

[Elbandi]

Hi,

I was in same situation: i had an unecrypted vm backup (~1T), and i needed to switch encrypted backup. The only was to "download" the vm images, and create a new backup (i use different datastore) from that images. I wrote a some tools for that (i like golang, sorry):
https://github.com/Elbandi/proxmox-backup-tools

• dump-hash: dump the chunk hash names for a fidx/didx file
• checksum: calculates the vm image sha256 checksum (same result if you use: proxmox-backup-client restore vm/198/2023-05-06T12:40:48Z drive-virtio0.img - |sha256sum)
• reencrypt: start restore for a vm, and immediately create a new backup. *

I have to add some new functions: https://github.com/Elbandi/proxmox-backup-qemu/commits/master
* there are some limitations: only qemu-server config and fidx image can be reencoded

I dont know rust yet, so i just hacky the func to rust, if the devs made it better and implement other missing functions (didx+logs+notes download/upload), i can upgrade the tool.