vzdump/fullbackup client side online encryption

[RolandK]

hello,

i wondered if we could have client-side online encryption to have vzdump backup stored encrypted on the target datastore.

i guess this is something many people can need and which currently is only available with proxmox backup server.

i have taken a look into that , and it should probably be not too hard to be implemented.

all you need to do is passsing the backup stream via pipe trough an encryption tool.

i assume this can perhaps be done via a vzdump hook script, but i need to get into the details how to use stdin/stdout with that. not sure if it's doable and maybe somebody can help with building such or maybe that already exists ?


anyhow, i think it would be valuable if it would be available as a standard option in proxmox.


here is what i did as a proof of concept:


- add encryption bash script:

# cat /root/enc.sh
#!/bin/bash
openssl aes-256-cbc -in /dev/stdin -out - -pbkdf2 -iter 1000000 -pass file:/root/openssl.pass

- add password to /root/openssl.pass

- add enc.sh to VZDump.pm (ok, this is _really_ a little bit of hacky )

# diff -Naur //usr/share/perl5/PVE/VZDump.pm.orig //usr/share/perl5/PVE/VZDump.pm
--- //usr/share/perl5/PVE/VZDump.pm.orig    2023-03-19 10:28:41.955284343 +0100
+++ //usr/share/perl5/PVE/VZDump.pm    2023-03-19 10:23:59.157730485 +0100
@@ -757,7 +757,7 @@
         my $cpuinfo = PVE::ProcFSTools::read_cpuinfo();
         $zstd_threads = int(($cpuinfo->{cpus} + 1)/2);
     }
-    return ("zstd --rsyncable --threads=${zstd_threads}", 'zst');
+    return ("zstd --rsyncable --threads=${zstd_threads} | /root/enc.sh", 'zst');
     } else {
     die "internal error - unknown compression option '$opt_compress'";
     }

- restart pvedeamon

- run backup

and voila:

# file vzdump-qemu-104-2023_03_19-10_25_07.vma.dat
vzdump-qemu-104-2023_03_19-10_25_07.vma.zst: openssl enc'd data with salted password

decryption :

cat vzdump-qemu-104-2023_03_19-10_25_07.vma.dat | openssl aes-256-cbc -d -in - -out - -pbkdf2 -iter 1000000 -pass file:/root/openssl.pass >unencrypted

 

[RolandK]

nobody interested in encrypted vzdump full backups ?

adding two links for reference:



Bug 1560 - add possibility for root to specify an arbitrary compressor for backups

https://bugzilla.proxmox.com/show_bug.cgi?id=1560


Bug 4602 - RFE: add client-side online encryption for full-backups

https://bugzilla.proxmox.com/show_bug.cgi?id=4602

 

[RolandK]

still nobody interested or no more votes for this ?

vzdump full backups still play an important role in enterprise backup, even if pbs is being used.

i always combine pbs with vzdump weekly full backups

i wonder that on-the-fly encryption for vzdump so little attention, as it's considerably large amount of effort to have your vzdumps encrypted after backup run for offsite storage - and for now you will need a lot of additional runtime or interim space for that.

that's one of my most wanted features....

 

[RolandK]

mhh, this is really weird.

i have had at least talk to 3 people in the past weeks which wondered, how vzdump encryption could be activated or why it was missing at all.

two discussions were related to hetzner backup storage box, which allows cifs sharing and which is quite fast, from the tests which i have seen

veeam backup & recovery has encryption feature for at least 7-8 years.

what is the good of pbs having encryption , when vzdump doesn't have that, too ?

who wants to rely on pbs alone ?

 

[fmgoodman]

mhh, this is really weird.

i have had at least talk to 3 people in the past weeks which wondered, how vzdump encryption could be activated or why it was missing at all.

two discussions were related to hetzner backup storage box, which allows cifs sharing and which is quite fast, from the tests which i have seen

veeam backup & recovery has encryption feature for at least 7-8 years.

what is the good of pbs having encryption , when vzdump doesn't have that, too ?

who wants to rely on pbs alone ?

I'm super confused by the lack of interest here. If we follow the 3-2-1 mode everyone keeps preaching, the logical place for the 1, in my opinion, is in the cloud which would demand encryption. I find it strange that I can encrypt with proxmox_backup_client for my hosts (including proxmox itself) but when proxmox uses proxmox backups server it can't encrypt the snapshots/dumps. Makes no sense to me.

Feel like I must be missing something, this seems like a weird oversight.

 

[Dunuin]

I'm super confused by the lack of interest here. If we follow the 3-2-1 mode everyone keeps preaching, the logical place for the 1, in my opinion, is in the cloud which would demand encryption. I find it strange that I can encrypt with proxmox_backup_client for my hosts (including proxmox itself) but when proxmox uses proxmox backups server it can't encrypt the snapshots/dumps. Makes no sense to me.

Feel like I must be missing something, this seems like a weird oversight.

I think the idea is...
Fast short term backup: Encrypted local backup via PBS on SSDs.
Offsite backup: slow encrypted offsite backup via PBS using sync jobs on SSD or SSD+HDD.
Long-term/offline backup: encrypted on tapes via PBS

But I agree. Vzdump might still be useful. Wouldn't hurt to have some additional Vzdump backups in case there is a critical bug in PBS so you don't lose all of your backups.
So for me it is more the "2" of the 3-2-1 rule. But here it also would be fine to simlpy encrypt the underlaying storage if you don't need to store them offsite.

 

[esi_y]

I'm super confused by the lack of interest here. If we follow the 3-2-1 mode everyone keeps preaching, the logical place for the 1, in my opinion, is in the cloud which would demand encryption. I find it strange that I can encrypt with proxmox_backup_client for my hosts (including proxmox itself) but when proxmox uses proxmox backups server it can't encrypt the snapshots/dumps. Makes no sense to me.

Feel like I must be missing something, this seems like a weird oversight.

There's just 3 people on that Bugzilla, this is always mind-boggling to me, but I am not a sociologist, why instead of having it reappear on the forum over and over, people do not go +1 themselves in Bugzilla, staff always gets an email on those. If there was +20 by now and adding up 3 per month, they would have more likely already put it up the priority list.

Why it was not done since day 1 is a valid speculation though, maybe as the tool is still name vz* it was meant to go away completely anyway?

 

[Dunuin]

I don't think they would remove it. This would be really bad in case you ever need to restore some very old backups done with vzdump.
But its understandable that they put most ressources in adding new features to PBS. At least I would allocate my ressources into the project that generates additional income on top of PVE and thats the better and more future-proof platform.

 

[RolandK]

anyhow, for me pbs is not a replacement for vzdump full backup, if you really care about backup.

it's an efficient and fast incremental forever backup solution, but i would NEVER rely on that alone.

even with veeam, you typically won't do incremental backup forever without adding a full backup from time to time.

following the 3-2-1 rule, we will still use vzdump full backups to a remote location via NFS once a week. and there they end up unencrypted in a "server room" which isn't even protected by some secure door.

if all that work that was put into vdzump post-backup-hook solutions like https://www.google.de/search?q=vzdump+encrypt+github would have been put into vdump on the fly datastream encryption , we would be a step closer...

 

[esi_y]

I don't think they would remove it. This would be really bad

I hope they would not, but it would probably deserve some explanation, like a roadmap in some of those bugzilla reports would have reassured everyone (e.g. they will be catching up).

At least I would allocate my ressources into the project that generates additional income on top of PVE

That's just normal course of doing business, but if goes down the path that everyone who wants good backups needs to move over to PBS or MacGyver it on their own ...

anyhow, for me pbs is not a replacement for vzdump full backup, if you really care about backup.

This!

following the 3-2-1 rule, we will still use vzdump full backups to a remote location via NFS once a week. and there they end up unencrypted in a "server room" which isn't even protected by some secure door.

It's also like ... why would one have to use PBS to get some features which should be integral to a platform live PVE. It's almost like next thing you know you will need PMG for getting notifications too? (It's just an example.)

if all that work that was put into vdzump post-backup-hook solutions like https://www.google.de/search?q=vzdump+encrypt+github would have been put into vdump on the fly datastream encryption , we would be a step closer...

They have not taken in the suggested PoCs, so ... I would just surmise they have other plans for some reason at this point.

 

[RolandK]

still no vzdump backup encryption. not even on the roadmap.

that's sad.

 

[esi_y]

At this point, I got convinced, whatever I expect from Proxmox (apart from very obvious low hanging fruits, which require a gentle nudge) - I better do myself first, then publish as a package and see where it gets me.

I think for that VZDump, you could just literally run your own and use dpkg-divert. I would charge extra for maintaining systems like that for 3rd parties.