vzdump/fullbackup client side online encryption

RolandK

Renowned Member
Mar 5, 2019
978
206
88
51
hello,

i wondered if we could have client-side online encryption to have vzdump backup stored encrypted on the target datastore.

i guess this is something many people can need and which currently is only available with proxmox backup server.

i have taken a look into that , and it should probably be not too hard to be implemented.

all you need to do is passsing the backup stream via pipe trough an encryption tool.

i assume this can perhaps be done via a vzdump hook script, but i need to get into the details how to use stdin/stdout with that. not sure if it's doable and maybe somebody can help with building such or maybe that already exists ?


anyhow, i think it would be valuable if it would be available as a standard option in proxmox.


here is what i did as a proof of concept:


- add encryption bash script:


Code:
# cat /root/enc.sh
#!/bin/bash
openssl aes-256-cbc -in /dev/stdin -out - -pbkdf2 -iter 1000000 -pass file:/root/openssl.pass
- add password to /root/openssl.pass

- add enc.sh to VZDump.pm (ok, this is _really_ a little bit of hacky ) ;)
Code:
# diff -Naur //usr/share/perl5/PVE/VZDump.pm.orig //usr/share/perl5/PVE/VZDump.pm
--- //usr/share/perl5/PVE/VZDump.pm.orig    2023-03-19 10:28:41.955284343 +0100
+++ //usr/share/perl5/PVE/VZDump.pm    2023-03-19 10:23:59.157730485 +0100
@@ -757,7 +757,7 @@
         my $cpuinfo = PVE::ProcFSTools::read_cpuinfo();
         $zstd_threads = int(($cpuinfo->{cpus} + 1)/2);
     }
-    return ("zstd --rsyncable --threads=${zstd_threads}", 'zst');
+    return ("zstd --rsyncable --threads=${zstd_threads} | /root/enc.sh", 'zst');
     } else {
     die "internal error - unknown compression option '$opt_compress'";
     }
- restart pvedeamon

- run backup

and voila:
Code:
# file vzdump-qemu-104-2023_03_19-10_25_07.vma.dat
vzdump-qemu-104-2023_03_19-10_25_07.vma.zst: openssl enc'd data with salted password

decryption :
Code:
cat vzdump-qemu-104-2023_03_19-10_25_07.vma.dat | openssl aes-256-cbc -d -in - -out - -pbkdf2 -iter 1000000 -pass file:/root/openssl.pass >unencrypted
 
Last edited:
still nobody interested or no more votes for this ?

vzdump full backups still play an important role in enterprise backup, even if pbs is being used.

i always combine pbs with vzdump weekly full backups

i wonder that on-the-fly encryption for vzdump so little attention, as it's considerably large amount of effort to have your vzdumps encrypted after backup run for offsite storage - and for now you will need a lot of additional runtime or interim space for that.

that's one of my most wanted features....
 
  • Like
Reactions: heribert
mhh, this is really weird.

i have had at least talk to 3 people in the past weeks which wondered, how vzdump encryption could be activated or why it was missing at all.

two discussions were related to hetzner backup storage box, which allows cifs sharing and which is quite fast, from the tests which i have seen

veeam backup & recovery has encryption feature for at least 7-8 years.

what is the good of pbs having encryption , when vzdump doesn't have that, too ?

who wants to rely on pbs alone ?
 
  • Like
Reactions: heribert
mhh, this is really weird.

i have had at least talk to 3 people in the past weeks which wondered, how vzdump encryption could be activated or why it was missing at all.

two discussions were related to hetzner backup storage box, which allows cifs sharing and which is quite fast, from the tests which i have seen

veeam backup & recovery has encryption feature for at least 7-8 years.

what is the good of pbs having encryption , when vzdump doesn't have that, too ?

who wants to rely on pbs alone ?
I'm super confused by the lack of interest here. If we follow the 3-2-1 mode everyone keeps preaching, the logical place for the 1, in my opinion, is in the cloud which would demand encryption. I find it strange that I can encrypt with proxmox_backup_client for my hosts (including proxmox itself) but when proxmox uses proxmox backups server it can't encrypt the snapshots/dumps. Makes no sense to me.

Feel like I must be missing something, this seems like a weird oversight.
 
  • Like
Reactions: RolandK
I'm super confused by the lack of interest here. If we follow the 3-2-1 mode everyone keeps preaching, the logical place for the 1, in my opinion, is in the cloud which would demand encryption. I find it strange that I can encrypt with proxmox_backup_client for my hosts (including proxmox itself) but when proxmox uses proxmox backups server it can't encrypt the snapshots/dumps. Makes no sense to me.

Feel like I must be missing something, this seems like a weird oversight.
I think the idea is...
Fast short term backup: Encrypted local backup via PBS on SSDs.
Offsite backup: slow encrypted offsite backup via PBS using sync jobs on SSD or SSD+HDD.
Long-term/offline backup: encrypted on tapes via PBS

But I agree. Vzdump might still be useful. Wouldn't hurt to have some additional Vzdump backups in case there is a critical bug in PBS so you don't lose all of your backups.
So for me it is more the "2" of the 3-2-1 rule. But here it also would be fine to simlpy encrypt the underlaying storage if you don't need to store them offsite.
 
Last edited:
I'm super confused by the lack of interest here. If we follow the 3-2-1 mode everyone keeps preaching, the logical place for the 1, in my opinion, is in the cloud which would demand encryption. I find it strange that I can encrypt with proxmox_backup_client for my hosts (including proxmox itself) but when proxmox uses proxmox backups server it can't encrypt the snapshots/dumps. Makes no sense to me.

Feel like I must be missing something, this seems like a weird oversight.
There's just 3 people on that Bugzilla, this is always mind-boggling to me, but I am not a sociologist, why instead of having it reappear on the forum over and over, people do not go +1 themselves in Bugzilla, staff always gets an email on those. If there was +20 by now and adding up 3 per month, they would have more likely already put it up the priority list.

Why it was not done since day 1 is a valid speculation though, maybe as the tool is still name vz* it was meant to go away completely anyway?
 
I don't think they would remove it. This would be really bad in case you ever need to restore some very old backups done with vzdump.
But its understandable that they put most ressources in adding new features to PBS. At least I would allocate my ressources into the project that generates additional income on top of PVE and thats the better and more future-proof platform.
 
anyhow, for me pbs is not a replacement for vzdump full backup, if you really care about backup.

it's an efficient and fast incremental forever backup solution, but i would NEVER rely on that alone.

even with veeam, you typically won't do incremental backup forever without adding a full backup from time to time.

following the 3-2-1 rule, we will still use vzdump full backups to a remote location via NFS once a week. and there they end up unencrypted in a "server room" which isn't even protected by some secure door.

if all that work that was put into vdzump post-backup-hook solutions like https://www.google.de/search?q=vzdump+encrypt+github would have been put into vdump on the fly datastream encryption , we would be a step closer...
 
Last edited:
I don't think they would remove it. This would be really bad

I hope they would not, but it would probably deserve some explanation, like a roadmap in some of those bugzilla reports would have reassured everyone (e.g. they will be catching up).

At least I would allocate my ressources into the project that generates additional income on top of PVE

That's just normal course of doing business, but if goes down the path that everyone who wants good backups needs to move over to PBS or MacGyver it on their own ...

anyhow, for me pbs is not a replacement for vzdump full backup, if you really care about backup.

This! :)

following the 3-2-1 rule, we will still use vzdump full backups to a remote location via NFS once a week. and there they end up unencrypted in a "server room" which isn't even protected by some secure door.

It's also like ... why would one have to use PBS to get some features which should be integral to a platform live PVE. It's almost like next thing you know you will need PMG for getting notifications too? (It's just an example.)

if all that work that was put into vdzump post-backup-hook solutions like https://www.google.de/search?q=vzdump+encrypt+github would have been put into vdump on the fly datastream encryption , we would be a step closer...

They have not taken in the suggested PoCs, so ... I would just surmise they have other plans for some reason at this point.
 
At this point, I got convinced, whatever I expect from Proxmox (apart from very obvious low hanging fruits, which require a gentle nudge) - I better do myself first, then publish as a package and see where it gets me. :)

I think for that VZDump, you could just literally run your own and use dpkg-divert. I would charge extra for maintaining systems like that for 3rd parties. ;)
 
I have asked AI for that. This is the result. :eek:

Me:
shouldn't proxmox pve get an encryption feature for classical vzdump full backups? as i see, most hypervisor backup solutions have support for online encryption.

ChatGPT:
You're absolutely right—encryption for backups is a feature that many modern hypervisor backup solutions already support, and it's something that would definitely make sense for Proxmox VE (PVE) to include for vzdump full backups.

Currently, Proxmox does allow for some basic encryption features in backup processes, but the encryption is more limited in scope, and there’s no built-in option for online encryption directly via vzdump. However, there are a few things to consider here:

1. Backup Storage Encryption:
While Proxmox doesn’t provide full vzdump backup encryption out of the box, it’s possible to use encrypted storage volumes (such as LUKS or ZFS encryption) to secure the backup destination. This adds an encryption layer to the backup storage, but it’s at the storage level rather than during the backup process itself.

2. Custom Backup Encryption:
Some users have resorted to manually encrypting backup archives using tools like gpg or openssl after the backup completes. This does require additional scripting and doesn’t offer the convenience or tight integration you might expect from a native solution, but it is a workaround that many in the community use to get that encryption feature.

3. Performance Considerations:
Real-time encryption during the backup process (especially for full backups, which can be quite large) could introduce performance overhead. While this is less of an issue on modern hardware, there may be trade-offs when it comes to backup speed and resource usage. Proxmox might prioritize minimizing backup time and system load, which is why the feature isn’t implemented yet.

4. Future Plans:
Proxmox is known for adding features and responding to community feedback. It’s possible they might integrate full online encryption into vzdump in a future release, given the growing demand for security and encryption features in backup solutions. If you’re interested in this feature, it’s worth checking out the Proxmox forums and suggesting it as a feature request, as they actively consider community input.

Until then, the best approach would be either using storage-level encryption or integrating a manual encryption process via scripts, as mentioned above.

Do you have any specific use cases or security concerns that are prompting you to look for this feature in particular?

Me:
yes, i have need for that, as the place where our backup server resides is not perfectly secure.

ChatGPT:
That makes a lot of sense, and it’s definitely a concern that a lot of organizations face—backup security is often overlooked, but it’s crucial when the backup server is in a less secure location. Encrypting backups directly is a great way to mitigate risks, especially if physical security is a concern.

In your situation, the lack of native encryption in Proxmox’s vzdump backups could leave your data vulnerable in case the backup server itself is compromised or accessed by unauthorized individuals.
Here’s what you can do to improve security:

Encrypted Storage for Backups: You can encrypt the storage that holds the backup data itself. For example, using LUKS (Linux Unified Key Setup) to create an encrypted disk volume on your backup server ensures that, even if someone gains access to the server, they can’t easily access the backup files without the encryption key. This is a good first line of defense.

Manual Post-Backup Encryption: You can implement a script to automatically encrypt your backups after they’re created by vzdump. For example, using gpg or openssl to encrypt the backup files as soon as they’re finished can ensure that they remain protected.

Here’s an example of what a simple post-backup encryption script might look like:

#!/bin/bash
BACKUP_DIR="/path/to/backup"
ENCRYPTED_DIR="/path/to/encrypted/backup"
BACKUP_FILE="backup_name.vma"
PASSPHRASE="your_secret_passphrase"

# Encrypt the backup using GPG
gpg --batch --yes --passphrase "$PASSPHRASE" -c "$BACKUP_DIR/$BACKUP_FILE"

# Move encrypted file to the secure location
mv "$BACKUP_DIR/$BACKUP_FILE.gpg" "$ENCRYPTED_DIR"

# Optionally remove the unencrypted file after encryption
rm -f "$BACKUP_DIR/$BACKUP_FILE"

This script would take your backup, encrypt it with a passphrase, and move it to a secure location. You can set it up to run automatically after vzdump completes. It's not quite as convenient as having built-in support for encryption, but it does get the job done.

Offsite or Cloud Backup: If the local backup server is physically vulnerable, another option is to push the backups to an offsite or cloud storage solution that supports encryption. Many cloud providers (like AWS S3, Google Cloud, or Backblaze B2) support server-side encryption, so your backups are encrypted at rest. You could automate this with something like rclone to push your backups to cloud storage securely.

Backup with Encryption via SSH Tunnel: If you're backing up to a remote server over SSH, you could use an encrypted tunnel to protect the backup during transit. Tools like rsync or scp can work well for this. To add an extra layer, you can also apply encryption on the backup before transferring it.

Monitor Access: Make sure you are logging and monitoring any access to the backup server. If someone does try to access the server, logs will give you a trail to follow. Proxmox provides logs of backup tasks, and you can combine this with system-level logging to ensure any unauthorized access attempts are flagged.

Request for Built-in Encryption:
Given the importance of data security and the growing demand for features like this, you could also bring it up as a feature request on the Proxmox forums. Proxmox is very responsive to the community and often implements features based on user feedback, especially when it comes to critical security improvements.
Conclusion:

While waiting for a potential native feature from Proxmox, using manual encryption (via gpg, openssl, or cloud backup solutions) would be the best way to secure your backups. Encrypted storage and offsite backups would also be excellent complementary strategies to ensure your data is protected in case of physical compromise.

If you need help setting up any of these solutions or need a more specific script, feel free to ask!
 
As a workaround ChatGPTs proposal of cloud backup ( with one of the available dedicated Tools with encryption support) is actually the best approach at the moment imho. I too would like a native function so I don't need the additional complexity of restic for them