Compression / Encryption and De-duplication

[sukerman]

Before I discovered this, I ran containers with the data on a separate volume so I could exclude the data volume from normal backup and use restic to backup the data volume. e.g, you have a mysql container which is a few gigs, but 1Tb of mysql data, so you put the data it on a separate volume and use restic to back it up so it only backs up the bits that have changed, because the normal backup ( I believe because the vm dump wrote things 'out of order' ) was always completely different despite very little changing in the actual vm). Otherwise, this meant if I used restic to incrementally backup my vm dumps every night it was sending 1Tb each time. Restic also did not like compression because this also appeared to write a completely different file even though nothing changed.

So my question is, in the backup setup in Proxmox VM, should I set compression to 'none'?. If the Proxmox datastore is on a ZFS volume, should I disable compression there?

As for Encryption, it says the in setup "Auto-generate a client encryption key, safed privately on cluster filesystem". I am syncing the backups to another server in case of failure, but I'm not running a cluster, if this node suffers a total failure, where is the key I need? is it the 'fingerprint'?

Thank for anyone who can make things clearer for me,

Also - this having the prune options, forget, keep 1 week etc looks a lot like restic to me, is it based on it? If so what's the performance like on large prunes. I've found with restic, once I backup my 1Tb db for a few weeks the prune becomes unfeasible as it takes more than a week.

Jack

 

[fabian]

Before I discovered this, I ran containers with the data on a separate volume so I could exclude the data volume from normal backup and use restic to backup the data volume. e.g, you have a mysql container which is a few gigs, but 1Tb of mysql data, so you put the data it on a separate volume and use restic to back it up so it only backs up the bits that have changed, because the normal backup ( I believe because the vm dump wrote things 'out of order' ) was always completely different despite very little changing in the actual vm). Otherwise, this meant if I used restic to incrementally backup my vm dumps every night it was sending 1Tb each time. Restic also did not like compression because this also appeared to write a completely different file even though nothing changed.

So my question is, in the backup setup in Proxmox VM, should I set compression to 'none'?. If the Proxmox datastore is on a ZFS volume, should I disable compression there?

the setting on the PVE side has no effect whatsoever when backing up to PBS (and is in fact not available in current dev versions for that reason ) you can leave compression enabled on the ZFS side, but it will probably not find much to compress (you can verify after a while with zfs get compressratio DATASET

As for Encryption, it says the in setup "Auto-generate a client encryption key, safed privately on cluster filesystem". I am syncing the backups to another server in case of failure, but I'm not running a cluster, if this node suffers a total failure, where is the key I need? is it the 'fingerprint'?

it will be in /etc/pve/priv/storage/STORAGENAME.enc , and yes, you absolutely need to save that somewhere in case your PVE node(s) fail, otherwise you can't access your encrypted backups. this will be made more obvious with an option to download the key when it gets generated.

Also - this having the prune options, forget, keep 1 week etc looks a lot like restic to me, is it based on it? If so what's the performance like on large prunes. I've found with restic, once I backup my 1Tb db for a few weeks the prune becomes unfeasible as it takes more than a week.

pruning only deletes the metadata associated with the pruned snapshots, so it's very fast. the actual data/chunk deletion happens when you run garbage collection. GC is limited by the metadata access and file deletion speed of your chunk/data store.

 

[sukerman]

Thanks that answers nearly everything for me, I'll have to see what the GC performance is like over time. RE: Encryption, I did find that confusing that it showed the fingerprint of the server, had the option to encrypt but I couldn't see the key, an option to download it will be great and having the fingerprint here as well as on the dashboard seems confusing as its not related to the backup.

Thanks again.

 

[fabian]

the fingerprint is needed as trust-anchor when using self-signed certificates, that's why it is prominently displayed on both sides.