Archiving to external HDD and deduplication

Fathi

Well-Known Member
May 13, 2016
123
2
58
52
Tunis, Tunisia
Hi,
I have a promox ve server, backing to a proxmox backup server with all the benefits of pbs.
I would like to also backup/archive to an external hdd once a week and take that hdd off-site offline.
This VM disk is around 3.2TB of size and could go up to 5.7TB in the future without adding additional disks to pve. As I would like to have ~ 6 months of backups, this is approximately 27 snapshots, so approximately 87TB of external HDD. Biggest one I could find here is 18TB (seaget or WD which also claim their external usb hdd are compatible with windows and palle and not a single word about linux).
How could i achieve deduplication with an external HDD.
I was planing to attach this external hdd to pve. Is it possible to do deduplication in this way or shoud i mount it to pbs, add a new datastore (the external hdd) and then backup to it to achieve this ?
TIA.
 
One way to do it is to have ZFS as the backing store of PBS and doing zfs replication to the external disk. You will use the PBS-internal deduplication and may be able to store the data, yet you have to check with time if it really fits. This depends heavily on the data and no one can give you an estimate on YOUR environment.

Just also consinder: deduplication may not be what you want with a backup system: if that one copy of a block that is used in all of your VMs gets corrupt, your backup is becomes useless.
 
  • Like
Reactions: Fathi
One way to do it is to have ZFS as the backing store of PBS and doing zfs replication to the external disk. You will use the PBS-internal deduplication and may be able to store the data, yet you have to check with time if it really fits. This depends heavily on the data and no one can give you an estimate on YOUR environment.

Just also consinder: deduplication may not be what you want with a backup system: if that one copy of a block that is used in all of your VMs gets corrupt, your backup is becomes useless.
Thank you @LnxBil . Didn't think of the corrupt block on which depends several vm backups that could compromise them all. Thank you again for having pointed that out.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!