[SOLVED] Host incremental backups?

devilkin

New Member
May 11, 2020
22
3
3
Hi,

Currently exploring to use PBS to backup some hosts on the network which aren't VM's. I'm noticing that it still needs to scan all the files on the host - but does it send all the data too or only the changed blocks?
 

devilkin

New Member
May 11, 2020
22
3
3
Well, I'm seeing huge transfer out towards pbs using the following command (even repeated), so I guess i'm going something wrong them :/

/usr/bin/proxmox-backup-client backup etc.pxar:/etc home.pxar:/home

PBS_REPOSITORY and PBS_PASSWORD are set in environment
 

Dunuin

Famous Member
Jun 30, 2020
4,314
940
143
Germany
As far as I understand it the pbs client will create something like a archive that contains all the files/folders. Then this archive will be split into chunks and the hash of each chunk will be computed. Then the pbs client will ask the pbs server if there is already a chunk saved with that hash. If it already exists it will only send the metadata, if the hash doesn't exist it will send the chunk and metadata.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
6,716
1,173
164
the backup client output will contain summaries of how much deduplication happened for each archive/index.. for host backups it will always have to read all data (to create the archive stream that gets chunked), but will only send chunks which are not part of the previous snapshot.
 
  • Like
Reactions: devilkin

gurubert

Active Member
Mar 12, 2015
101
16
38
Berlin, Germany
Would it be possible to implement something like librsync does for file based backup? Having to read each file for each backup run is not possible with a 13TB filesystem.
 

Dunuin

Famous Member
Jun 30, 2020
4,314
940
143
Germany
Would it be possible to implement something like librsync does for file based backup? Having to read each file for each backup run is not possible with a 13TB filesystem.
I think that wont work. PBS needs to read all files to be able to create the hashes that are need for deduplication and to see if that chunk is already stored on the PBS datastore. With rsync it also takes a long time to compare two folders if you enable hashing for file comparison instead of edit time or size. If you don't compute the hashes again and again you can't be sure about data integrity. For that you would need to use dirty bitmapping and that is only available for VMs and not LXCs.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!