Performance PBS vs. Restic for 1.3TB 500k Files

[cheesecake]

hello, first of all i am a fan of pbs, especially the backups of vm's and the enormous storage space savings, as well as the much faster backups are a dream. Without pbs 8h, with pbs 15 minutes, as well as less/no impact on the operation. therefore really a strong improvement.

the idea was to use pbs also for a backup of about 1,3tb data, slowly growing, but with 500k files. the initial backup is not of interest here, but the following ones, here performed pbs unfortunately not at all.

Parameters:
Files: 500k
total size: 1,3TB
network: 10Gbit
target system: 64gb ECC, AMD Ryzen 5 3600, ZFS10 4x 16TB
No other jobs running on the PBS server during several tests. The values are average, there are no outliers up or down. On average there are 500 new files per incremental backup, as well as under 10 changed and 40 new folders.

PBS takes over 1:30 - 2:00 hours for the incremental backups.
Restic only takes 3-5 minutes for the incremental backups.

Before anyone tries this too, I think PBS is not made for this and probably has "protocol/layout" related too much overhead? for so many files.

The post is just to give some info for others, as well as possibly help the Proxmox Backup Server team.

 

[Dunuin]

How do you store those files? I would for example guess that backing up a zvol containing the files would be faster than backing up a dataset with those files. Especially as only QEMU virtual disks can make use of dirty bitmapping when doing incremental backups.

 

[cheesecake]

CephFS

 

[Dunuin]

Then it's probably because it needs to read and hash the whole 1.3TB when doing an incremental backup to see what has changed.

 

[cheesecake]

I don't really care. the contribution is also more for the info for others, the causes are currently I do not care, my solution is simply to use restic and done. restic can handle it without problems in a good, manageable performance.

 

[Jarvar]

Curious about the setup. How are you making the backups?
Is the backup making a snapshot through PBS versus backing inside the VM or container directly using Restic?
Thank you so much for sharing

 

[cheesecake]

Curious about the setup. How are you making the backups?
Is the backup making a snapshot through PBS versus backing inside the VM or container directly using Restic?
Thank you so much for sharing

Restic and PBS Client run directly on host and access the files directly trough pve default mounted cephfs under /mnt/pve/...

 

[Jarvar]

Thank you.
What I am trying to understand is. Are you backing up system files within a VM using Restic to a remote repository?
Or are you using Restic to Backup a PBS Client or tp a remote?
I'm interested because I'm using Restic to backup a PBS on site to a remote site.
On the Remote site I will then make a backup of the remote PBS to another remote Restic Repository for extra safe measures.
Perhaps it's too redundant.

 

[Chris]

Hi,
thanks for the thread, we are aware of the current limitations of file level backups, see https://bugzilla.proxmox.com/show_bug.cgi?id=3174

In contrast to Proxmox Backup Server, restic supports to skip reading and rechunking of unchanged files based on metadata, therefore speeding up the backup process by a lot. There is however work in progress to introduce a similar feature also for the proxmox backup client.

 

[David Thistlethwaite]

hello, first of all i am a fan of pbs, especially the backups of vm's and the enormous storage space savings, as well as the much faster backups are a dream. Without pbs 8h, with pbs 15 minutes, as well as less/no impact on the operation. therefore really a strong improvement.

the idea was to use pbs also for a backup of about 1,3tb data, slowly growing, but with 500k files. the initial backup is not of interest here, but the following ones, here performed pbs unfortunately not at all.

Parameters:
Files: 500k
total size: 1,3TB
network: 10Gbit
target system: 64gb ECC, AMD Ryzen 5 3600, ZFS10 4x 16TB
No other jobs running on the PBS server during several tests. The values are average, there are no outliers up or down. On average there are 500 new files per incremental backup, as well as under 10 changed and 40 new folders.

PBS takes over 1:30 - 2:00 hours for the incremental backups.
Restic only takes 3-5 minutes for the incremental backups.

Before anyone tries this too, I think PBS is not made for this and probably has "protocol/layout" related too much overhead? for so many files.

The post is just to give some info for others, as well as possibly help the Proxmox Backup Server team.

Very useful information, did you observe any significant space savings by either system, thanks again.