Backup verification takes 4-5 days to complete

[jsengupta]

Hi,

we are taking Proxmox VM backup by Proxmox backup server. There are 10 large VMs having around 2TB of storage on each of the VMs. We do not have any issue backing up the VMs. However, the verification job takes around 4-5 days to complete and create high IO wait. sometimes even the verification job does not complete.

Is it necessary to verify the backup? Can we not keep the backup without verified? if not, what is the workaround?

Thanks in advanced.

 

[dietmar]

There are two setting to optimize that:

- Skip verified snapshots: Only verify new snapshots
- Re-verify after X days: But reverify after X days

 

[jsengupta]

The setting is there already. The query i have asked is after applying the settings you have mentioned.

Does it really matter if we verify backup?

 

[dietmar]

Does it really matter if we verify backup?

That is up to you. But we always suggest to verify backups, and do regular restore tests.

 

[jsengupta]

How can we do faster verification? If we create ZFS with journal drive having SSD and the data drive having the HDD, will the verification jobs be faster?

Has anybody tested this way?

 

[Dunuin]

No, using SSDs for caching metadata will only help with prune and GC jobs. Only way to get good verify speeds is to switch all HDDs to SSDs, as all the data has to be read and hashed again. And if your got 20TB of storage that would mean at a full re-verify the HDDs need to hash atleast 5 million small files and HDDs aren't good at that. Especially because the reads aren't that sequential because of the deduplication.

 

[BriBriBri]

Is there no role for a CPU's sha exenstions (or lack thereof) in determining what the verification performance could be?

I was hopeful that a CPU+motherboard upgrade to our pbs server might improve verification job duration. But since we're HDD-based (rather than SSD), is the consensus that the HDD bottleneck would render acceleration via sha extension moot?

ref: https://en.wikipedia.org/wiki/Intel_SHA_extensions