[SOLVED] Backup verification

rian81

rian81

Member
Proxmox Subscriber
Jun 20, 2023
64
3
13
Hi,

I want to ask, is the backup verification step important, because it takes a long time to do the verification.
If verification step pass, is there an effect on backup and restore?...

Thanks
 
the verification is intended to detect corruption in chunk files and missing chunks, so basically errors in the storage layer (or mishandling by an admin...)
so if you don't do that, you trust that the storage does never corrupt and the files are all written correcltly

when actually restoring backups, errors or missing files will prevent restoring, so doing it regularly makes sure everything is ok

note that some storages (e.g. zfs) already have built in checksums and with regular maintenance taks ( e.g. scrubs for zfs) that is already done on the storage layer and the gain from doing the verification in pbs is rather slim
(still can detect some issues, e.g. deleted files, etc.)

if the files are all ok, having done verification or not has no impact on restoring or backing up
 
Hi,

Thank you very much for clarification.

I need some more : suppose a chunk belongs to 50 VMs. And a verification task begins (datastore or namespace level).

How many times will this chunk be verified ?

Christophe.
 
Based on my experience :

1. When verification scheduler run, it will sequential verification job and will take so long to do all vm verification. The CPU resource consume was rather low.
2. When you set verify new snapshot, it will paralel verification job based on new snapshot backup. But it will consume highest CPU resource (at my pbs until 95-100%)

Note : I have 4 pve cluster (16 node), 1 pbs, and about 200 VMs
 
Last edited:
in a single verification job, a chunk will only be verified one time

of course if you have multiple verification jobs that include the same chunks, those will be verified multiple times, but only once per job
 
Hello

I have another question which comes into play with the new S3 tech preview in v4. With multiple backups in a group - let's say 15 backups of the same VM. A local backup of the VM would take up around 12GB.

During the verification of each of the backups. Are the 12GB downloaded every time or is only the diff downloaded between the backups? This is quite an inportant information to get a better idea of how much egress traffic will be produced on the S3 backend which costs money.
 
Hi,
hb9hil said:
Hello

I have another question which comes into play with the new S3 tech preview in v4. With multiple backups in a group - let's say 15 backups of the same VM. A local backup of the VM would take up around 12GB.

During the verification of each of the backups. Are the 12GB downloaded every time or is only the diff downloaded between the backups? This is quite an inportant information to get a better idea of how much egress traffic will be produced on the S3 backend which costs money.
Click to expand...
the verification with S3 backend does currently always fetch the chunks from the backend, the local store cache is always bypassed for this. There are plans to make this more lightweight by introducing a mode where only the existence of the chunk has to be checked, relying on the storage provider to guarantee the chunk payload data integrity, but this is not implemented yet. See https://bugzilla.proxmox.com/show_bug.cgi?id=4594

So yes, currently all the chunks needing verification are being fetched unconditionally.
 
Last edited:
Chris said:
So yes, currently all the chunks needing verification are being fetched unconditionally.
Click to expand...
Oh okay wow, I didn't expected this.

So for a VM which would eat up around 12GB in a local backup and with 15 Backups in the PBS I would totally download

15 x 12GB = 180GB Data from the S3 backend to verify all 15 Backups.

I had in mind that only the delta chunks will be verified. But I guess I was wrong
 
hb9hil said:
Oh okay wow, I didn't expected this.

So for a VM which would eat up around 12GB in a local backup and with 15 Backups in the PBS I would totally download

15 x 12GB = 180GB Data from the S3 backend to verify all 15 Backups.

I had in mind that only the delta chunks will be verified. But I guess I was wrong
Click to expand...
No, verification is smarter than this. Already verified chunks are not re-verified for the same verification job. So if you have 12 GB worth of chunk, 12 GB will be fetched. Duplicate chunks are not fetched twice, but each chunk at least once.

Otoh, if you do 15 verifies of each snaptshot individually, e.g. by doing a verify after backup, then yes these wii add up as there is no way of knowing which has already been verified before.

Note, verification is probably one of the most expensive operations since you have a lot of get request and egress costs.
 
Last edited:
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom