[SOLVED] Verify a backup - when do that PBS?

[fireon]

Hello all,

on all my backups are a questionmark on the "verify state" - "? None". What does that mean? Ok. So i started the verify process of the backuppool. It runs since 5h. And on this column i get now an nice green "all ok". Does this mean that the backup is not checked for consistency default? Or how does this mean?

In the docs i found only this:

The Manifest: index.json
The manifest contains the list of all backup files, their sizes and checksums. It is used to verify the consistency of a backup.

Very Thanks

 

[fabian]

there is no automatic/scheduled verification except for things like restore operations. if you want regular verification of your whole datastore, you can set up a cron job/systemd timer. note that verification like that is quite disk and CPU intensive (you need to read + hash each chunk in the datastore!) and can take quite a while

 

[dcsapak]

btw. scheduled verification is planned to be integrated (like sync-jobs already are) not timeframe yet though

 

[Jarvar]

Sounds like a great plan. I noticed when upgrading from 8-11 Beta to 8-14 the disappearance of individual VM backup verification.
Basically you can verify all the backups in the store, but not individual ones like I used to. Do I have to run the whole verification process again?
Or is there a command to complete and inidividual one?

It seems like the commands is proxmox-backup-manager verify store3

However, following the syntax for example proxmox-backup-manager verify store3:vm/107/2020-09-06T01:36:01Z
does not work.
I have tried proxmox-backup-manager verify group store3:vm/100 to atleast focus on the backups of a single VM instead of all of them.
Still doesn't work.

any ideas?
thanks.

 

[dcsapak]

I noticed when upgrading from 8-11 Beta to 8-14 the disappearance of individual VM backup verification.

we just moved it in the gui to inline actions, verification is the magnifying glass (not optimal, we plan to replace that with something more sensible, but font-awesome 4.7 did not have anything better...)

 

[Jarvar]

@dcsapak
Perfect!
I tried clicking the magnifying class before and it didn't show up. Maybe I had some other tasks going.
Thank you for your guidance.!

 

[tlex]

there is no automatic/scheduled verification except for things like restore operations. if you want regular verification of your whole datastore, you can set up a cron job/systemd timer. note that verification like that is quite disk and CPU intensive (you need to read + hash each chunk in the datastore!) and can take quite a while

I know this thread is quite old but since some of my backups are stored on ssd drives I was concerned about this comment regarding how intensive on disks the verification process can be. Does this mean that verification involves a lot of writting to the disks ? (Not sure to understand the "+hash each chunk in the datastore" portion of your comment) or is it just reading which I should be fine when using ssd drives ? Thanks

 

[Dunuin]

A verify should read all data on your SSD once, calculate the hashes and then update the metadata of each chunk. So it should write something, but it shouldn't be that much. It primarily reads a lot.

 

[Capo Ferro]

What is the reason "Verify new snapshots" in options isn't set as default on datastore's options? And if it's enabled, how does it interact with the "Skip verified" option on the backup verify jobs?

 

[Dunuin]

What is the reason "Verify new snapshots" in options isn't set as default on datastore's options?

My guess would be that you want the backups to finish as fast as possible, so the guests are slowed down or even offline as short as possible. If you verify each backup snapshot directly after the backup, the verify would probably slow down the remaining backups of that backup job?

 

[fireon]

In the end, you use a special device with ZFS, which speeds up our customers' systems by a factor of 10. Really incredible. Highly recommended.
The best way is of course to equip the backup server with "only SSD's". We already run some of these servers. This is really fun.

 

[fabian]

yes, like @Dunuin says - verification is rather expensive (both I/O and CPU wise). if you verify in an async fashion (e.g., once every week) you reduce the load by only verifying a chunk that is potentially used in multiple snapshots once.

 

[Capo Ferro]

@fabian so verifying backups verifies all the chunks for each backup even if `skip_verified` is enabled? So what does that option do then?

 

[Dunuin]

@fabian so verifying backups verifies all the chunks for each backup even if `skip_verified` is enabled? So what does that option do then?

No it will skip those chunks unless you define a "re-verify after X days".

 

[fabian]

whether to verify or not is always decided on the snapshot level (based on skip and reverify-after), not on the chunk level. if a snapshot is verified (again), all its chunks are verified, unless they have already been verified in the same verification task (e.g., when verifying a group/namespace/datastore).

 

[Capo Ferro]

Thank you. So just to clarify, activating both "Verify new snapshots" and "Skip verified" will verify a new snapshot and all the chunks it would take to restore it?

 

[Dunuin]

Thank you. So just to clarify, activating both "Verify new snapshots" and "Skip verified" will verify a new snapshot and all the chunks it would take to restore it?

Yes. Then you now that the backup snapshot was initially healthy.
To be sure it stays healty (bad disks with failed sectors, silent bit rot and whatever) you neen to run a re-verify regularily.

 

[fabian]

the two settings are orthogonal
- "verify new snapshots" controls whether an initial verification right after creating the snapshot is done
- "skip verified" controls the behaviour of a verification job (or manual verification task)

since a new snapshot can by definition not be verified yet, the latter does not affect the former. unless you enable "reverification", enabling both options means the snapshot will only be verified once, right when creating it.

 

[Capo Ferro]

Thank you both for the explanation. That's basically the behaviour that I want: verify each snapshot as it arrives only once. Backups are stored on a ZFS dataset which already gives me a measure of protection against corruption.