Verify Jobs, GC and large volumes

[cheezmonkey]

Hello!

Long time user, first time poster here.

I've been running PVE at home for years. I love it. I've even managed to pop it into a few places professionally when the opportunity has arisen. Of course like all the best engineers, my stuff at home is... well it works.

Recently, some bad RAM resulted in a big mess and long story short, I ended up buying a second set of 4x10TB drives for a data recovery operation. That all went well enough and is not quite why I'm here. Having done what I could with that situation, I was left with two things:

4x10TB disks in a USB enclosure
The abject realization that whilst it had served me well enough for many years, my quite basic backup routine needed moderninsing.

Of course, I'd seen PBS floating around, I've been using PVE since before PBS even came out. I've never really been in a position / had the need to give it a go, but now was the time. So I built a test VM, just to get a feel for it. Picked a handful of my ~40 VMs and a container and played with it for a week. I was impressed, I liked it a lot and thought it would make a fantastic upgrade for my setup. Now, my server cupboard only has so much space in it, and that's not enough for much more than what's already in there, so for better or worse, I bought a cheap little Dell 7050 Micro, slammed the USB enclosure with the disks into it and got to work.

I'll take a break from the story for a moment to acknowledge a couple of points. I know USB attached storage isn't great. I know there are better / alternative methods. This is a home lab, not production. I am not a wealthy man. This is the equipment I have to work with. I was already hundreds in the hole for the data recovery job and didn't have much pocket money left.

So I built it and set it up. All is well. Sure, it could be faster. The PBS machine is an older i5 with 16GB RAM and, as mentioned 4x10TB Dell Enterprise disks in a USB-C enclosure. Apparently not connected with UAS, but again, this is the hardware I have, the enclosure cost 100GBP and I'm probably not going to buy another soon.

I setup the beginnings of a routine, with a weekly verify and GC and an hourly prune, a nice retention system (certainly better than the old arrangement, which was "keep the last backup"_ - that could probably be reduced, but I don't think it's the problem. I set off a one-off backup of all machines and lxc's to get an idea of how long it would take, curious whether I could get away with daily backups of all the things. It ripped through 39 out of 40 machines in, IIRC, a couple of hours. The one machine, my file server, with a 20TB volume attached took about 40 hours, so all in, everything I own backed up fully in less than 48 hours. Not amazing, but with incrementals moving forward, absolutely manageable. Indeed, I Settled on daily backup everything and the subsequent jobs run in about an hour, ideal. This is looking great so far!

Tested a couple of restores, full machine here, container there, even dug a random file out of the depths of the file server backup using the file restore feature. Great great great! This is so much better than the old setup! What's left to do? Ah yes, verify. This seems a sensible thing to do, so after a few backups had accrued, I started a verify job. Much like the backups, it rips through all but the big file server in next to no time. The file server is now, at the time of writing, 3 days, 17 hours into the verify job and it doesn't appear to have finished the first snap yet, never mind the 3 more which have been created sine the job started (Yes, backups have continued running and running well enough throughout)

...And therein lay the problem! I'm not silly as to imagine that 20TB in this arrangement could be verified "quickly" - I know ZFS isn't ideally suited for this part of the operation. I jsut wish I had some idea how long it would take! There's no progress indicator in the job window, it's just been sat at

2025-05-11T20:58:53+01:00: verify group BackupPool:vm/198 (3 snapshots)
2025-05-11T20:58:53+01:00: verify BackupPool:vm/198/2025-05-11T16:26:48Z
2025-05-11T20:58:53+01:00: check qemu-server.conf.blob

Since, wel,, 2025-05-11. I don't believe it's hung, the disks are audibly grinding away. I can see Throughput (though I will observe that it has gone from ~100MB/sec for the first couple days to highly variable rate over the previous 24 hrs)

So my question... would anyone care to guess how long this might take? I won't hold you to it. Everything's waiting on this job, I have to do a bunch of other tasks which will involve powering off both PVE and PBS, so I'm pretty invested in this verify at the moment and definitely don't want to kill it when, for all I know, it's 14 seconds away from finishing! Am I just being silly for expecting this to ever work? (note I said "work" and not "Work well!") and the probably more important question (I suppose I could live without verify, if I were to regularly manually test restore) is - Will GC take this long as well? I'd imagine someone with some more experience with the product could anecdotally tell me something like "My GC usually takes about 50% the time of the verify" -or whatever the real ratio is. I'd appreciate any insights really.

If you're still here, thank you for reading. I really want to add PBS into my setup and hope there's a sensible way to handle this, which jsut hasn't occurred to me yet!

Feel free to request any additional info.

 

[fabian]

a verify is more I/O intensive than GC (assuming the scope is the full datastore for the verification, it always is for GC):

- verify has to iterate over all namespaces/groups/snapshots/indices, read each referenced chunk and calculate its digest, and do some housekeeping
- GC has to iterate over all namespaces/groups/snapshots/indices, and touch each referenced chunk (at least) once, then it has to iterate over all chunks and remove those it hasn't touched

so GC is very metadata/random I/O heavy, verify is random I/O heavy (data and metadata) and pegs the CPU for digest calculation.

GC in particular benefits a lot from anything speeding up random access and metadata updates. for ZFS, this means a (fast/flash) special vdev goes a long way - but it becomes the limiting factor for redundancy/failure probabilities, so you need to take that into account. for verification it depends what your bottle neck actually is:
- iterating/metadata access
- reading chunks/data access
- hash calculation

note that GC got a lot faster for most setups with PBS 3.4, and verifications with smaller scopes are less efficient (e.g., verifying each group or snapshot separately instead of the whole datastore/.. means a lot of chunks are read multiple times)

 

[cheezmonkey]

OK, thanks. That's useful info and corrects something I read elsewhere about special devices not providing benefit here. I acknowledge that you only mention them in relation to GC, not verify. I have one spare slot in the USB enclosure and the Dell box has an empty NVME slot. I wonder how broken things might get if I added an SSD to the enclosure and an NVME internally use use as special devices?
From the looks of things, I suspect, based on what you've said there, that the hash calculation is the main hold-up. The CPU has been bouncing off the top of the graph throughout, though mostly on iowait, unsurprisingly. Some time in the future I could probably find my way to acquiring an appropriately aged i7 to go in the PBS box, maybe that'll help.

Actually, could you clear up one more specific point please? On verify jobs, does the full data set have to be re-read for each backup instance? ie I'm intending to keep 6 copies (3 dailies, 2 weekly, 1 monthly) --so is this already guaranteed to take more than 6 times 3 days every time? Or will it only check the difference, so take much less time (the data churn is no more than about 10GB per typical week.)

 

[fabian]

the verify granularity is snapshot, not chunks, so at least one full snapshot will be verified. if you verify multiple snapshots in a single task (e.g., a whole group or namespace or datastore), then each chunk will only be verified once during that task even if verified multiple times.

for example, if you add a new snapshot every day, and verify it immediately, that will be slower than verifying a week's worth of snapshots once a week (assuming there is chunk reuse/deduplication across those snapshots, of course, but that is very likely ) there is no per-chunk state that is stored, so it's not possible to say "verify snapshot X but only chunks that haven't been verified yet".

the reason GC benefits the most from a special vdev is that it is basically almost only metadata operations:
- iterating over directories
- reading index files (not just metadata)
- touching chunks (this step in particular really benefits from special vdevs)
- iterating over chunks
- removing chunks

except for the index reading it's only metadata reading or updating, and depending on storage, removing chunks might be very efficient or a data-like operation.

verification always has to read and process the data it is verifying, so it's only the iteration part the gets sped up by special vdevs.

 

[cheezmonkey]

Awesome, great info. Thank you fabian. I think I'll just suck it up and see how long this verify ends up taking. Hopefully I don't trip over all the cables stretched across the room "temporarily" while I get things up and running. (BOY do I regret that decision!)
Once I know how long it's going to take, I guess I'll use that to inform the verify schedule. If it takes 8 days, I guess I'll verify every second week or something. This is definitely not enough of a detractor to turn me off the product, it's just surprised me a bit.

I'll stop taking your time away from people with problems they didn't create for themselves and try to stop staring at the job progress.

Huge thanks to the project. Huge-r thanks to the people who give their time for free to help loons like me doing silly things like this. As you were. Have a good one.