Windows Server Backup Shows 20+GiB dirty between backups within a couple hours!

[Tmanok]

Hi Everyone,

Something startling has occurred that I cannot seem to figure out. While reviewing backup logs, a specific Windows Server was backing up more data than expected, so I manually ran my backup task to PBS before today's nightly backup. After a successful backup of 170GiBs, I noticed that maybe 2-3 hours later yet another 20GiB had been marked dirty, no way! That server has had activity, but maybe 5-8GiB at most, especially during off-peak hours.

Is there a methodology to view which files have "changed" in that period of time? This seems like very inappropriate server behaviour. WS Standard 2016, PBS 2.0
Thanks!


Tmanok

 

[t.lamprecht]

Hi,

I noticed that maybe 2-3 hours later yet another 20GiB had been marked dirty, no way! That server has had activity, but maybe 5-8GiB at most, especially during off-peak hours.

Note that there's a potential for some write to dirtiness amplification, PBS uses 4 MiB blocked chunks for deduplication and so on, so PVE tells QEMU to track dirty blocks at that level. Now if the VM touches only a KiB or so in an area the whole 4 MiB block will be marked as dirty. Tracking at finer level's would have more overhead (the tracking info itself needs to be stored somewhere).

Is there a methodology to view which files have "changed" in that period of time? This seems like very inappropriate server behaviour.

I think that should be reported rather to Microsoft, we have no control about what a Windows servers dirties, or if it dirties small data amounts scattered over a lot of space.

 

[Tmanok]

You sir, just answered many questions in one post. Thank you, I have a related question regarding block sizes that I will create another post for, it concerns Ceph and SSD wearout and I think your point about larger blocks underneath virtual storage might be responsible for my SSDs wearing out quicker under Windows Server workloads!

Cheers, that checks out with PBS. Is there any way to reduce that block size? I'm willing to spend CPU cycles and some overhead if it saves me gigabytes of backup transfer and storage.

Tmanok

 

[t.lamprecht]

I have a related question regarding block sizes that I will create another post for, it concerns Ceph and SSD wearout and I think your point about larger blocks underneath virtual storage might be responsible for my SSDs wearing out quicker under Windows Server workloads!

Possibly. Coincidentally Ceph uses the same block size internally, albeit Ceph can cache + batch a few write operations if they happen shortly after each other, so it may not be amplified as big as in the PBS dirty-block tracking case.

Cheers, that checks out with PBS. Is there any way to reduce that block size? I'm willing to spend CPU cycles and some overhead if it saves me gigabytes of backup transfer and storage.

No, I'm afraid, those cannot be changed easily, they are quite backed in due to some coupling between PBS and the PVE VMs.

Note that PBS also compresses blocks with zstd, so it doesn't necessarily writes out the full 20 GiB to disk.

edit: fix zstd/zfs typo

 

[Tmanok]

Possibly. Coincidentally Ceph uses the same block size internally, albeit Ceph can cache + batch a few write operations if they happen shortly after each other, so it may not be amplified as big as in the PBS dirty-block tracking case.


No, I'm afraid, those cannot be changed easily, they are quite backed in due to some coupling between PBS and the PVE VMs.

Note that PBS also compresses blocks with ZFS, so it doesn't necessarily writes out the full 20 GiB to disk.

Ok, thanks again Thomas, that's precisely what I needed answered.


Tmanok