Windows Server Backup Shows 20+GiB dirty between backups within a couple hours!

Tmanok

Well-Known Member
Hi Everyone,

Something startling has occurred that I cannot seem to figure out. While reviewing backup logs, a specific Windows Server was backing up more data than expected, so I manually ran my backup task to PBS before today's nightly backup. After a successful backup of 170GiBs, I noticed that maybe 2-3 hours later yet another 20GiB had been marked dirty, no way! That server has had activity, but maybe 5-8GiB at most, especially during off-peak hours.

Is there a methodology to view which files have "changed" in that period of time? This seems like very inappropriate server behaviour. WS Standard 2016, PBS 2.0
Thanks!


Tmanok
 
Hi,

I noticed that maybe 2-3 hours later yet another 20GiB had been marked dirty, no way! That server has had activity, but maybe 5-8GiB at most, especially during off-peak hours.
Note that there's a potential for some write to dirtiness amplification, PBS uses 4 MiB blocked chunks for deduplication and so on, so PVE tells QEMU to track dirty blocks at that level. Now if the VM touches only a KiB or so in an area the whole 4 MiB block will be marked as dirty. Tracking at finer level's would have more overhead (the tracking info itself needs to be stored somewhere).

Is there a methodology to view which files have "changed" in that period of time? This seems like very inappropriate server behaviour.
I think that should be reported rather to Microsoft, we have no control about what a Windows servers dirties, or if it dirties small data amounts scattered over a lot of space.
 
  • Like
Reactions: Tmanok
You sir, just answered many questions in one post. Thank you, I have a related question regarding block sizes that I will create another post for, it concerns Ceph and SSD wearout and I think your point about larger blocks underneath virtual storage might be responsible for my SSDs wearing out quicker under Windows Server workloads!

Cheers, that checks out with PBS. Is there any way to reduce that block size? I'm willing to spend CPU cycles and some overhead if it saves me gigabytes of backup transfer and storage.

Tmanok
 
I have a related question regarding block sizes that I will create another post for, it concerns Ceph and SSD wearout and I think your point about larger blocks underneath virtual storage might be responsible for my SSDs wearing out quicker under Windows Server workloads!
Possibly. Coincidentally Ceph uses the same block size internally, albeit Ceph can cache + batch a few write operations if they happen shortly after each other, so it may not be amplified as big as in the PBS dirty-block tracking case.

Cheers, that checks out with PBS. Is there any way to reduce that block size? I'm willing to spend CPU cycles and some overhead if it saves me gigabytes of backup transfer and storage.
No, I'm afraid, those cannot be changed easily, they are quite backed in due to some coupling between PBS and the PVE VMs.

Note that PBS also compresses blocks with zstd, so it doesn't necessarily writes out the full 20 GiB to disk.

edit: fix zstd/zfs typo
 
Last edited:
Possibly. Coincidentally Ceph uses the same block size internally, albeit Ceph can cache + batch a few write operations if they happen shortly after each other, so it may not be amplified as big as in the PBS dirty-block tracking case.


No, I'm afraid, those cannot be changed easily, they are quite backed in due to some coupling between PBS and the PVE VMs.

Note that PBS also compresses blocks with ZFS, so it doesn't necessarily writes out the full 20 GiB to disk.
Ok, thanks again Thomas, that's precisely what I needed answered. :)


Tmanok
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!