Any Hints why my incremental Backups are way too large ?

[adoII]

We are still evaluating pbs Server and begin to like it more and more.
One problem ist that on some Servers e.g. mailserver or lamp server the incremental size ist unbevlievable high. An incremental after 24 hours is about 50% of the initial Backup size. But with a find-command run i find only few changed files.
I know, dirty bitmaps don't lie but what in hell is changing on the disks ? I am sure there is no trim job running and the mounts are noatime.
Do you have any hints what i can do on the virtual machines to reduce number of changed blocks ? Maybe there are mount options for ext4 or maybe ext4 as a copy on write filsystem is ineffective for incremental Backups ? Or Disks which are more than 90% full tend to change more blocks than emptier disks ?
Any Ideas what I can try and where I can search ?

 

[mira]

Please provide the output of pveversion -v from your PVE node where the VM is running, the output of proxmox-backup-manager versions --verbose and the complete task log of the backup.

 

[adoII]

I attach the output as a file.
Okay I see it is 123GB changed and 543GB unchanged. For me this still seems way to large,

 

[mira]

So the last 3 disks have (close to) no changes. I guess the first disk is your root?
What's running in the VM?

The way the dirty bitmaps work, is by marking a whole block dirty even if just a single bit in that block has changed. So if the writes are small but in a lot of different blocks, they all will be marked dirty.

 

[adoII]

Hmm, let's look at scsi0 80 GB Disk.
it constantly only has 20 GB in use and 60 GB free.
So it looks strange that 40GB change within a day
It is an ext3 filesystem.
I checked all cronjobs and logs an i have no idea what si happening ...

 

[abma]

is discard for scsi0 enabled?

running "fstrim" for this drive before the backup could help as you have a lot of free disk space.

 

[adoII]

I think the opposite.
If i run fstrim on an 80GB drive with only 20GB of data it will lead to 60GB changed data.
If I run fstri again it will lead to another 60GB of trimmed data.

 

[abma]

https://pve.proxmox.com/wiki/Shrink_Qcow2_Disk_Files

yes, it will change blocks, but it will mark the blocks as empty: this should reduce the backup size.

maybe "iotop" is your friend, for more details use "fatrace". disk writes heavily depend on the services running on it. ext4 uses journaling as most databases do, too which very likely causes a lot of changes to disk.

does a swap partion on this disk exist?
is a database running which does a lot of updates/changes?
is php running and creating (+deleting) a lot of session files?
are lvm snapshots or filesystem snapshots beeing used?