Checking the size of incremental backups

[larsen]

I am backing up a VM that is running MS SQL Server 2008 for use with GFI MailArchiver. Our company has around 45 users and a mail volume of about 3 Gb per month, ~100 Mb/day. However, the daily backup size on weekdays is around 100 Gb! On weekends with very little mail traffic, the backup size is around 20 Gb.

From another thread:

The way the dirty bitmaps work, is by marking a whole block dirty even if just a single bit in that block has changed. So if the writes are small but in a lot of different blocks, they all will be marked dirty.

Taking this, and overhead, and other OS stuff into account, I still think this is pretty big.

Is there any way to check what files in particular are causing the backup to grow that much?
Or is it impossible as PBS only sees blocks and cannot de-reference them to files?


Lars

 

[dcsapak]

Is there any way to check what files in particular are causing the backup to grow that much?
Or is it impossible as PBS only sees blocks and cannot de-reference them to files?

AFAIK there is no easy/cheap way to determine which files changed, since only blocks are tracked.

note that everything that is changing is backed up, this includes log files, system updates, and even 'trimmed' blocks (even though they will not take up much space)

can you post a task log of such backups?

 

[larsen]

Sure.

INFO: starting new backup job: vzdump 109 --quiet 1 --node atl-vm04 --mailto system@example.com --mailnotification failure --storage pbs-atl-vm03 --mode snapshot --compress zstd
INFO: Starting Backup of VM 109 (qemu)
INFO: Backup started at 2021-09-28 22:30:03
INFO: status = running
INFO: VM Name: sql01
INFO: include disk 'scsi0' 'lvmthin:vm-109-disk-0' 1000G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/109/2021-09-28T20:30:03Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '0f24a8e4-49f2-498a-93e8-67d50f90ada9'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (91.2 GiB of 1000.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 91.2 GiB dirty of 1000.0 GiB total
INFO: 0% (232.0 MiB of 91.2 GiB) in 3s, read: 77.3 MiB/s, write: 76.0 MiB/s
...snip...
INFO: 100% (91.2 GiB of 91.2 GiB) in 4m 43s, read: 44.0 MiB/s, write: 38.0 MiB/s
INFO: backup is sparse: 68.35 GiB (74%) total zero data
INFO: backup was done incrementally, reused 978.12 GiB (97%)
INFO: transferred 91.20 GiB in 283 seconds (330.0 MiB/s)
INFO: Finished Backup of VM 109 (00:04:57)
INFO: Backup finished at 2021-09-28 22:35:00
INFO: Backup job finished successfully
TASK OK

"backup is sparse: 68.35 GiB (74%) total zero data" seems strange to me.

 

[dcsapak]

"backup is sparse: 68.35 GiB (74%) total zero data" seems strange to me.

no thats exactly what i meant with 'even trimmed blocks'. qemu marks blocks that were overwritten with zeroes (or trimmed) as dirty (even if they were zero already).
pbs client does not know this until it reads that data, but it must send that to the server (since it does not know what exactly was in that block before, only that it 'changed')

so in reality, only about 22GiB of data were sent and 68GiB were 'zero'

 

[larsen]

That fits to the log from a Sunday where only ~20 Gb have been transferred:
INFO: backup is sparse: 232.00 MiB (1%) total zero data

So, it seems I have a constant value of 20 Gb that change and a constant value of 70-80 Gb that change but is zero.
Would love to know what exactly is happening here... =/

 

[dcsapak]

if you really want to know what changed between 2 backups, you can restore the disk images of 2 consecutive backups and compare the content manually

So, it seems I have a constant value of 20 Gb that change and a constant value of 70-80 Gb that change but is zero.
Would love to know what exactly is happening here... =/

logs/updates/filesystem changes/etc

the zero blocks probably come from created and then deleted files (e.g. temporary files)