Checking the size of incremental backups

larsen

Member
Feb 28, 2020
108
11
23
I am backing up a VM that is running MS SQL Server 2008 for use with GFI MailArchiver. Our company has around 45 users and a mail volume of about 3 Gb per month, ~100 Mb/day. However, the daily backup size on weekdays is around 100 Gb! On weekends with very little mail traffic, the backup size is around 20 Gb.

From another thread:
The way the dirty bitmaps work, is by marking a whole block dirty even if just a single bit in that block has changed. So if the writes are small but in a lot of different blocks, they all will be marked dirty.
Taking this, and overhead, and other OS stuff into account, I still think this is pretty big.

Is there any way to check what files in particular are causing the backup to grow that much?
Or is it impossible as PBS only sees blocks and cannot de-reference them to files?


Lars
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
7,874
952
163
33
Vienna
Is there any way to check what files in particular are causing the backup to grow that much?
Or is it impossible as PBS only sees blocks and cannot de-reference them to files?
AFAIK there is no easy/cheap way to determine which files changed, since only blocks are tracked.

note that everything that is changing is backed up, this includes log files, system updates, and even 'trimmed' blocks (even though they will not take up much space)

can you post a task log of such backups?
 

larsen

Member
Feb 28, 2020
108
11
23
Sure.

INFO: starting new backup job: vzdump 109 --quiet 1 --node atl-vm04 --mailto system@example.com --mailnotification failure --storage pbs-atl-vm03 --mode snapshot --compress zstd
INFO: Starting Backup of VM 109 (qemu)
INFO: Backup started at 2021-09-28 22:30:03
INFO: status = running
INFO: VM Name: sql01
INFO: include disk 'scsi0' 'lvmthin:vm-109-disk-0' 1000G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/109/2021-09-28T20:30:03Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '0f24a8e4-49f2-498a-93e8-67d50f90ada9'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (91.2 GiB of 1000.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 91.2 GiB dirty of 1000.0 GiB total
INFO: 0% (232.0 MiB of 91.2 GiB) in 3s, read: 77.3 MiB/s, write: 76.0 MiB/s
...snip...
INFO: 100% (91.2 GiB of 91.2 GiB) in 4m 43s, read: 44.0 MiB/s, write: 38.0 MiB/s
INFO: backup is sparse: 68.35 GiB (74%) total zero data
INFO: backup was done incrementally, reused 978.12 GiB (97%)
INFO: transferred 91.20 GiB in 283 seconds (330.0 MiB/s)
INFO: Finished Backup of VM 109 (00:04:57)
INFO: Backup finished at 2021-09-28 22:35:00
INFO: Backup job finished successfully
TASK OK
"backup is sparse: 68.35 GiB (74%) total zero data" seems strange to me.
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
7,874
952
163
33
Vienna
"backup is sparse: 68.35 GiB (74%) total zero data" seems strange to me.
no thats exactly what i meant with 'even trimmed blocks'. qemu marks blocks that were overwritten with zeroes (or trimmed) as dirty (even if they were zero already).
pbs client does not know this until it reads that data, but it must send that to the server (since it does not know what exactly was in that block before, only that it 'changed')

so in reality, only about 22GiB of data were sent and 68GiB were 'zero'
 

larsen

Member
Feb 28, 2020
108
11
23
That fits to the log from a Sunday where only ~20 Gb have been transferred:
INFO: backup is sparse: 232.00 MiB (1%) total zero data

So, it seems I have a constant value of 20 Gb that change and a constant value of 70-80 Gb that change but is zero.
Would love to know what exactly is happening here... =/
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
7,874
952
163
33
Vienna
if you really want to know what changed between 2 backups, you can restore the disk images of 2 consecutive backups and compare the content manually

So, it seems I have a constant value of 20 Gb that change and a constant value of 70-80 Gb that change but is zero.
Would love to know what exactly is happening here... =/
logs/updates/filesystem changes/etc

the zero blocks probably come from created and then deleted files (e.g. temporary files)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!