[SOLVED] Real size of diskspace used in backups

[krikey]

Is there a way to get the real value of how much disk space is used per server backup?

Currently I have 7 nightly rolling snapshot backups for my VMs and CTs but each one of them shows a very similar file size in PBS (understandably), generally growing slightly each night as more data is added.

I'd like to understand what the real storage footprint of a set of backup files for a VM/CT really is though. Is this possible?

 

[Dunuin]

As far as I know PBS can't show you the size of individual backups, because that would be technically difficult and produce a lot of overhead calculating this because of the deduplication.

This was already duscussed in a thread some weeks/months ago.

 

[krikey]

Understood. Do you happen to have the link to the other thread?

I wonder if theres a very rough calculation I can make to guestimate disk usage?

 

[Dunuin]

Link: https://forum.proxmox.com/threads/determine-vm-backup-size-growth-how.98420/

What might help to extrapolate the backup growth of a VM is looking at the PVE logs of the backup task. You will see lines like this:

INFO: backup is sparse: 352.00 MiB (6%) total zero data
INFO: backup was done incrementally, reused 61.62 GiB (93%)
INFO: transferred 4.96 GiB in 58 seconds (87.6 MiB/s)

As far as I understand PVE will do all the compression, encryption, hashing on the client side and will only send chunks that are not already deduplicated. So in that case that could mean the backup caused the datastore to grow by roughly 4.96GB.

 

[krikey]

is that 4.96GB for that VM?

So perhaps I could take the figure from the "size" column in PBE of the oldest backup, then add on all the transferred data for the subsequent 6?

 

[Dunuin]

is that 4.96GB for that VM?

Complete log of that single VMs backups is:

INFO: Starting Backup of VM 105 (qemu)
INFO: Backup started at 2022-01-07 05:31:32
INFO: status = running
INFO: VM Name: Zabbix
INFO: include disk 'scsi0' 'LVMthin:vm-105-disk-0' 2G
INFO: include disk 'scsi1' 'LVMthin:vm-105-disk-1' 64G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/105/2022-01-07T04:31:32Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '343cec4f-ce88-42ec-ab3a-e733b66815f4'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (32.0 MiB of 2.0 GiB dirty)
INFO: scsi1: dirty-bitmap status: OK (4.9 GiB of 64.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 5.0 GiB dirty of 66.0 GiB total
INFO:   6% (308.0 MiB of 5.0 GiB) in 3s, read: 102.7 MiB/s, write: 102.7 MiB/s
INFO:  11% (592.0 MiB of 5.0 GiB) in 6s, read: 94.7 MiB/s, write: 94.7 MiB/s
INFO:  18% (940.0 MiB of 5.0 GiB) in 9s, read: 116.0 MiB/s, write: 70.7 MiB/s
INFO:  22% (1.1 GiB of 5.0 GiB) in 12s, read: 65.3 MiB/s, write: 65.3 MiB/s
INFO:  26% (1.3 GiB of 5.0 GiB) in 15s, read: 74.7 MiB/s, write: 74.7 MiB/s
INFO:  31% (1.5 GiB of 5.0 GiB) in 18s, read: 74.7 MiB/s, write: 74.7 MiB/s
INFO:  35% (1.8 GiB of 5.0 GiB) in 21s, read: 74.7 MiB/s, write: 74.7 MiB/s
INFO:  40% (2.0 GiB of 5.0 GiB) in 24s, read: 77.3 MiB/s, write: 77.3 MiB/s
INFO:  45% (2.3 GiB of 5.0 GiB) in 27s, read: 94.7 MiB/s, write: 94.7 MiB/s
INFO:  50% (2.5 GiB of 5.0 GiB) in 30s, read: 88.0 MiB/s, write: 88.0 MiB/s
INFO:  56% (2.8 GiB of 5.0 GiB) in 33s, read: 93.3 MiB/s, write: 93.3 MiB/s
INFO:  61% (3.1 GiB of 5.0 GiB) in 36s, read: 92.0 MiB/s, write: 92.0 MiB/s
INFO:  68% (3.4 GiB of 5.0 GiB) in 39s, read: 106.7 MiB/s, write: 106.7 MiB/s
INFO:  73% (3.7 GiB of 5.0 GiB) in 42s, read: 94.7 MiB/s, write: 92.0 MiB/s
INFO:  80% (4.0 GiB of 5.0 GiB) in 45s, read: 110.7 MiB/s, write: 70.7 MiB/s
INFO:  84% (4.2 GiB of 5.0 GiB) in 48s, read: 70.7 MiB/s, write: 70.7 MiB/s
INFO:  92% (4.6 GiB of 5.0 GiB) in 51s, read: 128.0 MiB/s, write: 57.3 MiB/s
INFO:  98% (4.9 GiB of 5.0 GiB) in 54s, read: 113.3 MiB/s, write: 72.0 MiB/s
INFO: 100% (5.0 GiB of 5.0 GiB) in 57s, read: 21.3 MiB/s, write: 21.3 MiB/s
INFO: backup is sparse: 352.00 MiB (6%) total zero data
INFO: backup was done incrementally, reused 61.62 GiB (93%)
INFO: transferred 4.96 GiB in 58 seconds (87.6 MiB/s)
INFO: Finished Backup of VM 105 (00:00:59)
INFO: Backup finished at 2022-01-07 05:32:31

So its 66GB of virtual disks (this is what PBS will show in its "size" coloum) but only 5GB of that changed in 24 hours and need to be backed up incrementally, so this VMs backup size will grow by 5GB per day. If you want to know how much space your initial backup could consume on the PBS you could backup a VM with vzdump and zstd compression and look how big that archive is. It won't be deduplicated but could be a good estimation. Lets say my Vzdump backup file of that VM would be 30GB and I want to store daily backups for a week. Then this VM might possibly consume 30GB + 7 * 5GB = 65GB.

Or if you really care you could add a temporary second datastore to you PBS and do a additional backup of just a single VM to it. Then you can watch how big that second datastore is after 8 days and delete it again.

So perhaps I could take the figure from the "size" column in PBE of the oldest backup, then add on all the transferred data for the subsequent 6?

PBS is only showing the thick size of that VM. Its not taking zero data, compression or deduplication into account. So the size column won't help you much.

 

[krikey]

hmm, thanks for taking the time to explain.

I see that there is another thread too on the same subject https://forum.proxmox.com/threads/how-to-get-the-exactly-backup-size-in-proxmox-backup.93901/ as well as the thread you mentioned above.

I absolutely love the fact that Proxmox added better backup features into their suite of software and PBS certainly seems to do an admirable job of allowing us to compress, backup and de-duplicate multiple VMs as a whole, to save time and space, but it seems, due to the technologies used, it's not the ideal solution if you want to know how much real backup space each VM is using if you wanted to flag a vastly changing VM or even charge for backup space used.

I'll close this thread for now and say "thank you" and perhaps carry on the conversation on an existing thread.