Incrementally backup, but only 30% reused data

tmatthaeus

Member
Apr 26, 2012
9
0
21
Bremen, Germany
Hello all,


with a virtual machine (running on Proxmox of course) and which I backup via Proxmox Backup Server I have the problem that it reuses too little data in my opinion and thus my backup grows incredibly:

INFO: backup was done incrementally, reused 779.67 GiB (29%)
INFO: transferred 2.54 TiB in 12562 seconds (211.9 MiB/s)

(Full backup logfile is below).


The following scenario: A virtual machine is running Windows Server 2022. This VM has two hard disks: 1x 100GB and 1x 2.5TB. On the hard disk with 2.5TB there are heaps of images and other data, these data are provided to a client via Windows share. This client in turn changes about 100-200GB per day, but definitely not 1.8TB of data.

However, even though the backup runs daily, only about 800GB of the data is reused and it sends about 2TB of data to the backup daily.

Unfortunately I don't have the starting point to look for the problem. But when I look at the files changed/written in the last 24h in Windows, I don't get much more than 1TB but at most 100-200GB.

Someone has a tip how I can narrow down the problem?

Thanks a lot!



INFO: Starting Backup of VM 306 (qemu)
INFO: Backup started at 2022-06-16 01:03:28
INFO: status = running
INFO: VM Name: W2022
INFO: include disk 'scsi0' 'local-sata_pool:vm-306-disk-0' 100G
INFO: include disk 'scsi1' 'local-sata_pool:vm-306-disk-2' 2500G
INFO: include disk 'efidisk0' 'local-sata_pool:vm-306-disk-1' 1M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/306/2022-06-15T23:03:28Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '587f6e7d-2344-41b2-97fb-57fa51433896'
INFO: resuming VM again
INFO: efidisk0: dirty-bitmap status: created new
INFO: scsi0: dirty-bitmap status: created new
INFO: scsi1: dirty-bitmap status: created new
INFO: 0% (684.0 MiB of 2.5 TiB) in 3s, read: 228.0 MiB/s, write: 198.7 MiB/s
INFO: 1% (26.1 GiB of 2.5 TiB) in 3m 11s, read: 138.4 MiB/s, write: 134.7 MiB/s
INFO: 2% (52.1 GiB of 2.5 TiB) in 6m 30s, read: 133.7 MiB/s, write: 133.7 MiB/s
INFO: 3% (78.1 GiB of 2.5 TiB) in 9m 48s, read: 134.4 MiB/s, write: 134.4 MiB/s
INFO: 4% (104.0 GiB of 2.5 TiB) in 13m 7s, read: 133.5 MiB/s, write: 133.5 MiB/s
INFO: 5% (130.0 GiB of 2.5 TiB) in 16m 25s, read: 134.6 MiB/s, write: 134.6 MiB/s
INFO: 6% (156.0 GiB of 2.5 TiB) in 19m 44s, read: 133.8 MiB/s, write: 133.8 MiB/s
INFO: 7% (182.1 GiB of 2.5 TiB) in 23m 8s, read: 130.7 MiB/s, write: 130.7 MiB/s
INFO: 8% (208.0 GiB of 2.5 TiB) in 26m 35s, read: 128.5 MiB/s, write: 128.5 MiB/s
INFO: 9% (234.0 GiB of 2.5 TiB) in 30m 4s, read: 127.2 MiB/s, write: 127.2 MiB/s
INFO: 10% (260.1 GiB of 2.5 TiB) in 33m 28s, read: 130.9 MiB/s, write: 130.9 MiB/s
INFO: 11% (286.1 GiB of 2.5 TiB) in 36m 56s, read: 127.8 MiB/s, write: 127.8 MiB/s
INFO: 12% (312.0 GiB of 2.5 TiB) in 40m 21s, read: 129.8 MiB/s, write: 129.8 MiB/s
INFO: 13% (338.1 GiB of 2.5 TiB) in 43m 45s, read: 130.6 MiB/s, write: 130.6 MiB/s
INFO: 14% (364.1 GiB of 2.5 TiB) in 47m 17s, read: 125.8 MiB/s, write: 125.8 MiB/s
INFO: 15% (390.0 GiB of 2.5 TiB) in 50m 45s, read: 127.6 MiB/s, write: 127.6 MiB/s
INFO: 16% (416.0 GiB of 2.5 TiB) in 54m 13s, read: 128.1 MiB/s, write: 128.1 MiB/s
INFO: 17% (442.1 GiB of 2.5 TiB) in 57m 40s, read: 128.9 MiB/s, write: 128.9 MiB/s
INFO: 18% (468.0 GiB of 2.5 TiB) in 1h 1m 9s, read: 126.9 MiB/s, write: 126.9 MiB/s
INFO: 19% (494.0 GiB of 2.5 TiB) in 1h 4m 32s, read: 131.2 MiB/s, write: 131.2 MiB/s
INFO: 20% (520.0 GiB of 2.5 TiB) in 1h 7m 57s, read: 130.0 MiB/s, write: 130.0 MiB/s
INFO: 21% (546.1 GiB of 2.5 TiB) in 1h 11m 27s, read: 127.0 MiB/s, write: 127.0 MiB/s
INFO: 22% (572.0 GiB of 2.5 TiB) in 1h 14m 52s, read: 129.6 MiB/s, write: 129.6 MiB/s
INFO: 23% (598.0 GiB of 2.5 TiB) in 1h 18m 19s, read: 128.6 MiB/s, write: 128.6 MiB/s
INFO: 24% (624.1 GiB of 2.5 TiB) in 1h 21m 52s, read: 125.6 MiB/s, write: 125.6 MiB/s
INFO: 25% (650.1 GiB of 2.5 TiB) in 1h 25m 17s, read: 129.6 MiB/s, write: 129.6 MiB/s
INFO: 26% (676.1 GiB of 2.5 TiB) in 1h 28m 43s, read: 129.2 MiB/s, write: 129.2 MiB/s
INFO: 27% (702.1 GiB of 2.5 TiB) in 1h 32m 12s, read: 127.4 MiB/s, write: 127.4 MiB/s
INFO: 28% (728.1 GiB of 2.5 TiB) in 1h 35m 38s, read: 129.1 MiB/s, write: 129.1 MiB/s
INFO: 29% (754.0 GiB of 2.5 TiB) in 1h 39m 3s, read: 129.8 MiB/s, write: 129.8 MiB/s
INFO: 30% (780.0 GiB of 2.5 TiB) in 1h 42m 27s, read: 130.5 MiB/s, write: 130.5 MiB/s
INFO: 31% (806.1 GiB of 2.5 TiB) in 1h 45m 52s, read: 130.2 MiB/s, write: 130.2 MiB/s
INFO: 32% (832.0 GiB of 2.5 TiB) in 1h 49m 13s, read: 132.0 MiB/s, write: 132.0 MiB/s
INFO: 33% (858.1 GiB of 2.5 TiB) in 1h 52m 36s, read: 131.6 MiB/s, write: 131.6 MiB/s
INFO: 34% (884.1 GiB of 2.5 TiB) in 1h 55m 58s, read: 131.8 MiB/s, write: 131.8 MiB/s
INFO: 35% (910.1 GiB of 2.5 TiB) in 1h 59m 20s, read: 131.6 MiB/s, write: 131.6 MiB/s
INFO: 36% (936.1 GiB of 2.5 TiB) in 2h 2m 48s, read: 128.2 MiB/s, write: 128.2 MiB/s
INFO: 37% (962.1 GiB of 2.5 TiB) in 2h 6m 15s, read: 128.6 MiB/s, write: 128.6 MiB/s
INFO: 38% (988.1 GiB of 2.5 TiB) in 2h 9m 35s, read: 133.1 MiB/s, write: 133.1 MiB/s
INFO: 39% (1014.0 GiB of 2.5 TiB) in 2h 12m 58s, read: 130.8 MiB/s, write: 130.8 MiB/s
INFO: 40% (1.0 TiB of 2.5 TiB) in 2h 16m 23s, read: 130.2 MiB/s, write: 130.2 MiB/s
INFO: 41% (1.0 TiB of 2.5 TiB) in 2h 19m 26s, read: 145.5 MiB/s, write: 145.5 MiB/s
INFO: 42% (1.1 TiB of 2.5 TiB) in 2h 22m 26s, read: 147.6 MiB/s, write: 147.6 MiB/s
INFO: 43% (1.1 TiB of 2.5 TiB) in 2h 25m 24s, read: 149.4 MiB/s, write: 149.4 MiB/s
INFO: 44% (1.1 TiB of 2.5 TiB) in 2h 28m 29s, read: 144.5 MiB/s, write: 144.5 MiB/s
INFO: 45% (1.1 TiB of 2.5 TiB) in 2h 31m 17s, read: 157.8 MiB/s, write: 157.8 MiB/s
INFO: 46% (1.2 TiB of 2.5 TiB) in 2h 34m 2s, read: 162.1 MiB/s, write: 162.1 MiB/s
INFO: 47% (1.2 TiB of 2.5 TiB) in 2h 36m 46s, read: 161.8 MiB/s, write: 161.8 MiB/s
INFO: 48% (1.2 TiB of 2.5 TiB) in 2h 39m 26s, read: 166.7 MiB/s, write: 166.7 MiB/s
INFO: 49% (1.2 TiB of 2.5 TiB) in 2h 42m 5s, read: 167.8 MiB/s, write: 167.8 MiB/s
INFO: 50% (1.3 TiB of 2.5 TiB) in 2h 44m 43s, read: 168.2 MiB/s, write: 168.2 MiB/s
INFO: 51% (1.3 TiB of 2.5 TiB) in 2h 47m 7s, read: 185.0 MiB/s, write: 185.0 MiB/s
INFO: 52% (1.3 TiB of 2.5 TiB) in 2h 49m 25s, read: 193.2 MiB/s, write: 193.2 MiB/s
INFO: 53% (1.3 TiB of 2.5 TiB) in 2h 51m 35s, read: 204.7 MiB/s, write: 204.7 MiB/s
INFO: 54% (1.4 TiB of 2.5 TiB) in 2h 53m 21s, read: 251.2 MiB/s, write: 251.2 MiB/s
INFO: 55% (1.4 TiB of 2.5 TiB) in 2h 55m 9s, read: 247.0 MiB/s, write: 247.0 MiB/s
INFO: 56% (1.4 TiB of 2.5 TiB) in 2h 56m 55s, read: 250.8 MiB/s, write: 250.8 MiB/s
INFO: 57% (1.4 TiB of 2.5 TiB) in 2h 58m 45s, read: 241.3 MiB/s, write: 241.3 MiB/s
INFO: 58% (1.5 TiB of 2.5 TiB) in 3h 31s, read: 250.9 MiB/s, write: 250.9 MiB/s
INFO: 59% (1.5 TiB of 2.5 TiB) in 3h 2m 19s, read: 247.2 MiB/s, write: 247.2 MiB/s
INFO: 60% (1.5 TiB of 2.5 TiB) in 3h 4m 8s, read: 245.0 MiB/s, write: 245.0 MiB/s
INFO: 61% (1.5 TiB of 2.5 TiB) in 3h 5m 52s, read: 255.5 MiB/s, write: 254.9 MiB/s
INFO: 62% (1.6 TiB of 2.5 TiB) in 3h 7m 39s, read: 248.9 MiB/s, write: 248.9 MiB/s
INFO: 63% (1.6 TiB of 2.5 TiB) in 3h 9m 28s, read: 243.2 MiB/s, write: 243.2 MiB/s
INFO: 64% (1.6 TiB of 2.5 TiB) in 3h 11m 19s, read: 240.3 MiB/s, write: 240.3 MiB/s
INFO: 65% (1.7 TiB of 2.5 TiB) in 3h 13m 20s, read: 219.8 MiB/s, write: 219.8 MiB/s
INFO: 66% (1.7 TiB of 2.5 TiB) in 3h 16m 23s, read: 145.4 MiB/s, write: 145.4 MiB/s
INFO: 67% (1.7 TiB of 2.5 TiB) in 3h 18m 31s, read: 208.8 MiB/s, write: 208.8 MiB/s
INFO: 68% (1.7 TiB of 2.5 TiB) in 3h 20m 38s, read: 209.1 MiB/s, write: 209.1 MiB/s
INFO: 69% (1.8 TiB of 2.5 TiB) in 3h 22m 44s, read: 211.7 MiB/s, write: 211.7 MiB/s
INFO: 70% (1.8 TiB of 2.5 TiB) in 3h 23m 38s, read: 503.6 MiB/s, write: 194.5 MiB/s
INFO: 71% (1.8 TiB of 2.5 TiB) in 3h 23m 48s, read: 2.8 GiB/s, write: 0 B/s
INFO: 72% (1.8 TiB of 2.5 TiB) in 3h 23m 57s, read: 2.9 GiB/s, write: 0 B/s
INFO: 73% (1.9 TiB of 2.5 TiB) in 3h 24m 6s, read: 2.9 GiB/s, write: 0 B/s
INFO: 74% (1.9 TiB of 2.5 TiB) in 3h 24m 14s, read: 3.0 GiB/s, write: 0 B/s
INFO: 75% (1.9 TiB of 2.5 TiB) in 3h 24m 23s, read: 2.9 GiB/s, write: 0 B/s
INFO: 76% (1.9 TiB of 2.5 TiB) in 3h 24m 32s, read: 2.8 GiB/s, write: 0 B/s
INFO: 77% (2.0 TiB of 2.5 TiB) in 3h 24m 41s, read: 2.9 GiB/s, write: 0 B/s
INFO: 78% (2.0 TiB of 2.5 TiB) in 3h 24m 50s, read: 2.9 GiB/s, write: 0 B/s
INFO: 79% (2.0 TiB of 2.5 TiB) in 3h 24m 59s, read: 2.9 GiB/s, write: 0 B/s
INFO: 80% (2.0 TiB of 2.5 TiB) in 3h 25m 8s, read: 2.8 GiB/s, write: 0 B/s
INFO: 81% (2.1 TiB of 2.5 TiB) in 3h 25m 17s, read: 2.9 GiB/s, write: 0 B/s
INFO: 82% (2.1 TiB of 2.5 TiB) in 3h 25m 26s, read: 3.0 GiB/s, write: 0 B/s
INFO: 83% (2.1 TiB of 2.5 TiB) in 3h 25m 35s, read: 2.9 GiB/s, write: 0 B/s
INFO: 84% (2.1 TiB of 2.5 TiB) in 3h 25m 44s, read: 3.0 GiB/s, write: 0 B/s
INFO: 85% (2.2 TiB of 2.5 TiB) in 3h 25m 52s, read: 3.1 GiB/s, write: 0 B/s
INFO: 86% (2.2 TiB of 2.5 TiB) in 3h 26m 1s, read: 3.0 GiB/s, write: 0 B/s
INFO: 87% (2.2 TiB of 2.5 TiB) in 3h 26m 10s, read: 3.0 GiB/s, write: 0 B/s
INFO: 88% (2.2 TiB of 2.5 TiB) in 3h 26m 18s, read: 3.0 GiB/s, write: 0 B/s
INFO: 89% (2.3 TiB of 2.5 TiB) in 3h 26m 27s, read: 2.9 GiB/s, write: 0 B/s
INFO: 90% (2.3 TiB of 2.5 TiB) in 3h 26m 36s, read: 2.9 GiB/s, write: 0 B/s
INFO: 91% (2.3 TiB of 2.5 TiB) in 3h 26m 47s, read: 2.4 GiB/s, write: 372.4 KiB/s
INFO: 92% (2.3 TiB of 2.5 TiB) in 3h 26m 56s, read: 2.9 GiB/s, write: 0 B/s
INFO: 93% (2.4 TiB of 2.5 TiB) in 3h 27m 5s, read: 2.9 GiB/s, write: 0 B/s
INFO: 94% (2.4 TiB of 2.5 TiB) in 3h 27m 14s, read: 2.9 GiB/s, write: 0 B/s
INFO: 95% (2.4 TiB of 2.5 TiB) in 3h 27m 22s, read: 3.1 GiB/s, write: 0 B/s
INFO: 96% (2.4 TiB of 2.5 TiB) in 3h 27m 31s, read: 3.0 GiB/s, write: 455.1 KiB/s
INFO: 97% (2.5 TiB of 2.5 TiB) in 3h 28m 11s, read: 638.8 MiB/s, write: 198.0 MiB/s
INFO: 98% (2.5 TiB of 2.5 TiB) in 3h 29m 5s, read: 538.6 MiB/s, write: 170.6 MiB/s
INFO: 99% (2.5 TiB of 2.5 TiB) in 3h 29m 13s, read: 3.0 GiB/s, write: 0 B/s
INFO: 100% (2.5 TiB of 2.5 TiB) in 3h 29m 22s, read: 2.8 GiB/s, write: 14.2 KiB/s
INFO: backup is sparse: 765.02 GiB (29%) total zero data
INFO: backup was done incrementally, reused 779.67 GiB (29%)
INFO: transferred 2.54 TiB in 12562 seconds (211.9 MiB/s)
INFO: Finished Backup of VM 306 (03:29:27)
INFO: Backup finished at 2022-06-16 04:32:55
 
likely background trim/discard inside the VM - there are plenty of hits here on the forum for this exact issue with windows ;)
 
likely background trim/discard inside the VM - there are plenty of hits here on the forum for this exact issue with windows ;)

Dear Fabian,

thanks for the reply. The VM HardDisk is SCSI and discard is already enabled. Also the VirtIO drivers are installed. I also tried if the guest tools are running:

qm agent 306 get-osinfo
{
"id" : "mswindows",
"kernel-release" : "20348",
"kernel-version" : "10.0",
"machine" : "x86_64",
"name" : "Microsoft Windows",
"pretty-name" : "Windows Server 2022 Standard",
"variant" : "server",
"variant-id" : "server",
"version" : "N/A",
"version-id" : "N/A"
}


But if I try to call fstrim:
qm agent 306 fstrim
VM 306 qmp command 'guest-fstrim' failed - got timeout



I searched for the posts with trim/discard and backup but I found nothing :(


Best, Tobias
 
Last edited:
no - what I meant is that Windows has a background service for this which is usually the culprit. the other options just control whether discard properly works/is passed along to the storage.
 
Dear @fabian ,

thanks for the reply. Do you mean the QEMU-GA service? This one is running. So maybe another idea? The strange thing is: That´s the only windows machine, which is making this trouble. During the backup only a little bit of the files are re-used so the data usage on the PBS-Storage will increase with every backup extremely.
 

Attachments

  • Bildschirmfoto 2022-06-20 um 12.01.44.png
    Bildschirmfoto 2022-06-20 um 12.01.44.png
    444.3 KB · Views: 7

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!