backup size much larger than disk usage

[lijieamd]

Hi everyone,
I have an VM(ubuntu server) with 3 virtual disk:



here is my VM's disk usage (using command lsblk and df):





I have unchecked vm-100-disk-1's backup checkbox in PVE web gui and finished an backup operation, I'm confused about the backup size, the backup size is much larger (420GB) than VM's usage(about 27+62GB), it's normal?


here is my backup log:

INFO: starting new backup job: vzdump 100 --node pve --remove 0 --storage data --mode stop --compress zstd
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2021-08-22 14:05:06
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: ubuntu-server
INFO: include disk 'scsi0' 'local-lvm:vm-100-disk-1' 64G
INFO: include disk 'scsi1' 'data:100/vm-100-disk-0.raw' 1000G
INFO: exclude disk 'scsi2' 'data:100/vm-100-disk-1.raw' (backup=no)
INFO: creating vzdump archive '/mnt/pve/data/dump/vzdump-qemu-100-2021_08_22-14_05_05.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task 'fc39c95d-28b5-4c35-a020-7bb4f967a7d1'
INFO: 0% (482.8 MiB of 1.0 TiB) in 3s, read: 160.9 MiB/s, write: 113.4 MiB/s
INFO: 1% (10.7 GiB of 1.0 TiB) in 1m 44s, read: 103.4 MiB/s, write: 90.5 MiB/s
INFO: 2% (21.3 GiB of 1.0 TiB) in 3m 38s, read: 95.3 MiB/s, write: 87.1 MiB/s
INFO: 3% (32.0 GiB of 1.0 TiB) in 5m 38s, read: 91.3 MiB/s, write: 85.6 MiB/s
INFO: 4% (42.6 GiB of 1.0 TiB) in 7m 28s, read: 98.6 MiB/s, write: 89.3 MiB/s
INFO: 5% (53.2 GiB of 1.0 TiB) in 9m 35s, read: 86.1 MiB/s, write: 79.5 MiB/s
INFO: 6% (63.9 GiB of 1.0 TiB) in 11m 28s, read: 96.8 MiB/s, write: 86.7 MiB/s
INFO: 7% (74.6 GiB of 1.0 TiB) in 13m 26s, read: 92.7 MiB/s, write: 84.0 MiB/s
INFO: 8% (85.2 GiB of 1.0 TiB) in 15m 38s, read: 82.2 MiB/s, write: 77.2 MiB/s
INFO: 9% (95.8 GiB of 1.0 TiB) in 17m 44s, read: 85.7 MiB/s, write: 80.0 MiB/s
INFO: 10% (106.5 GiB of 1.0 TiB) in 19m 33s, read: 100.7 MiB/s, write: 88.8 MiB/s
INFO: 11% (117.1 GiB of 1.0 TiB) in 21m 14s, read: 107.2 MiB/s, write: 95.4 MiB/s
INFO: 12% (127.7 GiB of 1.0 TiB) in 23m 6s, read: 97.7 MiB/s, write: 87.4 MiB/s
INFO: 13% (138.3 GiB of 1.0 TiB) in 24m 56s, read: 98.6 MiB/s, write: 89.4 MiB/s
INFO: 14% (149.0 GiB of 1.0 TiB) in 26m 54s, read: 92.6 MiB/s, write: 84.0 MiB/s
INFO: 15% (159.7 GiB of 1.0 TiB) in 28m 42s, read: 101.7 MiB/s, write: 88.7 MiB/s
INFO: 16% (170.3 GiB of 1.0 TiB) in 30m 40s, read: 92.0 MiB/s, write: 84.9 MiB/s
INFO: 17% (180.9 GiB of 1.0 TiB) in 32m 39s, read: 91.2 MiB/s, write: 85.1 MiB/s
INFO: 18% (191.6 GiB of 1.0 TiB) in 34m 40s, read: 90.6 MiB/s, write: 81.7 MiB/s
INFO: 19% (202.2 GiB of 1.0 TiB) in 36m 38s, read: 91.3 MiB/s, write: 84.1 MiB/s
INFO: 20% (212.9 GiB of 1.0 TiB) in 38m 36s, read: 93.1 MiB/s, write: 86.7 MiB/s
INFO: 21% (223.4 GiB of 1.0 TiB) in 40m 37s, read: 89.2 MiB/s, write: 82.0 MiB/s
INFO: 22% (234.2 GiB of 1.0 TiB) in 42m 35s, read: 93.0 MiB/s, write: 80.4 MiB/s
INFO: 23% (244.8 GiB of 1.0 TiB) in 44m 27s, read: 97.4 MiB/s, write: 85.1 MiB/s
INFO: 24% (255.5 GiB of 1.0 TiB) in 46m 26s, read: 91.6 MiB/s, write: 86.3 MiB/s
INFO: 25% (266.2 GiB of 1.0 TiB) in 48m 24s, read: 93.2 MiB/s, write: 86.8 MiB/s
INFO: 26% (276.7 GiB of 1.0 TiB) in 50m 21s, read: 91.7 MiB/s, write: 86.3 MiB/s
INFO: 27% (287.4 GiB of 1.0 TiB) in 52m 16s, read: 95.5 MiB/s, write: 90.1 MiB/s
INFO: 28% (298.1 GiB of 1.0 TiB) in 54m 12s, read: 94.8 MiB/s, write: 88.3 MiB/s
INFO: 29% (308.6 GiB of 1.0 TiB) in 56m 9s, read: 91.5 MiB/s, write: 86.1 MiB/s
INFO: 30% (319.2 GiB of 1.0 TiB) in 58m 7s, read: 92.2 MiB/s, write: 86.1 MiB/s
INFO: 31% (329.9 GiB of 1.0 TiB) in 1h 0m 2s, read: 94.8 MiB/s, write: 79.6 MiB/s
INFO: 32% (340.5 GiB of 1.0 TiB) in 1h 1m 40s, read: 111.1 MiB/s, write: 87.0 MiB/s
INFO: 33% (351.2 GiB of 1.0 TiB) in 1h 3m 31s, read: 98.5 MiB/s, write: 86.6 MiB/s
INFO: 34% (361.9 GiB of 1.0 TiB) in 1h 5m 35s, read: 88.2 MiB/s, write: 81.0 MiB/s
INFO: 35% (372.5 GiB of 1.0 TiB) in 1h 7m 28s, read: 96.5 MiB/s, write: 87.1 MiB/s
INFO: 36% (383.1 GiB of 1.0 TiB) in 1h 9m 37s, read: 83.9 MiB/s, write: 76.9 MiB/s
INFO: 37% (393.8 GiB of 1.0 TiB) in 1h 11m 37s, read: 91.6 MiB/s, write: 86.2 MiB/s
INFO: 38% (404.3 GiB of 1.0 TiB) in 1h 13m 27s, read: 98.0 MiB/s, write: 90.9 MiB/s
INFO: 39% (415.0 GiB of 1.0 TiB) in 1h 15m 26s, read: 91.7 MiB/s, write: 85.8 MiB/s
INFO: 40% (425.6 GiB of 1.0 TiB) in 1h 17m 25s, read: 91.5 MiB/s, write: 85.9 MiB/s
INFO: 41% (436.3 GiB of 1.0 TiB) in 1h 19m 23s, read: 92.7 MiB/s, write: 85.9 MiB/s
INFO: 42% (447.0 GiB of 1.0 TiB) in 1h 21m 21s, read: 92.6 MiB/s, write: 83.5 MiB/s
INFO: 43% (459.8 GiB of 1.0 TiB) in 1h 21m 51s, read: 439.4 MiB/s, write: 90.8 MiB/s
INFO: 45% (482.5 GiB of 1.0 TiB) in 1h 21m 54s, read: 7.6 GiB/s, write: 16.0 KiB/s
INFO: 46% (494.0 GiB of 1.0 TiB) in 1h 21m 57s, read: 3.8 GiB/s, write: 10.7 KiB/s
INFO: 47% (505.7 GiB of 1.0 TiB) in 1h 22m 2s, read: 2.3 GiB/s, write: 6.4 KiB/s
INFO: 49% (528.3 GiB of 1.0 TiB) in 1h 22m 5s, read: 7.5 GiB/s, write: 16.0 KiB/s
INFO: 51% (550.6 GiB of 1.0 TiB) in 1h 22m 8s, read: 7.4 GiB/s, write: 3.2 MiB/s
INFO: 53% (573.6 GiB of 1.0 TiB) in 1h 22m 11s, read: 7.7 GiB/s, write: 14.7 KiB/s
INFO: 54% (580.3 GiB of 1.0 TiB) in 1h 22m 14s, read: 2.2 GiB/s, write: 63.1 MiB/s
INFO: 55% (588.4 GiB of 1.0 TiB) in 1h 22m 17s, read: 2.7 GiB/s, write: 87.4 MiB/s
INFO: 56% (601.0 GiB of 1.0 TiB) in 1h 22m 22s, read: 2.5 GiB/s, write: 89.2 MiB/s
INFO: 57% (613.5 GiB of 1.0 TiB) in 1h 22m 27s, read: 2.5 GiB/s, write: 101.0 MiB/s
INFO: 59% (636.1 GiB of 1.0 TiB) in 1h 22m 30s, read: 7.5 GiB/s, write: 16.0 KiB/s
INFO: 61% (659.0 GiB of 1.0 TiB) in 1h 22m 33s, read: 7.6 GiB/s, write: 14.7 KiB/s
INFO: 64% (681.4 GiB of 1.0 TiB) in 1h 22m 36s, read: 7.4 GiB/s, write: 14.7 KiB/s
INFO: 66% (704.4 GiB of 1.0 TiB) in 1h 22m 39s, read: 7.7 GiB/s, write: 16.0 KiB/s
INFO: 68% (727.3 GiB of 1.0 TiB) in 1h 22m 42s, read: 7.6 GiB/s, write: 14.7 KiB/s
INFO: 70% (748.0 GiB of 1.0 TiB) in 1h 22m 45s, read: 6.9 GiB/s, write: 13.3 KiB/s
INFO: 71% (763.3 GiB of 1.0 TiB) in 1h 22m 48s, read: 5.1 GiB/s, write: 10.7 KiB/s
INFO: 73% (786.6 GiB of 1.0 TiB) in 1h 22m 51s, read: 7.8 GiB/s, write: 24.0 KiB/s
INFO: 76% (809.9 GiB of 1.0 TiB) in 1h 22m 54s, read: 7.8 GiB/s, write: 14.7 KiB/s
INFO: 78% (833.2 GiB of 1.0 TiB) in 1h 22m 57s, read: 7.8 GiB/s, write: 185.3 KiB/s
INFO: 80% (856.4 GiB of 1.0 TiB) in 1h 23m 0s, read: 7.7 GiB/s, write: 82.7 KiB/s
INFO: 82% (880.0 GiB of 1.0 TiB) in 1h 23m 3s, read: 7.9 GiB/s, write: 14.7 KiB/s
INFO: 84% (903.4 GiB of 1.0 TiB) in 1h 23m 6s, read: 7.8 GiB/s, write: 16.0 KiB/s
INFO: 86% (919.2 GiB of 1.0 TiB) in 1h 23m 9s, read: 5.3 GiB/s, write: 10.7 KiB/s
INFO: 88% (942.6 GiB of 1.0 TiB) in 1h 23m 12s, read: 7.8 GiB/s, write: 16.0 KiB/s
INFO: 90% (966.1 GiB of 1.0 TiB) in 1h 23m 15s, read: 7.8 GiB/s, write: 16.0 KiB/s
INFO: 92% (986.2 GiB of 1.0 TiB) in 1h 23m 18s, read: 6.7 GiB/s, write: 13.6 MiB/s
INFO: 94% (1000.4 GiB of 1.0 TiB) in 1h 23m 21s, read: 4.7 GiB/s, write: 75.0 MiB/s
INFO: 95% (1010.9 GiB of 1.0 TiB) in 1h 24m 14s, read: 204.1 MiB/s, write: 164.3 MiB/s
INFO: 96% (1021.6 GiB of 1.0 TiB) in 1h 25m 12s, read: 188.4 MiB/s, write: 177.8 MiB/s
INFO: 97% (1.0 TiB of 1.0 TiB) in 1h 26m 11s, read: 183.9 MiB/s, write: 178.7 MiB/s
INFO: 98% (1.0 TiB of 1.0 TiB) in 1h 27m 13s, read: 175.7 MiB/s, write: 168.3 MiB/s
INFO: 99% (1.0 TiB of 1.0 TiB) in 1h 28m 12s, read: 184.4 MiB/s, write: 173.9 MiB/s
INFO: 100% (1.0 TiB of 1.0 TiB) in 1h 29m 8s, read: 193.0 MiB/s, write: 183.7 MiB/s
INFO: backup is sparse: 593.90 GiB (55%) total zero data
INFO: transferred 1.04 TiB in 5348 seconds (203.7 MiB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 424.01GB
INFO: Finished Backup of VM 100 (01:29:10)
INFO: Backup finished at 2021-08-22 15:34:15
INFO: Backup job finished successfully
TASK OK

 

[leesteken]

The backup is on block level, so maybe a lot of deleted data is still taking up space. Have you tried enabling Discard and do a trim inside the VM? From the screenshots, I see that you are using raw-files for most of the virtual drives but I'm afraid that those do not support thin provisioning. Maybe you can move them to a LVM or ZFS storage?

 

[lijieamd]

The backup is on block level, so maybe a lot of deleted data is still taking up space. Have you tried enabling Discard and do a trim inside the VM? From the screenshots, I see that you are using raw-files for most of the virtual drives but I'm afraid that those do not support thin provisioning. Maybe you can move them to a LVM or ZFS storage?

thanks for your reply!
I guess that's the problem because i just remove a large dir in VM, it makesense, i'll go to read documents about storage and try to change storge type
thank you again!

 

[spirit]

The backup job is backuping blocks, not files. So for example, if you have filled your disk once with a big file, then you delete it, the datas are still on the blocks. That's why you need to enable discard to really remove datas from blocks.

(it should works with raw-files too, even if you don't have thin provisionning enabled on your local storage).

 

[lijieamd]

The backup job is backuping blocks, not files. So for example, if you have filled your disk once with a big file, then you delete it, the datas are still on the blocks. That's why you need to enable discard to really remove datas from blocks.

(it should works with raw-files too, even if you don't have thin provisionning enabled on your local storage).

thank you! spirit
I checked the Discard checkbox in disk option, and do a manual fstrim in VM(it's a automatic system service in ubuntu server)
then backup again, the size shrink to usage size!
so it actually works with raw disk image