Low Deduplication

T

tmsg

Renowned Member
Jan 13, 2011
11
1
68
I'm hoping someone can point in the the right direction to investigate why so little of the data blocks on PBS are being reused for a backup.

I have 1.9TB vm (two disk 1.4TB/500G) where only 40% of the blocks are reused. The last back reused 770G of data and transferred 1.1T in the backup process. The vm is a web server where customers upload documents. The backups are running once a week and there is no way that 60% of the files on the vm are getting modified in a week. After 3 backups, I have 5TB of disk space used on PBS.

I also have a 300GB vm (database server) running on the same setup and it is reusing 96% of the blocks and only transferring 4%.

Any ideas on why so few of the blocks would be reused would be appreciated.

Log from the last backup:
Code: 
Proxmox
Virtual Environment 6.3-3
Search
Virtual Machine 100 (Web1) on node 'node01'
Server View
Logs
()
INFO: starting new backup job: vzdump 100 --mode snapshot --storage PBSBackup --compress zstd --quiet 1 --mailnotification always
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2021-01-22 23:11:01
INFO: status = running
INFO: VM Name: Web1
INFO: include disk 'virtio0' 'local:100/vm-100-disk-0.qcow2' 1410G
INFO: include disk 'virtio1' 'local:100/vm-100-disk-1.qcow2' 500G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2021-01-23T07:11:01Z'
INFO: started backup task '92af32a5-e2e7-487c-ba31-344b16894d40'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: OK (722.6 GiB of 1.4 TiB dirty)
INFO: virtio1: dirty-bitmap status: OK (418.7 GiB of 500.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 1.1 TiB dirty of 1.9 TiB total
INFO:   0% (112.0 MiB of 1.1 TiB) in  3s, read: 37.3 MiB/s, write: 37.3 MiB/s
INFO:   1% (11.5 GiB of 1.1 TiB) in  5m 13s, read: 37.5 MiB/s, write: 37.5 MiB/s
INFO:   2% (22.8 GiB of 1.1 TiB) in 10m 23s, read: 37.6 MiB/s, write: 37.6 MiB/s
INFO:   3% (34.2 GiB of 1.1 TiB) in 15m 28s, read: 38.3 MiB/s, write: 38.3 MiB/s
INFO:   4% (45.7 GiB of 1.1 TiB) in 20m 37s, read: 37.8 MiB/s, write: 37.8 MiB/s
INFO:   5% (57.1 GiB of 1.1 TiB) in 25m 46s, read: 37.9 MiB/s, write: 37.9 MiB/s
INFO:   6% (68.5 GiB of 1.1 TiB) in 31m 21s, read: 34.8 MiB/s, write: 34.8 MiB/s
INFO:   7% (79.9 GiB of 1.1 TiB) in 36m 56s, read: 34.9 MiB/s, write: 34.9 MiB/s
INFO:   8% (91.3 GiB of 1.1 TiB) in 41m 41s, read: 41.0 MiB/s, write: 41.0 MiB/s
INFO:   9% (102.7 GiB of 1.1 TiB) in 46m 50s, read: 37.8 MiB/s, write: 37.8 MiB/s
INFO:  10% (114.2 GiB of 1.1 TiB) in 53m 10s, read: 30.8 MiB/s, write: 30.8 MiB/s
INFO:  11% (125.6 GiB of 1.1 TiB) in 59m 56s, read: 28.8 MiB/s, write: 28.8 MiB/s
INFO:  12% (137.0 GiB of 1.1 TiB) in  1h  6m 53s, read: 28.0 MiB/s, write: 28.0 MiB/s
INFO:  13% (148.4 GiB of 1.1 TiB) in  1h 14m 12s, read: 26.6 MiB/s, write: 26.6 MiB/s
INFO:  14% (159.8 GiB of 1.1 TiB) in  1h 19m 41s, read: 35.6 MiB/s, write: 35.6 MiB/s
INFO:  15% (171.2 GiB of 1.1 TiB) in  1h 24m 50s, read: 37.9 MiB/s, write: 37.9 MiB/s
INFO:  16% (182.6 GiB of 1.1 TiB) in  1h 30m  8s, read: 36.6 MiB/s, write: 36.6 MiB/s
INFO:  17% (194.0 GiB of 1.1 TiB) in  1h 35m 23s, read: 37.1 MiB/s, write: 37.1 MiB/s
INFO:  18% (205.4 GiB of 1.1 TiB) in  1h 40m 24s, read: 38.9 MiB/s, write: 38.9 MiB/s
INFO:  19% (216.8 GiB of 1.1 TiB) in  1h 45m  3s, read: 41.8 MiB/s, write: 41.8 MiB/s
INFO:  20% (228.3 GiB of 1.1 TiB) in  1h 49m 49s, read: 41.1 MiB/s, write: 41.1 MiB/s
INFO:  21% (239.7 GiB of 1.1 TiB) in  1h 54m 33s, read: 41.1 MiB/s, write: 41.1 MiB/s
INFO:  22% (251.1 GiB of 1.1 TiB) in  1h 59m 17s, read: 41.1 MiB/s, write: 41.1 MiB/s
INFO:  23% (262.5 GiB of 1.1 TiB) in  2h  4m  2s, read: 40.9 MiB/s, write: 40.9 MiB/s
INFO:  24% (273.9 GiB of 1.1 TiB) in  2h  8m 51s, read: 40.6 MiB/s, write: 40.6 MiB/s
INFO:  25% (285.3 GiB of 1.1 TiB) in  2h 13m 46s, read: 39.5 MiB/s, write: 39.5 MiB/s
INFO:  26% (296.8 GiB of 1.1 TiB) in  2h 18m 31s, read: 41.1 MiB/s, write: 41.1 MiB/s
INFO:  27% (308.2 GiB of 1.1 TiB) in  2h 23m 19s, read: 40.6 MiB/s, write: 40.6 MiB/s
INFO:  28% (319.6 GiB of 1.1 TiB) in  2h 28m  4s, read: 40.9 MiB/s, write: 40.9 MiB/s
INFO:  29% (331.0 GiB of 1.1 TiB) in  2h 32m 48s, read: 41.1 MiB/s, write: 41.1 MiB/s
INFO:  30% (342.4 GiB of 1.1 TiB) in  2h 37m 32s, read: 41.3 MiB/s, write: 41.3 MiB/s
INFO:  31% (353.9 GiB of 1.1 TiB) in  2h 42m 27s, read: 39.7 MiB/s, write: 39.7 MiB/s
INFO:  32% (365.2 GiB of 1.1 TiB) in  2h 47m 13s, read: 40.6 MiB/s, write: 40.6 MiB/s
INFO:  33% (376.6 GiB of 1.1 TiB) in  2h 52m  5s, read: 40.0 MiB/s, write: 40.0 MiB/s
INFO:  34% (388.0 GiB of 1.1 TiB) in  2h 56m 48s, read: 41.3 MiB/s, write: 41.3 MiB/s
INFO:  35% (399.4 GiB of 1.1 TiB) in  3h  1m 38s, read: 40.3 MiB/s, write: 40.3 MiB/s
INFO:  36% (410.9 GiB of 1.1 TiB) in  3h  6m 24s, read: 41.0 MiB/s, write: 41.0 MiB/s
INFO:  37% (422.3 GiB of 1.1 TiB) in  3h 10m 50s, read: 43.9 MiB/s, write: 43.8 MiB/s
INFO:  38% (433.7 GiB of 1.1 TiB) in  3h 13m 36s, read: 70.7 MiB/s, write: 70.5 MiB/s
INFO:  39% (445.1 GiB of 1.1 TiB) in  3h 16m 32s, read: 66.2 MiB/s, write: 66.2 MiB/s
INFO:  40% (456.5 GiB of 1.1 TiB) in  3h 19m 12s, read: 72.7 MiB/s, write: 72.3 MiB/s
INFO:  41% (468.0 GiB of 1.1 TiB) in  3h 21m 47s, read: 75.8 MiB/s, write: 75.7 MiB/s
INFO:  42% (479.4 GiB of 1.1 TiB) in  3h 25m  6s, read: 58.7 MiB/s, write: 58.7 MiB/s
INFO:  43% (490.8 GiB of 1.1 TiB) in  3h 28m 35s, read: 55.8 MiB/s, write: 55.4 MiB/s
INFO:  44% (502.2 GiB of 1.1 TiB) in  3h 33m  1s, read: 43.9 MiB/s, write: 43.9 MiB/s
INFO:  45% (513.6 GiB of 1.1 TiB) in  3h 37m 46s, read: 41.0 MiB/s, write: 41.0 MiB/s
INFO:  46% (525.0 GiB of 1.1 TiB) in  3h 42m 37s, read: 40.1 MiB/s, write: 40.1 MiB/s
INFO:  47% (536.4 GiB of 1.1 TiB) in  3h 46m 44s, read: 47.2 MiB/s, write: 46.2 MiB/s
INFO:  48% (547.8 GiB of 1.1 TiB) in  3h 49m 27s, read: 71.8 MiB/s, write: 71.8 MiB/s
INFO:  49% (559.2 GiB of 1.1 TiB) in  3h 52m 14s, read: 69.9 MiB/s, write: 69.8 MiB/s
INFO:  50% (570.6 GiB of 1.1 TiB) in  3h 55m  2s, read: 69.6 MiB/s, write: 69.6 MiB/s
INFO:  51% (582.1 GiB of 1.1 TiB) in  3h 57m 45s, read: 71.9 MiB/s, write: 71.9 MiB/s
INFO:  52% (593.5 GiB of 1.1 TiB) in  4h  0m 32s, read: 69.9 MiB/s, write: 69.9 MiB/s
INFO:  53% (604.9 GiB of 1.1 TiB) in  4h  5m 13s, read: 41.7 MiB/s, write: 41.7 MiB/s
INFO:  54% (616.3 GiB of 1.1 TiB) in  4h 10m 10s, read: 39.4 MiB/s, write: 39.4 MiB/s
INFO:  55% (627.7 GiB of 1.1 TiB) in  4h 14m 59s, read: 40.4 MiB/s, write: 40.4 MiB/s
INFO:  56% (639.1 GiB of 1.1 TiB) in  4h 19m 31s, read: 42.8 MiB/s, write: 42.8 MiB/s
INFO:  57% (650.5 GiB of 1.1 TiB) in  4h 24m 20s, read: 40.4 MiB/s, write: 40.4 MiB/s
INFO:  58% (661.9 GiB of 1.1 TiB) in  4h 29m 11s, read: 40.2 MiB/s, write: 40.2 MiB/s
INFO:  59% (673.3 GiB of 1.1 TiB) in  4h 33m 53s, read: 41.5 MiB/s, write: 41.5 MiB/s
INFO:  60% (684.8 GiB of 1.1 TiB) in  4h 38m 40s, read: 40.7 MiB/s, write: 40.7 MiB/s
INFO:  61% (696.2 GiB of 1.1 TiB) in  4h 43m 28s, read: 40.6 MiB/s, write: 40.6 MiB/s
INFO:  62% (707.6 GiB of 1.1 TiB) in  4h 48m 16s, read: 40.7 MiB/s, write: 40.7 MiB/s
INFO:  63% (719.0 GiB of 1.1 TiB) in  4h 52m 58s, read: 41.5 MiB/s, write: 41.5 MiB/s
INFO:  64% (730.4 GiB of 1.1 TiB) in  4h 57m 41s, read: 41.2 MiB/s, write: 41.2 MiB/s
INFO:  65% (741.9 GiB of 1.1 TiB) in  5h  2m 33s, read: 40.2 MiB/s, write: 40.2 MiB/s
INFO:  66% (753.2 GiB of 1.1 TiB) in  5h  7m 19s, read: 40.7 MiB/s, write: 40.7 MiB/s
INFO:  67% (764.6 GiB of 1.1 TiB) in  5h 12m 10s, read: 40.1 MiB/s, write: 40.1 MiB/s
INFO:  68% (776.1 GiB of 1.1 TiB) in  5h 17m  0s, read: 40.5 MiB/s, write: 40.5 MiB/s
INFO:  69% (787.5 GiB of 1.1 TiB) in  5h 21m 46s, read: 40.8 MiB/s, write: 40.8 MiB/s
INFO:  70% (798.9 GiB of 1.1 TiB) in  5h 25m 56s, read: 46.8 MiB/s, write: 46.8 MiB/s
INFO:  71% (810.3 GiB of 1.1 TiB) in  5h 28m 43s, read: 69.9 MiB/s, write: 69.9 MiB/s
INFO:  72% (821.7 GiB of 1.1 TiB) in  5h 33m 33s, read: 40.3 MiB/s, write: 40.3 MiB/s
INFO:  73% (833.1 GiB of 1.1 TiB) in  5h 38m 22s, read: 40.5 MiB/s, write: 40.5 MiB/s
INFO:  74% (844.5 GiB of 1.1 TiB) in  5h 43m 11s, read: 40.2 MiB/s, write: 40.2 MiB/s
INFO:  75% (856.0 GiB of 1.1 TiB) in  5h 48m  4s, read: 40.1 MiB/s, write: 40.1 MiB/s
INFO:  76% (867.4 GiB of 1.1 TiB) in  5h 53m  2s, read: 39.2 MiB/s, write: 39.2 MiB/s
INFO:  77% (878.8 GiB of 1.1 TiB) in  5h 57m 55s, read: 39.9 MiB/s, write: 39.9 MiB/s
INFO:  78% (890.2 GiB of 1.1 TiB) in  6h  2m 44s, read: 40.5 MiB/s, write: 40.5 MiB/s
INFO:  79% (901.6 GiB of 1.1 TiB) in  6h  7m 32s, read: 40.5 MiB/s, write: 40.5 MiB/s
INFO:  80% (913.0 GiB of 1.1 TiB) in  6h 12m 26s, read: 39.6 MiB/s, write: 39.6 MiB/s
INFO:  81% (924.4 GiB of 1.1 TiB) in  6h 17m 15s, read: 40.4 MiB/s, write: 40.4 MiB/s
INFO:  82% (935.8 GiB of 1.1 TiB) in  6h 22m 10s, read: 39.6 MiB/s, write: 39.6 MiB/s
INFO:  83% (947.3 GiB of 1.1 TiB) in  6h 27m  1s, read: 40.3 MiB/s, write: 40.3 MiB/s
INFO:  84% (958.6 GiB of 1.1 TiB) in  6h 31m 47s, read: 40.7 MiB/s, write: 40.7 MiB/s
INFO:  85% (970.0 GiB of 1.1 TiB) in  6h 36m 32s, read: 41.0 MiB/s, write: 41.0 MiB/s
INFO:  86% (981.5 GiB of 1.1 TiB) in  6h 41m 34s, read: 38.8 MiB/s, write: 38.8 MiB/s
INFO:  87% (992.9 GiB of 1.1 TiB) in  6h 46m 14s, read: 41.7 MiB/s, write: 41.7 MiB/s
INFO:  88% (1004.3 GiB of 1.1 TiB) in  6h 50m 56s, read: 41.4 MiB/s, write: 41.4 MiB/s
INFO:  89% (1015.7 GiB of 1.1 TiB) in  6h 55m 50s, read: 39.8 MiB/s, write: 39.8 MiB/s
INFO:  90% (1.0 TiB of 1.1 TiB) in  7h  0m 46s, read: 39.4 MiB/s, write: 39.4 MiB/s
INFO:  91% (1.0 TiB of 1.1 TiB) in  7h  5m 43s, read: 39.5 MiB/s, write: 39.5 MiB/s
INFO:  92% (1.0 TiB of 1.1 TiB) in  7h 10m 39s, read: 39.5 MiB/s, write: 39.5 MiB/s
INFO:  93% (1.0 TiB of 1.1 TiB) in  7h 15m 36s, read: 39.2 MiB/s, write: 39.2 MiB/s
INFO:  94% (1.0 TiB of 1.1 TiB) in  7h 21m  2s, read: 35.9 MiB/s, write: 35.9 MiB/s
INFO:  95% (1.1 TiB of 1.1 TiB) in  7h 26m 24s, read: 36.3 MiB/s, write: 36.3 MiB/s
INFO:  96% (1.1 TiB of 1.1 TiB) in  7h 31m 30s, read: 38.1 MiB/s, write: 38.1 MiB/s
INFO:  97% (1.1 TiB of 1.1 TiB) in  7h 36m 33s, read: 38.7 MiB/s, write: 38.7 MiB/s
INFO:  98% (1.1 TiB of 1.1 TiB) in  7h 41m 34s, read: 38.7 MiB/s, write: 38.7 MiB/s
INFO:  99% (1.1 TiB of 1.1 TiB) in  7h 46m 18s, read: 41.4 MiB/s, write: 41.4 MiB/s
INFO: 100% (1.1 TiB of 1.1 TiB) in  7h 51m  7s, read: 40.2 MiB/s, write: 40.2 MiB/s
INFO: backup was done incrementally, reused 769.26 GiB (40%)
INFO: transferred 1.11 TiB in 28273 seconds (41.3 MiB/s)
INFO: Finished Backup of VM 100 (07:51:15)
INFO: Backup finished at 2021-01-23 07:02:16
INFO: Backup job finished successfully
TASK OK
 
tmsg said:
The vm is a web server where customers upload documents. The backups are running once a week and there is no way that 60% of the files on the vm are getting modified in a week.
Click to expand...
the way vm backups are deduplicated is by splicing the disk images into 4MiB chunks and deduplicate that, so it is enough that 1 bit of 4MiB chunk changes that it cannot be deduplicated
(so if small writes are sprinkled over the vm disk this can lead to such a behaviour)

what system does run inside the vm?
maybe some other filesystem in the guest handles such things better?
 
The vm is running Centos 7.9 with LVM/XFS.

Any advice on what to look for inside the VM? Any suggested tuning?

Even with this lack of deduplication, the performance has been better than other backup methods that I've used for this vm, just hoping for a lot better.
 
what may be a factor is if you have trimming enabled? afaik qemu marks trimmed part of the disks as 'dirty', so maybe deactive trimming and check if thats the reason? (or activate discard on the vm disk if it is not already)
 
I deleted about 2/3 of the disc contents of a Centos 6 VM, and the next two PBBackups took 23 hours each. However the third has only taken 11 minutes.
 
I'm also looking into this, but with the proxmox-backup-client on Linux:

1st backup of a folder:

Code: 
www.pxar: had to upload 28.88 GiB of 29.61 GiB in 2704.82s, average speed 10.93 MiB/s).
www.pxar: backup was done incrementally, reused 753.43 MiB (2.5%)

2nd backup directly after that:

Code: 
www.pxar: had to upload 28.92 GiB of 29.65 GiB in 2712.35s, average speed 10.92 MiB/s).
www.pxar: backup was done incrementally, reused 753.43 MiB (2.5%)

So it has uploaded all the stuff again to the PBS and it took again ~2700s. Is this behavior expected?

Source is an ext4 filesystem and no trimming happened during backup.s
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom