Backing Up similar VM from different Proxmox VE to a single place

[nimishbansal]

I have a use case where
I have a VM(vm id = 999), that I have backed up to PBS, now I have created 10 more VMs(vmid=1,2,3,4,5,6,7,8,9,10) by restoring from the same PBS on 10 different Proxmox VEs, now these 11(10 + 1) VMs undergo a deploy, and now I again want to backup to PBS any one of the 11 VMs after the deploy, But I want to make use of incremental backups, so that full backup don't needs to be done.

But on trying it, I found out that VM backups are made on the basis of VM ID . So If I want to backup VM (say vm id = 5) on vm id = (999) so that I can use deduplication, and full backup is not required, and VM gets backup pretty soon, is it possible?

In short, I don't want 11 backups, I want only one backup, but source can be any of the 11 VMs, and I am able to use incremental backups instead of full backups.

Is it possible?

 

[Hannes Laimer]

Hey,

as long as you backup to the same datastore, there won't be any duplicate data. So the backups from all VMs to the same datastore are basically incremental, but no, you can't really map VMIDs when backing them up.

 

[nimishbansal]

as long as you backup to the same datastore, there won't be any duplicate data. So the backups from all VMs to the same datastore are basically incremental

So does that mean, If I am backing up (VM id = 5) , that will still be very quick as backup for VM Id = 999 in PBS will already have many duplicate blocks.?

 

[Hannes Laimer]

yes, it'll basically reuse the data from id=999.

 

[nimishbansal]

Hi, I took the backup of VM 4323 using the below command

vzdump 4323 --storage pbs --mode snapshot

Then I cloned VM(id=4323) and created VM(id=4324) , in same Proxmox VE and then ran

vzdump 4324 --storage pbs --mode snapshot

This gave the output as

INFO: starting new backup job: vzdump 4324 --mode snapshot --storage pbs
INFO: Starting Backup of VM 4324 (qemu)
INFO: Backup started at 2024-05-13 08:03:49
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: 433-clone
INFO: include disk 'sata0' 'isos:4324/vm-4324-disk-0.qcow2' 153G
INFO: creating Proxmox Backup Server archive 'vm/4324/2024-05-13T15:03:49Z'
INFO: starting kvm to execute backup task
INFO: started backup task '69c452f3-75a6-4d8e-b4dd-04e7f7f6ccd0'
INFO: sata0: dirty-bitmap status: created new
INFO: 0% (608.0 MiB of 153.0 GiB) in 3s, read: 202.7 MiB/s, write: 50.7 MiB/s
INFO: 1% (1.6 GiB of 153.0 GiB) in 19s, read: 63.5 MiB/s, write: 63.2 MiB/s
INFO: 2% (3.1 GiB of 153.0 GiB) in 45s, read: 60.2 MiB/s, write: 60.2 MiB/s
INFO: 3% (4.6 GiB of 153.0 GiB) in 1m 11s, read: 60.3 MiB/s, write: 60.3 MiB/s
INFO: 4% (6.2 GiB of 153.0 GiB) in 1m 31s, read: 77.8 MiB/s, write: 77.6 MiB/s
INFO: 5% (7.7 GiB of 153.0 GiB) in 1m 48s, read: 92.7 MiB/s, write: 92.5 MiB/s
INFO: 6% (9.2 GiB of 153.0 GiB) in 2m 11s, read: 68.2 MiB/s, write: 68.0 MiB/s
INFO: 7% (10.7 GiB of 153.0 GiB) in 2m 36s, read: 61.3 MiB/s, write: 61.3 MiB/s
INFO: 8% (12.3 GiB of 153.0 GiB) in 3m 1s, read: 63.2 MiB/s, write: 63.2 MiB/s
INFO: 9% (13.8 GiB of 153.0 GiB) in 3m 26s, read: 61.4 MiB/s, write: 61.3 MiB/s
INFO: 10% (15.4 GiB of 153.0 GiB) in 3m 53s, read: 59.9 MiB/s, write: 59.9 MiB/s
INFO: 11% (16.9 GiB of 153.0 GiB) in 4m 18s, read: 62.7 MiB/s, write: 62.6 MiB/s
INFO: 12% (18.4 GiB of 153.0 GiB) in 4m 42s, read: 65.7 MiB/s, write: 65.5 MiB/s
INFO: 13% (19.9 GiB of 153.0 GiB) in 5m 8s, read: 59.7 MiB/s, write: 59.5 MiB/s
INFO: 14% (21.4 GiB of 153.0 GiB) in 5m 33s, read: 61.9 MiB/s, write: 61.9 MiB/s
INFO: 15% (23.0 GiB of 153.0 GiB) in 5m 59s, read: 61.7 MiB/s, write: 61.7 MiB/s
INFO: 16% (24.5 GiB of 153.0 GiB) in 6m 24s, read: 61.0 MiB/s, write: 60.3 MiB/s
INFO: 17% (26.0 GiB of 153.0 GiB) in 6m 48s, read: 65.7 MiB/s, write: 64.5 MiB/s
INFO: 18% (27.6 GiB of 153.0 GiB) in 7m 13s, read: 62.7 MiB/s, write: 61.1 MiB/s
INFO: 19% (29.1 GiB of 153.0 GiB) in 7m 38s, read: 61.4 MiB/s, write: 60.0 MiB/s
INFO: 20% (30.6 GiB of 153.0 GiB) in 8m 4s, read: 62.0 MiB/s, write: 60.2 MiB/s
INFO: 21% (32.2 GiB of 153.0 GiB) in 8m 28s, read: 65.2 MiB/s, write: 63.3 MiB/s
INFO: 22% (33.7 GiB of 153.0 GiB) in 8m 52s, read: 65.5 MiB/s, write: 64.3 MiB/s
INFO: 23% (35.2 GiB of 153.0 GiB) in 9m 16s, read: 63.2 MiB/s, write: 62.0 MiB/s
INFO: 24% (36.8 GiB of 153.0 GiB) in 9m 42s, read: 62.9 MiB/s, write: 61.1 MiB/s
INFO: 25% (38.3 GiB of 153.0 GiB) in 10m 6s, read: 63.3 MiB/s, write: 63.2 MiB/s
INFO: 26% (39.8 GiB of 153.0 GiB) in 10m 33s, read: 58.7 MiB/s, write: 58.5 MiB/s
INFO: 27% (41.3 GiB of 153.0 GiB) in 10m 59s, read: 59.2 MiB/s, write: 59.2 MiB/s
INFO: 28% (42.9 GiB of 153.0 GiB) in 11m 25s, read: 60.3 MiB/s, write: 60.3 MiB/s
INFO: 29% (44.4 GiB of 153.0 GiB) in 11m 52s, read: 57.9 MiB/s, write: 57.9 MiB/s
INFO: 30% (45.9 GiB of 153.0 GiB) in 12m 18s, read: 60.8 MiB/s, write: 60.6 MiB/s
INFO: 31% (47.4 GiB of 153.0 GiB) in 12m 44s, read: 60.0 MiB/s, write: 59.7 MiB/s
INFO: 32% (49.0 GiB of 153.0 GiB) in 13m 9s, read: 63.2 MiB/s, write: 62.2 MiB/s
INFO: 33% (50.5 GiB of 153.0 GiB) in 13m 36s, read: 58.8 MiB/s, write: 58.8 MiB/s
INFO: 34% (52.0 GiB of 153.0 GiB) in 14m 3s, read: 57.0 MiB/s, write: 56.6 MiB/s
INFO: 35% (53.6 GiB of 153.0 GiB) in 14m 30s, read: 58.7 MiB/s, write: 58.7 MiB/s
INFO: 36% (55.1 GiB of 153.0 GiB) in 14m 59s, read: 53.7 MiB/s, write: 53.7 MiB/s
INFO: 37% (56.6 GiB of 153.0 GiB) in 15m 28s, read: 53.2 MiB/s, write: 53.2 MiB/s
INFO: 38% (58.2 GiB of 153.0 GiB) in 15m 57s, read: 55.3 MiB/s, write: 55.2 MiB/s
INFO: 39% (59.7 GiB of 153.0 GiB) in 16m 24s, read: 56.4 MiB/s, write: 56.3 MiB/s
INFO: 40% (61.3 GiB of 153.0 GiB) in 16m 52s, read: 58.0 MiB/s, write: 58.0 MiB/s
INFO: 41% (62.8 GiB of 153.0 GiB) in 17m 17s, read: 61.9 MiB/s, write: 61.9 MiB/s
INFO: 42% (64.3 GiB of 153.0 GiB) in 17m 45s, read: 56.1 MiB/s, write: 56.1 MiB/s
INFO: 43% (65.8 GiB of 153.0 GiB) in 18m 13s, read: 54.4 MiB/s, write: 54.3 MiB/s
INFO: 44% (67.4 GiB of 153.0 GiB) in 18m 38s, read: 64.6 MiB/s, write: 62.2 MiB/s
INFO: 45% (68.9 GiB of 153.0 GiB) in 19m 5s, read: 56.7 MiB/s, write: 56.7 MiB/s
INFO: 46% (70.4 GiB of 153.0 GiB) in 19m 33s, read: 56.7 MiB/s, write: 56.7 MiB/s
INFO: 47% (71.9 GiB of 153.0 GiB) in 20m, read: 57.2 MiB/s, write: 57.2 MiB/s
INFO: 48% (73.6 GiB of 153.0 GiB) in 20m 25s, read: 68.0 MiB/s, write: 63.2 MiB/s
INFO: 49% (75.0 GiB of 153.0 GiB) in 20m 45s, read: 71.0 MiB/s, write: 61.8 MiB/s
INFO: 50% (76.5 GiB of 153.0 GiB) in 21m 10s, read: 63.8 MiB/s, write: 63.7 MiB/s
INFO: 51% (78.0 GiB of 153.0 GiB) in 21m 36s, read: 59.1 MiB/s, write: 58.9 MiB/s
INFO: 52% (79.6 GiB of 153.0 GiB) in 22m 3s, read: 60.0 MiB/s, write: 58.5 MiB/s
INFO: 53% (81.1 GiB of 153.0 GiB) in 22m 29s, read: 60.2 MiB/s, write: 60.2 MiB/s
INFO: 54% (82.7 GiB of 153.0 GiB) in 22m 55s, read: 60.2 MiB/s, write: 60.0 MiB/s
INFO: 55% (84.2 GiB of 153.0 GiB) in 23m 23s, read: 55.4 MiB/s, write: 55.4 MiB/s
INFO: 56% (85.7 GiB of 153.0 GiB) in 23m 51s, read: 56.4 MiB/s, write: 56.4 MiB/s
INFO: 57% (87.2 GiB of 153.0 GiB) in 24m 18s, read: 57.5 MiB/s, write: 57.2 MiB/s
INFO: 58% (88.7 GiB of 153.0 GiB) in 24m 44s, read: 58.9 MiB/s, write: 58.9 MiB/s
INFO: 59% (90.3 GiB of 153.0 GiB) in 25m 11s, read: 59.3 MiB/s, write: 59.1 MiB/s
INFO: 60% (91.8 GiB of 153.0 GiB) in 25m 34s, read: 66.8 MiB/s, write: 62.4 MiB/s
INFO: 61% (93.4 GiB of 153.0 GiB) in 25m 51s, read: 94.6 MiB/s, write: 66.6 MiB/s
INFO: 62% (94.9 GiB of 153.0 GiB) in 26m 19s, read: 55.9 MiB/s, write: 55.9 MiB/s
INFO: 63% (96.4 GiB of 153.0 GiB) in 26m 46s, read: 57.9 MiB/s, write: 57.9 MiB/s
INFO: 64% (97.9 GiB of 153.0 GiB) in 27m 12s, read: 59.7 MiB/s, write: 59.5 MiB/s
INFO: 65% (99.5 GiB of 153.0 GiB) in 27m 39s, read: 58.1 MiB/s, write: 58.1 MiB/s
INFO: 66% (101.0 GiB of 153.0 GiB) in 28m 5s, read: 60.6 MiB/s, write: 60.6 MiB/s
INFO: 67% (102.5 GiB of 153.0 GiB) in 28m 33s, read: 54.7 MiB/s, write: 54.7 MiB/s
INFO: 68% (104.1 GiB of 153.0 GiB) in 29m 1s, read: 58.3 MiB/s, write: 58.3 MiB/s
INFO: 69% (105.6 GiB of 153.0 GiB) in 29m 25s, read: 63.2 MiB/s, write: 63.2 MiB/s
INFO: 70% (107.1 GiB of 153.0 GiB) in 29m 55s, read: 52.8 MiB/s, write: 52.8 MiB/s
INFO: 71% (108.7 GiB of 153.0 GiB) in 30m 22s, read: 57.9 MiB/s, write: 57.8 MiB/s
INFO: 72% (110.2 GiB of 153.0 GiB) in 30m 51s, read: 55.0 MiB/s, write: 54.9 MiB/s
INFO: 73% (111.7 GiB of 153.0 GiB) in 31m 17s, read: 59.5 MiB/s, write: 59.4 MiB/s
INFO: 74% (113.2 GiB of 153.0 GiB) in 31m 44s, read: 57.5 MiB/s, write: 57.5 MiB/s
INFO: 75% (114.8 GiB of 153.0 GiB) in 32m 11s, read: 57.0 MiB/s, write: 57.0 MiB/s
INFO: 76% (116.3 GiB of 153.0 GiB) in 32m 39s, read: 58.0 MiB/s, write: 57.7 MiB/s
INFO: 77% (117.9 GiB of 153.0 GiB) in 33m 7s, read: 55.4 MiB/s, write: 55.3 MiB/s
INFO: 78% (119.3 GiB of 153.0 GiB) in 33m 33s, read: 58.9 MiB/s, write: 58.9 MiB/s
INFO: 79% (120.9 GiB of 153.0 GiB) in 34m, read: 58.1 MiB/s, write: 58.1 MiB/s
INFO: 80% (122.4 GiB of 153.0 GiB) in 34m 28s, read: 55.9 MiB/s, write: 55.9 MiB/s
INFO: 81% (124.0 GiB of 153.0 GiB) in 34m 57s, read: 54.9 MiB/s, write: 54.9 MiB/s
INFO: 82% (125.5 GiB of 153.0 GiB) in 35m 23s, read: 59.5 MiB/s, write: 59.5 MiB/s
INFO: 83% (127.0 GiB of 153.0 GiB) in 35m 50s, read: 59.4 MiB/s, write: 59.4 MiB/s
INFO: 84% (128.6 GiB of 153.0 GiB) in 36m 18s, read: 55.7 MiB/s, write: 55.7 MiB/s
INFO: 85% (130.1 GiB of 153.0 GiB) in 36m 44s, read: 60.3 MiB/s, write: 60.2 MiB/s
INFO: 86% (131.6 GiB of 153.0 GiB) in 37m 11s, read: 58.4 MiB/s, write: 58.1 MiB/s
INFO: 87% (133.1 GiB of 153.0 GiB) in 37m 36s, read: 60.6 MiB/s, write: 60.3 MiB/s
INFO: 88% (134.7 GiB of 153.0 GiB) in 38m 3s, read: 59.6 MiB/s, write: 59.6 MiB/s
INFO: 89% (136.2 GiB of 153.0 GiB) in 38m 29s, read: 59.2 MiB/s, write: 58.9 MiB/s
INFO: 90% (137.7 GiB of 153.0 GiB) in 38m 56s, read: 57.9 MiB/s, write: 57.6 MiB/s
INFO: 91% (139.3 GiB of 153.0 GiB) in 39m 24s, read: 57.3 MiB/s, write: 57.3 MiB/s
INFO: 92% (140.8 GiB of 153.0 GiB) in 39m 50s, read: 59.7 MiB/s, write: 59.7 MiB/s
INFO: 93% (142.3 GiB of 153.0 GiB) in 40m 18s, read: 55.9 MiB/s, write: 55.9 MiB/s
INFO: 94% (143.8 GiB of 153.0 GiB) in 40m 45s, read: 56.7 MiB/s, write: 56.7 MiB/s
INFO: 95% (145.4 GiB of 153.0 GiB) in 41m 13s, read: 56.9 MiB/s, write: 56.9 MiB/s
INFO: 96% (146.9 GiB of 153.0 GiB) in 41m 40s, read: 58.8 MiB/s, write: 58.8 MiB/s
INFO: 97% (148.4 GiB of 153.0 GiB) in 42m 10s, read: 51.5 MiB/s, write: 51.5 MiB/s
INFO: 98% (150.0 GiB of 153.0 GiB) in 42m 37s, read: 59.9 MiB/s, write: 59.9 MiB/s
INFO: 99% (151.5 GiB of 153.0 GiB) in 43m 2s, read: 61.8 MiB/s, write: 61.8 MiB/s
INFO: 100% (153.0 GiB of 153.0 GiB) in 43m 24s, read: 68.9 MiB/s, write: 58.2 MiB/s
INFO: backup is sparse: 2.06 GiB (1%) total zero data
INFO: backup was done incrementally, reused 2.12 GiB (1%)
INFO: transferred 153.00 GiB in 2604 seconds (60.2 MiB/s)
INFO: stopping kvm after backup task
INFO: Finished Backup of VM 4324 (00:43:27)
INFO: Backup finished at 2024-05-13 08:47:16
INFO: Backup job finished successfully
INFO: notified via target `mail-to-root`

But this should not take so long. What can be the reason?

If I had used the same vm id(id=4323) to backup, it would have finished quickly.

 

[nimishbansal]

@Hannes Laimer , is this expected?

 

[Hannes Laimer]

Yes, for the first run, since there's no 'dirty' sections yet, so everything has to be read. But no additional storage will be used on the PBS.