Hi!
As we know, gzip (or pigz) launched by vzdump is always launched with the --rsyncable switch. By default.
So we can use rsync to quickly back up a backup file to another network location.
I wrote an example script of a subsystem called SLB - Second Level Backup.
This works fine, but only for one virtual machine at a time. If there are multiple virtual machines in the backup schedule, it will not work.
I don't have enough brainpower to do this.
I want to give a gift to someone who is willing and able to continue this.
Tell me if you want to have my SLB scripts.
Algorithm is easy:
1. Line from /etc/vzdump.conf which describe hook script:
script: /root/second_layer_backup/slb-main.sh
2. The slb-main.sh must know "TO" path for second layer backup. "FROM" path and "FILENAME" it can take from vzdump environment.
We can keep "TO" in /etc/pve/nodes/PVE-HOST-NAME/qemu-server/VMNUM.conf wich able to edit from webinterface as "Notes" in "Summary" of VM.
3. Then, when "FILENAME" is present, SLB launches local copying previous (not todays!!!) backup file on remote storage to file on same storage and path but with suffix "fake". In order to create a file over of which the today's backup file will be copied.
4. These two processes of backing up run in parallel. First is vzdump which make VM backup on PVE host and second is just "copy" command which make copy from last bkp file of same VM to file with suffix "fake" on some remote host "TO".
5. Then they need to wait each other. We dont know who will finish first.
6. Then rsync command starts copying new backup file from PVE host over fake file on remote host.
7. Then "fake" file is renamed to correct name.
8. End
I got very large speedup from rsync. I put here my log for example.
Report from SecondLayerBackup subsystem at pveX.XXX.XXX.
Me is /root/second_layer_backup/slb-mail-send.sh.
2023-10-20_03-32-27 slb-main.sh: Script started with parameter job-end.
2023-10-20_03-32-27 slb-main.sh: Run script slb-rsync.sh for copying local bkp file to over remote "fake" file.
2023-10-20_03-32-27 slb-rsync.sh: Present slb-copy.sh /mnt/pve/hdd-sdb-WD1003FZEX/dump/vzdump-qemu-120-2023_10_20-03_01_01.vma.gz Waitng 300 sec...
2023-10-20_03-37-27 slb-rsync.sh: Present slb-copy.sh /mnt/pve/hdd-sdb-WD1003FZEX/dump/vzdump-qemu-120-2023_10_20-03_01_01.vma.gz Waitng 300 sec...
2023-10-20_03-42-27 slb-rsync.sh: Present slb-copy.sh /mnt/pve/hdd-sdb-WD1003FZEX/dump/vzdump-qemu-120-2023_10_20-03_01_01.vma.gz Waitng 300 sec...
2023-10-20_03-47-27 slb-rsync.sh: Present slb-copy.sh /mnt/pve/hdd-sdb-WD1003FZEX/dump/vzdump-qemu-120-2023_10_20-03_01_01.vma.gz Waitng 300 sec...
2023-10-20_03-50-24 slb-copy.sh: Finished copy last bkp to new fake bkp.
2023-10-20_03-52-27 slb-rsync.sh: Start rsync /mnt/pve/hdd-sdb-WD1003FZEX/dump/vzdump-qemu-120-2023_10_20-03_01_01.vma.gz to remote fake bkp. rsync -avht --partial --inplace --append
2023-10-20_03-52-27 sending incremental file list vzdump-qemu-120-2023_10_20-03_01_01.vma.gz sent 2.04G bytes received 35 bytes 44.76M bytes/sec total size is 58.08G speedup is 28
2023-10-20_03-53-12 slb-rsync.sh: Start rsync /mnt/pve/hdd-sdb-WD1003FZEX/dump/vzdump-qemu-120-2023_10_20-03_01_01.log to remote end. rsync -avht --partial --inplace --append --progre
2023-10-20_03-53-12 sending incremental file list vzdump-qemu-120-2023_10_20-03_01_01.log sent 13.80K bytes received 35 bytes 9.23K bytes/sec total size is 13.68K speedup is 0.99
2023-10-20_03-53-18 slb-rsync.sh: Rename vzdump-qemu-120-2023_10_20-03_01_01.vma.gz.fake to vzdump-qemu-120-2023_10_20-03_01_01.vma.gz at remote end ssh pavels-server mv /tank0/pveX.X
renamed '/tank0/pveX.XXX.XXX/dump/vzdump-qemu-120-2023_10_20-03_01_01.vma.gz.fake' -> '/tank0/pveX.XXX.XXX/dump/vzdump-qemu-120-2023_10_20-03_01_01.vma.gz'
2023-10-20_03-53-19 slb-rsync.sh: done.
2023-10-20_03-53-19 slb-rsync.sh: Delete old backups at remote end ssh pavels-server find /tank0/pveX.XXX.XXX/dump/ -type f \( -name *.log -o -name *.vma.gz \) -ctime +30 -print
/tank0/pveX.XXX.XXX/dump/vzdump-qemu-120-2023_09_19-03_01_01.vma.gz
/tank0/pveX.XXX.XXX/dump/vzdump-qemu-120-2023_09_19-03_01_01.log
2023-10-20_03-53-21 slb-rsync.sh: done.
2023-10-20_03-53-21 slb-rsync.sh: Run script slb-mail-send.sh.
As we know, gzip (or pigz) launched by vzdump is always launched with the --rsyncable switch. By default.
So we can use rsync to quickly back up a backup file to another network location.
I wrote an example script of a subsystem called SLB - Second Level Backup.
This works fine, but only for one virtual machine at a time. If there are multiple virtual machines in the backup schedule, it will not work.
I don't have enough brainpower to do this.
I want to give a gift to someone who is willing and able to continue this.
Tell me if you want to have my SLB scripts.
Algorithm is easy:
1. Line from /etc/vzdump.conf which describe hook script:
script: /root/second_layer_backup/slb-main.sh
2. The slb-main.sh must know "TO" path for second layer backup. "FROM" path and "FILENAME" it can take from vzdump environment.
We can keep "TO" in /etc/pve/nodes/PVE-HOST-NAME/qemu-server/VMNUM.conf wich able to edit from webinterface as "Notes" in "Summary" of VM.
3. Then, when "FILENAME" is present, SLB launches local copying previous (not todays!!!) backup file on remote storage to file on same storage and path but with suffix "fake". In order to create a file over of which the today's backup file will be copied.
4. These two processes of backing up run in parallel. First is vzdump which make VM backup on PVE host and second is just "copy" command which make copy from last bkp file of same VM to file with suffix "fake" on some remote host "TO".
5. Then they need to wait each other. We dont know who will finish first.
6. Then rsync command starts copying new backup file from PVE host over fake file on remote host.
7. Then "fake" file is renamed to correct name.
8. End
I got very large speedup from rsync. I put here my log for example.
Report from SecondLayerBackup subsystem at pveX.XXX.XXX.
Me is /root/second_layer_backup/slb-mail-send.sh.
2023-10-20_03-32-27 slb-main.sh: Script started with parameter job-end.
2023-10-20_03-32-27 slb-main.sh: Run script slb-rsync.sh for copying local bkp file to over remote "fake" file.
2023-10-20_03-32-27 slb-rsync.sh: Present slb-copy.sh /mnt/pve/hdd-sdb-WD1003FZEX/dump/vzdump-qemu-120-2023_10_20-03_01_01.vma.gz Waitng 300 sec...
2023-10-20_03-37-27 slb-rsync.sh: Present slb-copy.sh /mnt/pve/hdd-sdb-WD1003FZEX/dump/vzdump-qemu-120-2023_10_20-03_01_01.vma.gz Waitng 300 sec...
2023-10-20_03-42-27 slb-rsync.sh: Present slb-copy.sh /mnt/pve/hdd-sdb-WD1003FZEX/dump/vzdump-qemu-120-2023_10_20-03_01_01.vma.gz Waitng 300 sec...
2023-10-20_03-47-27 slb-rsync.sh: Present slb-copy.sh /mnt/pve/hdd-sdb-WD1003FZEX/dump/vzdump-qemu-120-2023_10_20-03_01_01.vma.gz Waitng 300 sec...
2023-10-20_03-50-24 slb-copy.sh: Finished copy last bkp to new fake bkp.
2023-10-20_03-52-27 slb-rsync.sh: Start rsync /mnt/pve/hdd-sdb-WD1003FZEX/dump/vzdump-qemu-120-2023_10_20-03_01_01.vma.gz to remote fake bkp. rsync -avht --partial --inplace --append
2023-10-20_03-52-27 sending incremental file list vzdump-qemu-120-2023_10_20-03_01_01.vma.gz sent 2.04G bytes received 35 bytes 44.76M bytes/sec total size is 58.08G speedup is 28
2023-10-20_03-53-12 slb-rsync.sh: Start rsync /mnt/pve/hdd-sdb-WD1003FZEX/dump/vzdump-qemu-120-2023_10_20-03_01_01.log to remote end. rsync -avht --partial --inplace --append --progre
2023-10-20_03-53-12 sending incremental file list vzdump-qemu-120-2023_10_20-03_01_01.log sent 13.80K bytes received 35 bytes 9.23K bytes/sec total size is 13.68K speedup is 0.99
2023-10-20_03-53-18 slb-rsync.sh: Rename vzdump-qemu-120-2023_10_20-03_01_01.vma.gz.fake to vzdump-qemu-120-2023_10_20-03_01_01.vma.gz at remote end ssh pavels-server mv /tank0/pveX.X
renamed '/tank0/pveX.XXX.XXX/dump/vzdump-qemu-120-2023_10_20-03_01_01.vma.gz.fake' -> '/tank0/pveX.XXX.XXX/dump/vzdump-qemu-120-2023_10_20-03_01_01.vma.gz'
2023-10-20_03-53-19 slb-rsync.sh: done.
2023-10-20_03-53-19 slb-rsync.sh: Delete old backups at remote end ssh pavels-server find /tank0/pveX.XXX.XXX/dump/ -type f \( -name *.log -o -name *.vma.gz \) -ctime +30 -print
/tank0/pveX.XXX.XXX/dump/vzdump-qemu-120-2023_09_19-03_01_01.vma.gz
/tank0/pveX.XXX.XXX/dump/vzdump-qemu-120-2023_09_19-03_01_01.log
2023-10-20_03-53-21 slb-rsync.sh: done.
2023-10-20_03-53-21 slb-rsync.sh: Run script slb-mail-send.sh.
Last edited: