backup deduplication

[moihuguesjoyal]

Hi,

I'm looking to create a script that will create a backup of my current VM and stored them in a deduplication system. (I'm using BorgBackup with a remote repo for storing)

I've tried to use vzdump which create a .vma file that doesn't seem to work well with deduplication. To by-pass that, I've try to create a VM snapshot then convert the image snapshot to a qcow2 file which seems to be working great but I'm not sure if I'm doing it right.

Suppose I have a running VM with ID 100, then first I create a snapshot using the following command :
qm snapshot 100 qmsnap

If the VM is using qcow2 format, I can use qemu-img info to see the snapshot.
qemu-img info vm-100-disk-0.qcow2

I can also use qemu-img convert to create a new image from the snapshot.
qemu-img convert -O qcow2 -s qmsnap vm-100-disk-0.qcow2 vm-100-snap.qcow2

If the VM is using LVM storage, I can also use qemu-img to convert the LVM to a .qcow2 file, but I need to activate the volume before.
vgchange -ay
qemu-img convert -O qcow2 /dev/pve/snap_vm-100-disk-0_qmsnap vm-100-snap.qcow2

After the conversion is done, I have a new image that I've been able to boot without problem for now. I can move backup with borg and I delete the snapshot using :
qm delsnapshot 100 qmsnap

So my question is does it make sense to do that?

 

[LnxBil]

I've tried to use vzdump which create a .vma file that doesn't seem to work well with deduplication.

That's true. Just unpack the file with the vma utility and you'll get a RAW file which is perfect for deduplcation.

 

[moihuguesjoyal]

If I do this, It takes twice the time and amount of space for my backups (temporary)... that's why I'm wondering if using snapshot like this would be a good strategy.

 

[LnxBil]

If I do this, It takes twice the time and amount of space for my backups (temporary)...

That's the catch, yes. Yet it works flawlessly. We have a 2 TB "scratch" filesystem exactly for this and store the extracted backups inside of a 48 TB ZFS oool.

that's why I'm wondering if using snapshot like this would be a good strategy.

The only option to use snapshots and replication for off-site-backup is ZFS.

 

[moihuguesjoyal]

The only option to use snapshots and replication for off-site-backup is ZFS.

I'm not sure why LVM and qcow2 snapshot should not be considered? Isn't what the vzdump doing after all? In the source of vzdump it's creating a snapshot with qm snapshot before creating the vma file.

As a reference :
The VZDump source /usr/share/perl5/PVE/VZDump.pm create the snapshot using the driver of the VM/Disk by calling the prepare sub (which create a live snapshot or a suspend snapshot) before calling calling archive sub (which create the .vma)

 

[LnxBil]

I'm not sure why LVM and qcow2 snapshot should not be considered?

Both cannot be incrementally transferred and are "thick" per definition. You can obviously use them, but having consistent backups requires that you hook into the VM and do the backup tasks that vzdump does manually (freeze, thawe via qemu agent) to have a consistent backup. Why this is necessary is described further down.

Isn't what the vzdump doing after all? In the source of vzdump it's creating a snapshot with qm snapshot before creating the vma file.

As a reference :
The VZDump source /usr/share/perl5/PVE/VZDump.pm create the snapshot using the driver of the VM/Disk by calling the prepare sub (which create a live snapshot or a suspend snapshot) before calling calling archive sub (which create the .vma)

Yes, but the text talks about a QEMU live snapshot feature, not a storage snapshot. Otherwise you would not be able to backup a thick lvm or RAW image. It is described here: https://wiki.qemu.org/Features/Snapshots
If every storage would allow snapshots, the backup utility would be able to use them, but it currently does not. Every VM backupped like this:
1) tell QEMU that it's going to be backed up (guest OS freezes IO via qemu agent)
2) QEMU snapshots the current storage layer via live snapshot, guest OS thaws IO writes
3) outer process is retrieving all snapshotted blocks for writing to backup file
4) every new writing IO to the disk is written to a off-storage location to not interfere with the snaphot
5) after the backup file written, QEMU merges the written blocks into the "real" storage location and VM continues to work like it should

 

[moihuguesjoyal]

Both cannot be incrementally transferred and are "thick" per definition. You can obviously use them, but having consistent backups requires that you hook into the VM and do the backup tasks that vzdump does manually (freeze, thawe via qemu agent) to have a consistent backup. Why this is necessary is described further down.

qm snapshot is calling a fs-freeze through qemu-guest-agent and the snapshot generated is read-only.

Yes, but the text talks about a QEMU live snapshot feature, not a storage snapshot. Otherwise you would not be able to backup a thick lvm or RAW image. It is described here: https://wiki.qemu.org/Features/Snapshots
If every storage would allow snapshots, the backup utility would be able to use them, but it currently does not. Every VM backupped like this:
1) tell QEMU that it's going to be backed up (guest OS freezes IO via qemu agent)
2) QEMU snapshots the current storage layer via live snapshot, guest OS thaws IO writes
3) outer process is retrieving all snapshotted blocks for writing to backup file
4) every new writing IO to the disk is written to a off-storage location to not interfere with the snaphot
5) after the backup file written, QEMU merges the written blocks into the "real" storage location and VM continues to work like it should

I known that the method could only be done with the storage type qcow2 and LVM-thin.

With qcow2, the snapshot is an internal snapshot, that why I need to specify to only extract the snapshot with -s params in the case of qcow2 with qemu-img convert.

With lvm-thin, the snapshot is created into another LVM that need to be activated before dumping it.

Normally the 2 snapshots are supposed to be consistent and read-only since it used fs-freeze?

 

[LnxBil]

I known that the method could only be done with the storage type qcow2 and LVM-thin.

vzdump can do it on all storage types, that's the beauty of this approach.

The really detailed explanation is in the utility itself:

https://git.proxmox.com/?p=pve-qemu.git;a=blob_plain;f=backup.txt

Normally the 2 snapshots are supposed to be consistent and read-only since it used fs-freeze?

per default crash consistent, yes. Nothing more, nothing less.

If you have guest support, you can have VSS for windows and can run hook scripts in linux that do more than just crash consistent backups.

 

[Bengt Nolin]

For deduplicating the VMA files you need to decrease the possible deduplicating block size (it is typically variable with a minimum and maximum size). Your deduplicating system of choice most likely have parameters in order for you to tweak that. Here's an link where deduping VMA files is discussed in the context of using Borg backup, but the reasoning behind it is true for other deduping systems as well: https://mail.python.org/pipermail/borgbackup/2017q4/000945.html

 

[LnxBil]

For deduplicating the VMA files you need to decrease the possible deduplicating block size (it is typically variable with a minimum and maximum size).

unpacking will yield the best deduplicatability (is that a word??)

You can even have good space savings without using deduplication if you use a cow-like system like ZFS, as I stated here, here, here and in my ProxTalks presentation (currently only in german, but I'm working on an english version).

Here is an example of one of my VMs in my backup architecture: This is a backup of a redmine VM with a vzdump archive size of 14 GB (according to the logfile) and we have here 64 backups which have combined almost 900 GB or vzdump files, we have used 23,4 GB in the ZFS - that's almost a factor 40 in savings:

root@backup ~ > grep size /proxmox/dump/vzdump-qemu-1005-2019_03_12-18_30_02.log
Mar 12 18:38:23 INFO: archive file size: 14.26GB

root@backup ~ > zfs list -r -t all rpool/proxmox/1005
NAME                                                   USED  AVAIL  REFER  MOUNTPOINT
rpool/proxmox/1005                                    23,4G  4,75T  14,6G  /rpool/proxmox/1005
rpool/proxmox/1005@2017_09_15-18_33_08                1008M      -  13,7G  -
rpool/proxmox/1005@2018_01_15-18_30_02                 767M      -  13,7G  -
rpool/proxmox/1005@2018_07_19-18_30_02                 127M      -  14,1G  -
rpool/proxmox/1005@2018_08_20-18_30_02                 245M      -  14,2G  -
rpool/proxmox/1005@2018_09_17-18_30_02                 242M      -  14,2G  -
rpool/proxmox/1005@2018_10_17-18_30_01                 218M      -  14,3G  -
rpool/proxmox/1005@2018_11_16-18_32_59                 196M      -  14,4G  -
rpool/proxmox/1005@2018_12_06-18_30_02                 121M      -  14,4G  -
rpool/proxmox/1005@2018_12_13-18_30_02                99,5M      -  14,4G  -
rpool/proxmox/1005@2018_12_17-18_30_01                61,8M      -  14,4G  -
rpool/proxmox/1005@2018_12_18-18_30_02                64,1M      -  14,4G  -
rpool/proxmox/1005@2018_12_25-18_30_02                 103M      -  14,5G  -
rpool/proxmox/1005@2018_12_27-18_30_02                 106M      -  14,5G  -
rpool/proxmox/1005@2019_01_01-18_30_01                67,2M      -  14,5G  -
[.... many more .... ]
rpool/proxmox/1005@2019_03_11-18_30_01                49,6M      -  14,6G  -
rpool/proxmox/1005@2019_03_12-18_30_02                   0B      -  14,6G  -

 

[moihuguesjoyal]

Thanks everyone... I've been doing a few tests using vzdump hook script instead of snapshotting manually...

Here is the script I'm using and it seems to be working correctly for now, any suggestion?

#!/bin/sh
export BORG_REPO=
export BORG_PASSPHRASE=

NOW=$(date +"%Y-%m-%d@%T")
RATE_LIMIT=9000 # about 90Mbps

if [ $? != 0 ] ; then exit 1 ; fi
PHASE=$1

if [ $PHASE = 'backup-end' ]; then
    echo "Transfert to borg repo $BORG_REPO"

    MODE=$2
    VMID=$3

    lzop --to-stdout -d $TARFILE | borg create \
        --verbose \
        --filter AME \
        --list \
        --stats \
        --show-rc \
        --compression lz4 \
        --exclude-caches \
        --remote-ratelimit $RATE_LIMIT \
        --chunker-params 12,20,15,4095 \
        ::vm-$VMID-$NOW -

    echo "Prune $BORG_REPO"
    borg prune \
        --list \
        --prefix "vm-$VMID-" \
        --show-rc \
        --keep-daily 7 \
        --keep-weekly 4 \
        --keep-monthly 2
fi

 

[rcd]

I know it's been a while, but I'm messing with similar issues and just came across the thread.

So if I understand this right, you're piping a vzdump into the script? What I don't get is why you have to lzop -d the dump - why not just not compress it in the first place if you know it's going to go to The Borg ?

Sorry if I completely misunderstood everything

 

[Alan Urquhart]

So if I understand this right, you're piping a vzdump into the script? What I don't get is why you have to lzop -d the dump - why not just not compress it in the first place if you know it's going to go to The Borg ?

The vzdump is compressed with LZO, the lzop --to-stdout -d is decompressing it before it's piped to BORG. The reason being to improve the deduplication.

If you send the compressed vzdump directly to borg, in my testing you get approximately 20% deduplication (on VM which hasn't changed much between dumps).

If you send the decompressed vzdump (as in this script) I achieve a 92% deduplication.

Compressed VZDUMP:

Number of files: 1
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                2.45 GB              2.35 GB              1.88 GB
All archives:                4.91 GB              4.70 GB              4.21 GB

Uncompressed (as per the script)

Number of files: 1
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                5.10 GB              2.48 GB            179.61 MB
All archives:               10.20 GB              4.96 GB              2.47 GB

 

[Bengt Nolin]

I know it's been a while, but I'm messing with similar issues and just came across the thread.

So if I understand this right, you're piping a vzdump into the script? What I don't get is why you have to lzop -d the dump - why not just not compress it in the first place if you know it's going to go to The Borg ?

Sorry if I completely misunderstood everything

Yes, you can of course configure the built-in PVE Backups system to not compress the VMA image when taking backups, and then pipe it to Borg without decompressing.

You can even schedule VZDUMP manually using cron and pipe directly from VZDUMP into Borg without a file even landing on disk. But that is not possible with the built-in backups system managed from the web UI as far as I understand.

 

[Alan Urquhart]

Here's what I use to achieve this. I have a 3 node cluster which backups up VM's into an NFS share (located on one of the nodes). For this reason, didn't want to use a VZDUMP hook script to avoid locking issues with borg. So I run this script via cron (from the node that stores the vzdump backups) once all backups have finished.

#!/bin/bash
export BORG_REPO=''
export BORG_PASSPHRASE=""

LOGDIR=/var/log/borgbackup
LOGFILE=$LOGDIR/vm_backup_$(date +"%Y_%m_%d_%I_%M_%p").log

SOURCEDIR=/nfs/proxmox_backup/dump
VMFILE=/tmp/vms.txt
BACKUPLIST=/tmp/backup.txt
pvesh get /cluster/resources --type vm --output-format yaml | egrep -i 'vmid|name' | paste - - -d' '| awk '{ print $4 " " $2 }' > $VMFILE
find $SOURCEDIR -mtime -1 -name '*.lzo' > $BACKUPLIST

if [ ! -d "$LOGDIR" ]
then
    mkdir $LOGDIR
fi

touch $LOGFILE
#redirect all output to logfile
exec &>> $LOGFILE

# some helpers and error handling:
info() { printf "\n%s %s\n\n" "$( date )" "$*";} >> $LOGFILE

START_TIME=$(date +%s)

info "Starting Transfer of Today's backups to $BORG_REPO"
info "Found $( cat "$BACKUPLIST" | wc -l ) backups"

while read line;
do

VMID=$( echo "$line" | cut -d '-' -f 3 )
VMNAME=$( grep "$VMID" $VMFILE | cut -d ' ' -f2 )
BACKUPDATE=$( echo "$line" | cut -d '-' -f 4 )
info "Transferring Backup of VM: $VMID - $VMNAME to Borg"
lzop --to-stdout -d $line | borg create --verbose --filter AME  --list --stats --show-rc --compression lz4 --exclude-caches --chunker-params 12,20,15,4095 ::vm-$VMID-$VMNAME-$BACKUPDATE -

done < ${BACKUPLIST}


info "Total time elapsed: $(date -ud "@$(($(date +%s) - $START_TIME))" +%T) (HH:MM:SS)"

 

[Bengt Nolin]

Here's what I use to achieve this. I have a 3 node cluster which backups up VM's into an NFS share (located on one of the nodes). For this reason, didn't want to use a VZDUMP hook script to avoid locking issues with borg. So I run this script via cron (from the node that stores the vzdump backups) once all backups have finished.

#!/bin/bash
export BORG_REPO=''
export BORG_PASSPHRASE=""

LOGDIR=/var/log/borgbackup
LOGFILE=$LOGDIR/vm_backup_$(date +"%Y_%m_%d_%I_%M_%p").log

SOURCEDIR=/nfs/proxmox_backup/dump
VMFILE=/tmp/vms.txt
BACKUPLIST=/tmp/backup.txt
pvesh get /cluster/resources --type vm --output-format yaml | egrep -i 'vmid|name' | paste - - -d' '| awk '{ print $4 " " $2 }' > $VMFILE
find $SOURCEDIR -mtime -1 -name '*.lzo' > $BACKUPLIST

if [ ! -d "$LOGDIR" ]
then
    mkdir $LOGDIR
fi

touch $LOGFILE
#redirect all output to logfile
exec &>> $LOGFILE

# some helpers and error handling:
info() { printf "\n%s %s\n\n" "$( date )" "$*";} >> $LOGFILE

START_TIME=$(date +%s)

info "Starting Transfer of Today's backups to $BORG_REPO"
info "Found $( cat "$BACKUPLIST" | wc -l ) backups"

while read line;
do

VMID=$( echo "$line" | cut -d '-' -f 3 )
VMNAME=$( grep "$VMID" $VMFILE | cut -d ' ' -f2 )
BACKUPDATE=$( echo "$line" | cut -d '-' -f 4 )
info "Transferring Backup of VM: $VMID - $VMNAME to Borg"
lzop --to-stdout -d $line | borg create --verbose --filter AME  --list --stats --show-rc --compression lz4 --exclude-caches --chunker-params 12,20,15,4095 ::vm-$VMID-$VMNAME-$BACKUPDATE -

done < ${BACKUPLIST}


info "Total time elapsed: $(date -ud "@$(($(date +%s) - $START_TIME))" +%T) (HH:MM:SS)"

This is very similar to what I do in one of my setups, but I use one Borg repository per VM in order to reduce memory consumption, speed up prunes and checks, and also provide the possibility to run Borg operations in parallel. It uses slightly more disk space, but I find it compelling due to the other benefits. It lets me run 3-5 backups in parallel which speeds things up a lot.

 

[linum]

Is the new proxmox backup system not exactly what you try to solve?

 

[Bengt Nolin]

Is the new proxmox backup system not exactly what you try to solve?

It more or less is, and more.

It also is in beta.

 

[Alan Urquhart]

Is the new proxmox backup system not exactly what you try to solve?

Yes the new proxmox backup system would indeed solve the issue of backing up proxmox clusters, in an efficient and safe manner. However for our use case we use our backup nodes with borg not only for proxmox but for many other systems and use cases. We'd have to completely redesign and reconfigure our servers and processes - which isn't worth it. And also it's still in beta.