Allow different compression binaries and configuration for backups

Discussion in 'Proxmox VE 1.x: Installation and configuration' started by obrienmd, Aug 11, 2010.

  1. obrienmd

    obrienmd Member

    Joined:
    Oct 14, 2009
    Messages:
    109
    Likes Received:
    0
    We had great success adding pigz (multithreaded gzip), symlinking gzip to pigz, then turning off --rsyncable in your backup scripts (not compatible with pigz) and adding a cores limit (in our case 2) in your backup scripts gzip command line.

    This sped up backups dramatically (almost linear per core). Perhaps you could add the option to use different compression binaries and options in Proxmox VE 2.0?
     
  2. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,257
    Likes Received:
    276
    OK, added to the TODO list.

    What other (besides pigz)?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. obrienmd

    obrienmd Member

    Joined:
    Oct 14, 2009
    Messages:
    109
    Likes Received:
    0
    Not sure other than pigz... Not an expert on this stuff, I just know there are tons of people out there who use Proxmox w/ 8+ cores, and given backups are usually done at off-hours, why not use more than one (if only 2 or 3)?
     
  4. udo

    udo Well-Known Member
    Proxmox VE Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,721
    Likes Received:
    149
    Hi,
    it's an old thread, but what's about pbzip2 for parallel bzip2? http://compression.ca/pbzip2/

    bzip2 has the effort, if the archive is damage (e.g. one bit flip) only one block is corrupt! AFAIK with gzip the whole content after the damage is lost. Not very suitable for backup.

    Udo
     
  5. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,257
    Likes Received:
    276
    Thanks for that hint - I will test that.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,257
    Likes Received:
    276
    Seems that bzip2 needs much more CPU power that gzip (too much for a backup solution). I have some file where bzip2 is 3 times slower that gzip. So in the end parallel bzip is slower than single threaded gzip!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. escoreal

    escoreal Member
    Proxmox VE Subscriber

    Joined:
    Dec 22, 2010
    Messages:
    78
    Likes Received:
    0
    Hi,

    if you are planing to change something in the backup feature of Proxmox I recommend to take a look at storeBackup.
    http://storebackup.org/

    Especially the feature to use deduplication for block devices for the whole backup history looks promising.

    At the moment I am testing this to use with an external script for KVM guests on logical volumes (taking backups with snapshots).

    And I think such advanced backup features (version history, deduplication) would be great for Proxmox.

    esco
     
  8. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,257
    Likes Received:
    276
    Thanks for that link.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. udo

    udo Well-Known Member
    Proxmox VE Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,721
    Likes Received:
    149
    Hi Dietmar,
    it's right that gzip is fast - but if you have enough cores pbzip2 is not too bad. It will be nice if we had the choice...

    My test for the compression of a 20GB-disk-file:
    Code:
    time gzip xx
    real    10m1.951s
    
    time bzip2 xx
    real    47m56.478s
    
    time pbzip2 -p8 -k -f xx
    real    6m55.389s
    
    time pbzip2 -p16 -k -f xx
    real    3m46.964s
    
    time pbzip2 -p32 -k -f xx
    real    2m5.743s
    
    OK, it's depends on the server but for some people this will be a nice option.

    Udo
     
  10. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,257
    Likes Received:
    276
    I guess that is even faster when using pigz ;-)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. udo

    udo Well-Known Member
    Proxmox VE Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,721
    Likes Received:
    149
    Hi Dietmar,
    of course - but with bzip the backup is safer agains bitfailure.

    Udo
     
  12. escoreal

    escoreal Member
    Proxmox VE Subscriber

    Joined:
    Dec 22, 2010
    Messages:
    78
    Likes Received:
    0
    Hi,

    some results with storeBackup:

    I made a very basic backup script with storeBackup and the performance is impressive. For example a 50 GB LV (NTFS, used space 17,8 GB) took 12m30s for the first and 6m36s for the second run. Used space with bzip2 is 6,5 GB on the backup server.

    To access the backup system I use a local Samba share over an 1 GBit/s link.

    The used script:
    Code:
    #!/bin/bash
    volumes="$@"
    snapshot_size="2G"
    backup_dir="/mnt/backup"
    
    for lv_path in $volumes
    do
            lv_name=$(lvdisplay -c $lv_path|awk -F: '{ print $1 }' | awk -F/ '{ print $4 }')
            lvcreate -L $snapshot_size -s -n ${lv_name}_snapshot $lv_path
    done
    
    for lv_path in $volumes
    do
            lv_name=$(lvdisplay -c $lv_path|awk -F: '{ print $1 }' | awk -F/ '{ print $4 }')
            storeBackup --checkDevicesCompr0 --linkToRecent ${lv_name}_last --logInBackupDir --checkDevices0 ${lv_path}_snapshot --checkDevicesDir0 volumes --checkDevicesBS0 10M --sourceDir /root/empty --backupDir $backup_dir --series ${lv_name}
    done
    
    for lv_path in $volumes
    do
            lvremove -f ${lv_path}_snapshot
    done
    
    esco
     
  13. flow

    flow New Member

    Joined:
    Nov 10, 2010
    Messages:
    28
    Likes Received:
    1
    Hi Escoreal,

    i must say I'm very intrigued by storeBackup (especially the deduplication feature) :D

    I tried your script manually (for testing purposes just one VM)

    Backing up und restoring a VM which was shut down during backup went fine.

    But restoring a VM which was running didn't go so well.
    The restored VM didn't find the bootsector and therefore didn't boot.

    Have you had the same experience or do you backup your machines when they aren't running?

    How do you restore your VMs?

    I used something like this:

    -cd into the directory containing the archive files

    -bzcat -k *.bz2 | pv -cN bzcat > /dev/<VG-NAme>/vm-<ID>-disk-1

    And thanks for pointing out storeBackup :)
     
  14. escoreal

    escoreal Member
    Proxmox VE Subscriber

    Joined:
    Dec 22, 2010
    Messages:
    78
    Likes Received:
    0
    Hi flow,

    I didn't any full recovery tests at the moment and made only some simple tests which worked. Made a diff of a "backup and recover" of storeBackup and and a copy with dd of the LV.
    Example of recovery:
    Code:
    for i in $(ls);do bzcat $i >> /mnt/backup/recover.raw; done
    But it shouldn't make any difference if the VM is running or not because storeBackup copies the snapshot of the LV. And the recovery should only be done in an inactive guest (LV or raw-file).

    Another hint for backup (not only storeBackup) is to zero empty space. For windows you can use precompact.exe (for vhd-files) and for linux sfill:
    http://serverfault.com/questions/16...-free-space-on-windows-for-better-compression

    esco
     
  15. flow

    flow New Member

    Joined:
    Nov 10, 2010
    Messages:
    28
    Likes Received:
    1
    Thanks for the hint with precompact :)

    Will try it later...

    My restores are now working (it was just a typo :) )
     
  16. vcp_ai

    vcp_ai Member

    Joined:
    Jul 28, 2010
    Messages:
    175
    Likes Received:
    0
    Can anybody please post results (improvements ?) by using precompact ??

    Can it be used with windows XP KVM (ide raw) virtual machine ?

    Also can not find a link to obtain this program (all what I've seen is that is a part of Windows 7 ...)

    Thanks
     
  17. vcp_ai

    vcp_ai Member

    Joined:
    Jul 28, 2010
    Messages:
    175
    Likes Received:
    0
    I've found a copy of precompact, and made tests.

    Tests were done against a KVM Windows XP machine with a single 10 GB (IDE raw) disc.
    There are 5,5 GB free, but may be sometime in the past disk was almost full, so it was supposed to be full of 'old-deleted' data. So mi interest for precompact....
    Backup is against an external iscsi.
    The results are:

    Backup without compress:
    Code:
     [FONT=Times New Roman][SIZE=2]
    [/SIZE][SIZE=2]VMID   NAME                STATUS  TIME     SIZE      FILENAME  
      659  vpruautoma1-5.39   OK        00:12:02      8.79GB    /mnt/iscsi1/vzdump-qemu-659-2011_05_07-19_28_01.tar  
                                            TOTAL     00:12:02      8.79GB [/SIZE][/FONT]
    
    Backup with compress:
    Code:
    [FONT=Times New Roman][SIZE=2]VMID    NAME                STATUS      TIME      SIZE         FILENAME  
       659    vpruautoma1-5.39 OK        00:17:17     4.36GB    /mnt/iscsi1/vzdump-qemu-659-2011_05_07-14_25_01.tgz 
                                           TOTAL       00:17:17     4.36GB [/SIZE][/FONT]
    Same after precompact:
    Code:
    [FONT=Times New Roman][SIZE=2]VMID    NAME                STATUS      TIME      SIZE         FILENAME  
       659   vpruautoma1-5.39  OK        00:12:23     2.24GB    /mnt/iscsi1/vzdump-qemu-659-2011_05_07-14_25_01.tgz 
                                            TOTAL      00:12:23     2.24GB [/SIZE][/FONT]
    So with my configuration, I obtain very good results with precompact:
    Same backup time than without compress, and half the space used than with compress ....

    Regards

    Vicente
     
    #17 vcp_ai, May 8, 2011
    Last edited: May 8, 2011
  18. tafkaz

    tafkaz Member

    Joined:
    Dec 1, 2010
    Messages:
    79
    Likes Received:
    0
    Hi there,
    i do have problems with the backup-speed in Proxmox too now and so i am very interested in better solutions.
    Especially your approach with storebackup, Escoreal, seems to be very promising to me.
    I have been using storebackup for other backups for nearly 7 years now, and it always worked like a charm.

    But i am not very familiar with qemu, lvm, openvz and so i don't really know how to implement some vm-backup solution with storebackup.

    Would it be possible for you to post a quick step-by-step "howto backup and restore proxmox-vms using storebackup"?
    That would be just great!!!

    And what do you think? Is integration of this outstanding backup-tool into Proxmox VE Web-GUI possible in the long run?
    Could be just the solution you were looking for...

    Thanks a lot for your help
    Sascha
     
  19. escoreal

    escoreal Member
    Proxmox VE Subscriber

    Joined:
    Dec 22, 2010
    Messages:
    78
    Likes Received:
    0
    Hi tafkaz,

    I use storeBackup since a month now to backup LVM snapshots without any problems.

    Running the above script for backup with a cron entry works for me.
    Recovery with bzcat works, too.

    Yes, but then you wouldn't have one handy backup file and some requirements to the backup storage (hard links, handling a lot files). On the other hand the backup would be faster and could be versionized.
    At the moment I am using ext3 and zfs over CIFS without any problems.

    esco
     
  20. X1M

    X1M New Member

    Joined:
    Apr 19, 2010
    Messages:
    18
    Likes Received:
    0
    I use BackupPC for my servers, it have a nice web interface. I run it inside each Virtual server, but not sure if that is optimal compared to the storebackup solution, I only take backup of the production data.

    There is a LVM add-on for the BackupPC that focus on Virtual servers, it looks nice, maybe that could be a good solution to an Enterprise backup solution for Proxmox VE. I have not used it myself yet.

    Have anyone experience with this?

    Link:
    http://www.biodec.com/tools/software/backuppc-lvm
    http://sourceforge.net/apps/mediawiki/backuppc/index.php?title=User_Scripts_-_Client_-_Backuppc-lvm
     
    #20 X1M, Jun 16, 2011
    Last edited: Jun 16, 2011
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice