Kinda feature request for backup.

[tomtom13]

Hi,
You may treat this it as a kinda feature request
So I've got most of stuff at my workplace running on proxmox, but I've noticed that backup is somehow lacking. This is not meant as bitching topic - so no flames please.

So main problem I've got is disaster recovery, right now backup only allows to backup to local storage ... which one would have to be mounted by hand, unmounted and caried out .... this poses few problem:
- you can't give it to a monkey ... they are bound to f**** things up in console
- sysadmin will f*** things up in console anyway
- there is no way to automatically move stuff to tape
- there is no incremental backup ...
- backing up large VM is bound to give you night mares :/ For example I've got one VM that has >15TB of data - how I'm going to make a back up of that into a single disk ... ? where I'll keep temp files ...
- hdd fails when transported.
- no means of backing up to external servers / nfs / s3 ....


I've been digging some time and I found this:
http://backy2.com
after some reading you can find that it's capable of backing stuff in rather nice fashion ... and it's ceph friendly.
Maybe proxmox could employ power of this little software to give people a true disaster recovery option ... I mean if S3 will stop working than it means that 95% of earth surface is unhabitable ... what's the point of restoring stuff anyway

 

[Milkwyrm]

I don't know why you think you can only back up to local storage? I back up to another server using a simple NFS share. It's all there in the webGUI for you. From that second server you can always TAR it up (or not) to Tape if you want to take it off site or replicate to a remote server.

on Node:
Storage -> add -> NFS share.
On Datacenter
Backup -> add -> configure and point at your NFS share from above.

 

[remark]

You can backup to ANY storage that you can mount. Just mount it, add 'Directory' storage, point it to your mount point, allow Backups in storage properties and that's all.

 

[tomtom13]

@remark thanks for you remark about being able to backup to anything that is mounted - if you would carefully read few words further you would understand that I had problem with stuff that is mounted because people tent to mess it up. Anyway thanks for you remark @remark

If using setup as @MIlkwerm suggests, this will require me to have a backup server with large SAS connected to it that has to sink 50TB and then propagate it to tapes .... every week .... As I've mentioned this can pose a lot of trouble.

 

[FastLaneJB]

You can run scripts on the end of the backup. I've got mine uploading some backups to Azure Cold Blob Storage and also I restore some VM's across to a second host with it's own storage so I have a simple DR solution for them as well.

First script is what vzdump calls (See the Backup docs on that but you can supply a script). Others are then called by this. I used PHP just because I'm used to it and don't do a lot of coding.

WARNING!: If you use this I'm not responsible for problems. The DR one will need that there is no VM on the 2nd host with the same VMID. If there is your going to find it'll get deleted and replaced with the VM from the first host. So just be careful please. I'm only sharing incase any of this is useful for others.

P.S. I do agree the backup could do with a bit of improving. The main issue personally is the lack of incremental or differential backups. So like a once a month full and then differentials for the rest of the month would be a nice option. They could probably add in something to upload to S3 / Azure Storage pretty easily as it didn't take me long to add that in, it's pretty much a one liner to upload something to Azure Blob Storage as you can see.

vzdump-script.pl

#!/usr/bin/perl -w

# Script to run during backup runs

use strict;

print "HOOK: " . join (' ', @ARGV) . "\n";

my $phase = shift;

if ($phase eq 'job-start' ||
    $phase eq 'job-end'  ||
    $phase eq 'job-abort') {

    my $dumpdir = $ENV{DUMPDIR};

    my $storeid = $ENV{STOREID};

    print "HOOK-ENV: dumpdir=$dumpdir;storeid=$storeid\n";

    # Add steps to do here if we need something

} elsif ($phase eq 'backup-start' ||
         $phase eq 'backup-end' ||
         $phase eq 'backup-abort' ||
         $phase eq 'log-end' ||
         $phase eq 'pre-stop' ||
         $phase eq 'pre-restart' ||
         $phase eq 'post-restart') {

    my $mode = shift; # stop/suspend/snapshot

    my $vmid = shift;

    my $vmtype = $ENV{VMTYPE}; # openvz/qemu

    my $dumpdir = $ENV{DUMPDIR};

    my $storeid = $ENV{STOREID};

    my $hostname = $ENV{HOSTNAME};

    # tarfile is only available in phase 'backup-end'
    my $tarfile = $ENV{TARFILE};

    # logfile is only available in phase 'log-end'
    my $logfile = $ENV{LOGFILE};

    print "HOOK-ENV: vmtype=$vmtype;dumpdir=$dumpdir;storeid=$storeid;hostname=$hostname;tarfile=$tarfile;logfile=$logfile\n";

    # Run this if the backup has finished
    if ($phase eq 'backup-end') {
        # VM's to upload to Azure storage
        if ($vmid eq 410 || $vmid eq 700 || $vmid eq 8000 || $vmid eq 8010) {
            system ("/usr/local/bin/dr-second-proxmox.php file=$tarfile remote=proxmox-02 remotepath=/storage2/backups remotestorage=ssd") == 0 ||
                die "Failed to copy the backup to the remote proxmox server";
            system ("/usr/local/bin/azure-delete-expired.php daystokeep=90")  == 0 ||
                die "Removing files older than 90 days from Azure Blob Storage failed";
            system ("/usr/local/bin/azure-backup-upload.php file=$tarfile") == 0 ||
                die "Upload tar file to Azure Blob Storage failed";
        }

        # VM's to restore to SSD storage on proxmox-02
        if ($vmid eq 8000 || $vmid eq 8010 || $vmid eq 900) {
            system ("/usr/local/bin/dr-second-proxmox.php file=$tarfile remote=proxmox-02 remotepath=/storage2/backups remotestorage=ssd") == 0 ||
                die "Failed to copy the backup to the remote proxmox server";
        }

        # VM's to restore to HDD storage on proxmox-02
        if ($vmid eq 150 || $vmid eq 160 || $vmid eq 410 || $vmid eq 700) {
            system ("/usr/local/bin/dr-second-proxmox.php file=$tarfile remote=proxmox-02 remotepath=/storage2/backups remotestorage=store") == 0 ||
                die "Failed to copy the backup to the remote proxmox server";
        }
    }

    # Run this if the log file has finished
    if ($phase eq 'log-end') {
        if ($vmid eq 8000 || $vmid eq 8010 || $vmid eq 700) {
            system ("/usr/local/bin/azure-backup-upload.php file=$logfile") == 0 ||
                die "Upload log file to Azure Blob Storage failed";
        }
    }

} else {
    die "got unknown phase '$phase'";
}

exit (0);

azure-backup-upload.php

Note: You'll need to install the Azure CLI 2.0 onto the Proxmox host for this to work.

#!/usr/bin/php
<?php

// Check that the input for the script is valid
parse_str(implode('&', array_slice($argv, 1)), $_GET);

// Check if we have a file to update
if (!(isset($_GET['file']))) {
        die ("Error: We need a file specified to upload file=filename\n\n");
}
$file = $_GET['file'];
$dstFile = basename($file);

// Set variables
$storageAccount = "**StorageAccountName**";
$storageKey = "**PutStorageKeyHere**";
$container = "proxmox-backup";

// Upload the specified file to Azure Blob Storage
if (file_exists($file)) {
        echo ("Uploading $file to $storageAccount storage account\n\n");
        $cmdAzureCLI = "az storage blob upload --account-name $storageAccount --account-key $storageKey --container-name $container --file $file --name $dstFile";
        shell_exec($cmdAzureCLI);
} else {
        echo ("File does not exist to upload: $file\n\n");
}

?>

azure-delete-expired.php

#!/usr/bin/php
<?php

// Check that the input for the script is valid
parse_str(implode('&', array_slice($argv, 1)), $_GET);

// Check if we have a file to update
if (!(isset($_GET['daystokeep']))) {
        die ("Error: We need a daystokeep passed in as daystokeep=90\n\n");
}
$daysToKeep = $_GET['daystokeep'];

// Set variables
$storageAccount = "**StorageAccountName**";
$storageKey = "**PutStorageKeyHere**";
$container = "proxmox-backup";

// Build the date that we need to check modify date on
$checkDate = date('c', strtotime("-$daysToKeep days"));

// Get a list of every blob in the storage account
$cmdAzureCLI = "az storage blob list --account-name $storageAccount --account-key $storageKey --container-name $container --output=json";
$cmdOutput = shell_exec($cmdAzureCLI);
$blobList = json_decode($cmdOutput, true);

foreach ($blobList as $blob) {
        $blobName = $blob["name"];
        $blobLastModified = date($blob["properties"]["lastModified"]);

        echo (strtotime($blobLastModified));
        echo ("\n\n");

        if (strtotime($blobLastModified) < strtotime($checkDate)) {
                // This blob is over daysToKeep so delete it
                echo "Deleting $blobName as it is older than $daysToKeep days\n";
                $cmdAzureCLI = "az storage blob delete --account-name $storageAccount --account-key $storageKey --container-name $container --name $blobName";
                shell_exec($cmdAzureCLI);
        }
}

?>

dr-second-proxmox.php

SSH needs to be possible between the hosts without login so using SSH keys.

#!/usr/bin/php
<?php

// Check that the input for the script is valid
parse_str(implode('&', array_slice($argv, 1)), $_GET);

// Check if we have a backup to copy
if (!(isset($_GET['file']))) {
        die ("Error: We need a file specified to copy file=filename\n");
}

// Check that we have the name of the remote Proxmox system to copy to
if (!(isset($_GET['remote']))) {
        die ("Error: We need a remote server specified as remote=servername\n");
}

// Check that we have the name of the remote Proxmox system to copy to
if (!(isset($_GET['remotepath']))) {
        die ("Error: We need a remote path specified as remotepath=directory\n");
}

// Check that we have the name of the remote storage to restore the vm to
if (!(isset($_GET['remotestorage']))) {
        die ("Error: We need a remote storage specified as remotestorage=storage\n");
}

// Grab the variables
$file = $_GET['file'];
$remote = $_GET['remote'];
$remotePath = $_GET['remotepath'];
$remoteStorage = $_GET['remotestorage'];
$dstFile = basename($file);
$remoteFile = "$remotePath\/$dstFile";
if (strpos($dstFile, "vzdump-lxc") !== false) {
        $vmID = str_replace("vzdump-lxc-", "", $dstFile);
        $vmType = "lxc";
} else {
        $vmID = str_replace("vzdump-qemu-", "", $dstFile);
        $vmType = "qemu";
}
$vmID = explode("-", $vmID)[0];
$description = "'Restored\ from\ " . gethostname() . "\ on\ " . date("d-m-Y") . "\ ". date("H:i:s") . "'";

// Process the file
if (file_exists($file)) {
        // Copy the file to the remote system
        echo ("Copying $file to $remote proxmox system\n");
        $cmdCLI = "scp $file root@$remote:$remoteFile";
        shell_exec($cmdCLI);

        // Delete VM on host
        echo ("Deleting the VM if it already exists\n");
        $cmdCLI = "ssh root@$remote pct destroy $vmID";
        shell_exec($cmdCLI);

        // Restore the backup
        echo ("Restoring the VM $dstFile to $remote\n");
        if ($vmType == "qemu") {
                $cmdCLI = "ssh root@$remote qmrestore $remoteFile $vmID --storage $remoteStorage";
                shell_exec($cmdCLI);

                // Disable the server from starting on boot
                echo ("Making sure the server doesn't start on boot\n");
                $cmdCLI = "ssh root@$remote qm set $vmID -onboot 0";
                shell_exec($cmdCLI);

                // Set the notes field to note when the machine was restored from
                echo ("Setting the notes field for the VM\n");
                $cmdCLI = "ssh root@$remote qm set $vmID -description $description";
                shell_exec($cmdCLI);

                // Switch protection off on the VM
                echo ("Switching off the protection on the VM\n");
                $cmdCLI = "ssh root@$remote qm set $vmID -protection 0";
                shell_exec($cmdCLI);
        } else {
                $cmdCLI = "ssh root@$remote pct restore $vmID $remoteFile --storage $remoteStorage";
                shell_exec($cmdCLI);

                // Disable the server from starting on boot
                echo ("Making sure the server doesn't start on boot\n");
                $cmdCLI = "ssh root@$remote pct set $vmID -onboot 0";
                shell_exec($cmdCLI);

                // Set the notes field to note when the machine was restored from
                echo ("Setting the notes field for the VM\n");
                $cmdCLI = "ssh root@$remote pct set $vmID -description $description";
                shell_exec($cmdCLI);

                // Switch protection off on the VM
                echo ("Switching off the protection on the VM\n");
                $cmdCLI = "ssh root@$remote pct set $vmID -protection 0";
                shell_exec($cmdCLI);
        }

        // Delete backups for that server that are older than a day
        echo ("Deleting backups for this server $vmID which are over 1 day old on $remote\n");
        $cmdCLI = "ssh root@$remote find $remotePath -type f -name '*$vmID*' -mtime +1 -delete";
        shell_exec($cmdCLI);
} else {
        echo ("File does not exist to copy: $file\n");
}

?>

 

[tomtom13]

Thanks @FastLaneJB, I was slowly thinking that nobody here understands real issues with backups :/
Anyway check out the backy2 I'm intrigued with it because I run everything on top of ceph and ceph -> S3 seems like a logical move. That's why I wanted somebody from team to look at it and maybe bring it up as a feature idea in the future .... proxmox is doing a great work and today my HA "just worked" when monkey switched of power to whole rack ... but backup seems like rocks and clubs ....

 

[FastLaneJB]

Thanks @FastLaneJB, I was slowly thinking that nobody here understands real issues with backups :/
Anyway check out the backy2 I'm intrigued with it because I run everything on top of ceph and ceph -> S3 seems like a logical move. That's why I wanted somebody from team to look at it and maybe bring it up as a feature idea in the future .... proxmox is doing a great work and today my HA "just worked" when monkey switched of power to whole rack ... but backup seems like rocks and clubs ....

I do agree it could do with a few more features but at the same time I think the official line is if you want more you need a third party product. For instance if backing up a Windows VM you really want something that is VSS aware. I do have the guest agent inside the Windows VM's which is supposed to kick off the VSS writers however the end result say on backing up a DC is that it's unaware it's been backed up. So it's not AD aware and I'm guessing things like SQL / Exchange would be similar so they won't clear their logs out.

However I do think there's a few things they could improve on the backups regardless that wouldn't need too much effort just to make it that bit better without going too far and spending too much time on it.

 

[tomtom13]

Funnily enough I was googling for aws stuff and found my old topic ... so to the original detractors making "remarks" - I guess creation of PBS proven YOU wrong :*