Always current hot-standby & backup rotation (script by me)

T

tkienzle

New Member
Feb 6, 2011
6
0
1
Hi There!

Our scenario in our student dormitory (500 Users): two main cluster servers (each dual-quad xeon/16gig) are hosting our machines. They back up to a NFS-Share on each other, so if one fails the other is able to run all important servers, they back up in raw and do not compress to make the backup as smooth as possible.

Now it was time to set up a third server with more storage for archiving and keeping some backups (compressed) ... i thougt: why not having the last backup imported on that machine to have a always actual hot-standby of ALL our computers? finally it's only a question of storage... the seperate machine has enough time to import, export & compress (i used the pigz-tip here in the forum to speed up compression) everything, so the idea is: the seperate machine fetches tonights backup, imports everything and manages the backup-files.

So I wrote something, below you can see the script. Please forgive the crappy and dirty way of coding, I'm not studying engeneering but medicine ;-)

Feel free co comment, use or what ever you like with it.
Regards from Aachen (Germany),

Thomas

P.S.: Since there is a lot of german language in the script i added some comments in english...


Code: 
crontab:

30 10   * * *   root    /root/backup-script.sh >> /root/backupscriptlog.log

/root/backup-script.sh:

#!/bin/bash
   
#zum abschalten aller funktionen und des automatischen löschens der VMs hier ein exit einfügen!
# to switch of everything (case of failed main server...) insert a exit below...


#exit  




heute=`date +%Y_%m_%d` #today
heutekurz=`date +%Y%m%d` #todayshort
jahrestag=`date +%j` #day of the year
wochentag=`date +%u` #day of the week
monatstag=`date +%e`#day of the month





#if [ -f $heute.heute ]
#then
#    echo the file exists
#    rm $heute.heute
#    exit
#fi

#rm *.heute

#touch $heute.heute

   
echo lösche alle 400er vms #removing all VM-machines with id 400-499 ---caution, machines are really deleted, essential to have a range for backup

   COUNTER=400
while [  $COUNTER -lt 499 ]; do
   let COUNTER=COUNTER+1 
   qm destroy $COUNTER
   vzctl destroy $COUNTER
done


date >>/mnt/euro35/proxback/daily/$heute.log


echo kopiere alle maschinen aus dem letzten täglichen backup #get today backups from the four source-directorys (two machines each backing up to each other daily and one weekly copy)

   
      COUNTER=400
for i in $( ls /mnt/pve/wechsels-b1-day/*vzdump-qemu-*$heute*.tar /mnt/pve/wechsels-b2-day/*vzdump-qemu-*$heute*.tar /mnt/pve/wechsels-b1-week/*vzdump-qemu-*$heute*.tar /mnt/pve/wechsels-b2-week/*vzdump-qemu-*$heute*.tar); do
    let COUNTER=COUNTER+1 
	echo importing: $i >>/mnt/euro35/proxback/daily/$heute.log
	echo "as machine"$COUNTER >>/mnt/euro35/proxback/daily/$heute.log
    qmrestore --storage proxb3 $i $COUNTER
done

for i in $( ls /mnt/pve/wechsels-b2-day/*vzdump-open*$heute*.tar /mnt/pve/wechsels-b1-day/*vzdump-open*$heute*.tar /mnt/pve/wechsels-b1-week/*vzdump-open*$heute*.tar /mnt/pve/wechsels-b2-week/*vzdump-open*$heute*.tar ); do
    let COUNTER=COUNTER+1 
	echo importing: $i  >>/mnt/euro35/proxback/daily/$heute.log
	echo "as machine"$COUNTER  >>/mnt/euro35/proxback/daily/$heute.log
    vzrestore $i $COUNTER
done

qm list

vzlist -a

qm list >> /mnt/euro35/proxback/daily/$heute.log #since the id's are screwed up it's essential to have one file that helps to identify the machine
vzlist -a >> /mnt/euro35/proxback/daily/$heute.log

# making the daily backup to the specified directory - i used the pigz-trick to make compression really fast
# result: seven days backup
vzdump --node 10 --snapshot --dumpdir /mnt/euro35/proxback/daily --compress --tmpdir /var/lib/vz/backuptemp --maxfiles 7 --mailto root@kullen.rwth-aachen.de -all

#alt: vzdump --node 10 --snapshot --storage wechsels-b1-week --maxfiles 1 --mailto tkienzle@kullen.rwth-aachen.de -all >>$heute.log



echo $heute $heutekurz $jahrestag $wochentag $monatstag


if [ $jahrestag = 028 ] #if it's a specific day of the year the backup is moved to the archive
	then
	  echo juhu, zeit für ein jahresbackup!
	  mv /mnt/euro35/proxback/daily/*$heute* /mnt/euro35/proxback/yehaar/
fi

if [ $monatstag = 15 ] #same but month
	then
	  echo juhu, zeit für ein monatsbackup!
	  mv /mnt/euro35/proxback/daily/*$heute* /mnt/euro35/proxback/monthly/
fi



if [ $wochentag = 1 ] same but week
	then
	  echo juhu, zeit für ein wochenbackup!
	   mv /mnt/euro35/proxback/daily/*$heute* /mnt/euro35/proxback/weekly/
fi

#altercheck




echo ---------ALTERSCHECK monate - max 1 jahr #of course old backups need to be deleted this deletes everything older than one year

	echo heute: $heutekurz


for i in $( ls /mnt/euro35/proxback/monthly/*.tgz ); do
# i=/mnt/pve/wechsels-b1-day/vzdump-qemu-120-2010_01_13-04_28_50.tar
    echo löschkandidat: $i #candidate for deletion
	loeschkandidat=$i

# reading out the date out of the filename... the f4 specifies the block seperated by "-" = dash characters
 datumloeschkandidatkurz=`echo $i | cut -d "-" -f4 | tr -d [_]` #entferne die underscores und nehme den mit nummer bezeichneten block zwischen den bindestrichen im pfadnamen
 datumloeschkandidat=`echo $i | cut -d "-" -f4`
#	echo $loeschkandidat
	echo datum löschkand: $datumloeschkandidat
#	ls /mnt/euro35/proxback/daily/*$datumloeschkandidat*.*


#!/usr/bin/bash

#some calculating to convert the dates into secounds


date1=$heutekurz
date2=$datumloeschkandidatkurz



#give the dates to awk
tageabstand=$(echo $date1 $date2 | awk '{

#parse
year1=substr($1,1,4);
month1=substr($1,5,2);
day1=substr($1,7,2);

year2=substr($2,1,4);
month2=substr($2,5,2);
day2=substr($2,7,2);

#get seconds
secs1=((year1 - 1970)*365.25+(month1*30.5)+day1)*24*60*60;
secs2=((year2 - 1970)*365.25+(month2*30.5)+day2)*24*60*60;

#subtract

print ((secs1 - secs2)/86400);
}'| sed -e 's/\..*$//')
#sed removes everything thats not a complete number after the dot

echo alter der datei etwa: $tageabstand Tage #age about ... days

if [ "$tageabstand" -ge "365" ]
	then
	  echo also über ein jahr #that means more than a year
	  rm /mnt/euro35/proxback/monthly/*$datumloeschkandidat*.*
fi
    
done


echo ---------ALTERSCHECK woche - max 1 monat #exactly the same system, but now everything older than one month is deleted

	echo heute: $heutekurz


for i in $( ls /mnt/euro35/proxback/weekly/*.tgz ); do
# i=/mnt/pve/wechsels-b1-day/vzdump-qemu-120-2010_01_13-04_28_50.tar
    echo löschkandidat: $i
	loeschkandidat=$i


 datumloeschkandidatkurz=`echo $i | cut -d "-" -f4 | tr -d [_]` #entferne die underscores und nehme den mit nummer bezeichneten block zwischen den bindestrichen im pfadnamen
 datumloeschkandidat=`echo $i | cut -d "-" -f4`
#	echo $loeschkandidat
	echo datum löschkand: $datumloeschkandidat
#	ls /mnt/euro35/proxback/daily/*$datumloeschkandidat*.*


#!/usr/bin/bash

date1=$heutekurz
date2=$datumloeschkandidatkurz



#give the dates to awk
tageabstand=$(echo $date1 $date2 | awk '{

#parse
year1=substr($1,1,4);
month1=substr($1,5,2);
day1=substr($1,7,2);

year2=substr($2,1,4);
month2=substr($2,5,2);
day2=substr($2,7,2);

#get seconds
secs1=((year1 - 1970)*365.25+(month1*30.5)+day1)*24*60*60;
secs2=((year2 - 1970)*365.25+(month2*30.5)+day2)*24*60*60;

#subtract

print ((secs1 - secs2)/86400);
}'| sed -e 's/\..*$//')

echo alter der datei etwa: $tageabstand Tage

if [ "$tageabstand" -ge "31" ]
	then
	  echo also über einen monat
	  rm /mnt/euro35/proxback/weekly/*$datumloeschkandidat*.*
fi
    
done
 
Hi Thomas,
two things:
I would see backup as backup and no "hot standby" - daily backup is not very hot ;)
What's about drbd? With drdb you can have the data in sync the whole time (you need only to backup the config of the vm).

I haven't read (and understand) your script fully, but an easy way to delete backups older than 31days are something like this
Code: 
find /mnt/euro35/proxback -name \*.tar -ctime +31 -exec rm {} \;
Udo
 
udo said:
Hi Thomas,
two things:
I would see backup as backup and no "hot standby" - daily backup is not very hot ;)
What's about drbd? With drdb you can have the data in sync the whole time (you need only to backup the config of the vm).

I haven't read (and understand) your script fully, but an easy way to delete backups older than 31days are something like this
Code: 
find /mnt/euro35/proxback -name \*.tar -ctime +31 -exec rm {} \;
Udo
Click to expand...
Hi Udo!

You're absolutely right, it's a stand-by of the last backup, so let's better say warm-standby ;-)
Find uses the file-date... i liked it more to use the date of the filename ... i doubt the dates... but your way is much more elegant!!
I thought drdb is not integrated in debian yet, or is that thing in my head outdated? drdb+proxmox woult fit together ;-)

Regards from Aachen,

Thomas
 
tkienzle said:
I thought drdb is not integrated in debian yet, or is that thing in my head outdated? drdb+proxmox woult fit together ;-)
Click to expand...

I have 12 servers running DRBD on Proxmox 1.9 in production and two running 2.0 beta totalling roughly 30TB of DRBD awesomeness.
Using DRBD since Proxmox 1.5

As always I suggest having two DRBD volumes in each server pair as the wiki suggests to help make recovery from split brain easy.
http://pve.proxmox.com/wiki/DRBD

I wrote a simple cron task copies the VM config files to the other node into a /configs folder.
So if a node ever fails all I need to do is move the config files into /etc/qemu-server/ on the remaining node and start the VMs.

With 2.0 beta and the new HA feature Proxmox will start your VM on the other node for you when one node fails!
I have not had a chance to test 2.0 HA feature with DRBD yet but plan to do so soon.
The 2.0 forum is here http://forum.proxmox.com/forums/16-Proxmox-VE-2-0-beta if you are interested in learning more about 2.0
 
Last edited:
You must log in or register to reply here.
Top Bottom