Active / Standby with rsync

[somecallmemike]

Hi,

I've been able to work out a system in which a master proxmox server has a number of containers, which are duplicated in size, distribution, and memory on a cluster slave. Then I created some scripts that rsync the entire container over to the remote server, overwriting the corresponding container with all relevant data (in this case a web server's IP schemes, mysql tables, httpd confs, the whole shebang). I can stop the running virtual machine, start the remote version of that machine, and have a replicated copy running after a disaster event at any given time. The part that really excited me was the ability to sync the entire file structure with locks, dynamic mysql data, and configurations. Each time I changed a server, the changes got pushed over every five minutes by a cron. Althought it requires manual intervention to stop/start the virtual machines in a disaster event it was neat to see a backup copy of my web server up and running within four seconds of the other being stopped. The best part about this setup is that the remote container doesn't have to be turned on to be updated, unlike most replication technology that requires the operating system be turned on. There is more than likely a much better way to do this, but it was simple, took no time at all to setup, and gave me some peace of mind. Any thoughts?

 

[typo3usa]

I am thinking it would be great to share the scripts / the how to on this.

Few quick questions

1. We have some vps that are over 800GB in size - how well would this method work w/ something that large?

2. have you tried any heartbeat methods? ie - ping if no answer - issue shutdown vzctl control to host node A - issue startup to vzctl control to host node B ?

 

[somecallmemike]

I am thinking it would be great to share the scripts / the how to on this.

Few quick questions

1. We have some vps that are over 800GB in size - how well would this method work w/ something that large?

2. have you tried any heartbeat methods? ie - ping if no answer - issue shutdown vzctl control to host node A - issue startup to vzctl control to host node B ?

For your first question, we have not tried rsyncing anything larger than a few gigs for testing purposes. But rysnc is an excellent tool for backing up incremental amounts of data, so once you do a full dump of the machine, it would only need to move the amount of data that has changed. Actually, I believe that vzdump uses rsync to make the backups. For these purposes I would recommend a server with very fast drives that could scan an 800 gigabyte vps without impacting the performance of the applications running on the machine.

I have thought about heartbeat, just haven't had time to try it out. I imagine it would be simple to do exactly what you just mentioned

My script is really simple:

#!/bin/sh

rsync -avrztp --delete /var/lib/vz/private/112/ 10.1.1.2:/var/lib/vz/private/113

echo "The virtual machine has been rsynced"

In essence I created two virtual machines from the master server, one on the master (112), and one on the slave (113) that were exactly the same (memory, disk space, linux distro). I built a LAMP install on 112, built a wordpress blog that I could make changes to, then ran the script to write the 112 vps over the 113 vps and had an exact copy.

Restoring the vps is the same in reverse, just run the script from the slave machine to restore the files to the master server.

Let me know if you have success with such a large machine in this configuration!

 

[jmbeuken]

Hi,

My script is really simple:

#!/bin/sh

rsync -avrztp --delete /var/lib/vz/private/112/ 10.1.1.2:/var/lib/vz/private/113

echo "The virtual machine has been rsynced"

In essence I created two virtual machines from the master server, one on the master (112), and one on the slave (113) that were exactly the same (memory, disk space, linux distro). ...

what about the hardlink and the option -H for rsync ?

regards

jmb

 

[l.mierzwa]

What are You going to do if master proxmox server will die during rsync? You may end with half rsynced mysql files. Well it does not need to die, rsyncing large database that is doing a lot of writes will never be 100% consistent with db running.

 

[kobuki]

Do you take measures against unwritten data in database buffers for example? No database host can be safely backed up without notifying the DB of the action (flushing/locking tables, checkpointing logs, etc). DB engines are just an example, but any application relying on memory buffers/session data are potentially affected and can render a backup inconsistent or unusable.

 

[dietmar]

No database host can be safely backed up without notifying the DB of the action (flushing/locking tables, checkpointing logs, etc).

Note: That is not really true. DB engines use transactions and try hard to make sure that there is always a safe state on the disk. At least the state can be recovered by using the journal. Also most modern files ystem work that way (using journals). But this only works if you make a snapshot of the data before you sync.

 

[kobuki]

Sorry, I have to disagree here. Making an LVM snapshot creates a consistent view of the block device from the point of view of the host, not the VE and its running applications. This way a backup very much behaves like a system that is instantly turned off without shutting down the operating system. I know from experience that database applications sometimes refuse to start up after restoring an online backup of their files, if they haven't been properly handled at the time of the backup. After that you can start a (lenghty) database recovery process, with questionable results and a lot of downtime. If you don't have something like a series of consistent binary logs/archive logs or a full dump, you might get out of luck very quickly and lose a lot of important data. The same applies to file systems as well. In short, to make a consistent backup, one must always notify the applications of the process. If one doesn't, the rest is just luck.

 

[dietmar]

Sorry, I have to disagree here. Making an LVM snapshot creates a consistent view of the block device from the point of view of the host, not the VE and its running applications. This way a backup very much behaves like a system that is instantly turned off without shutting down the operating system.

Exactly - and any reasonable database/filesystem should be able to handle that (at least they are designed to handle exactly that case).

 

[kobuki]

I see, so you basically agree that the backup can be inconsistent when one makes a backup the described way. Sorry, from my point of view that's not the purpose of a backup. An inconsistent backup is just a little better than not having a backup at all. A best effort made on the part of the software might not be enough - and I've seen more than one scenarios when in fact it wasn't enough. I think no theoretical thinking can help me with those. To add to that, for example MySQL's MyISAM doesn't have a transaction log. The vzdump utility itself provides more than one method for the backup procedure. I was under the impression that the purpose of including a start/stop method of working was aimed to avoid the possible data losses that can arise from backing up a potentially inconsistent state.

Of course other trivial problems can arise from the snapshot method, like waiting for hours for an ext3 filesystem check itself thus lengthening downtime considerably. After the filesystem has checked/repaired itself, we need to take care of the database itself, and so on. To be clear: I do use vzdump for regular backup tasks, but also keep full daily (or more frequent) database dumps and/or binlogs in case of failures.

Sorry for bringing this convo forward, but for me it's a very important subject. We can disagree, but other than that I think it's good to think a little (or a lot) before planning a regular maintenance such as backups of live data.

 

[tom]

In most situations, a vzdump snapshot is just working, e.g. if you run reliable database with transactions and filesystems with journaling (which I highly recommend, btw). And yes, its not a application aware backup tool, its doing snapshots via LVM (or power off and power on, depends on the selected mode). Snapshots are not always the final solution, so you should think of backing up your data with a backup tool - there are a lot around.

So I do not understand this discussion - what do you expect from vzdump?

 

[kobuki]

In most situations, a vzdump snapshot is just working, e.g. if you run reliable database with transactions and filesystems with journaling (which I highly recommend, btw). And yes, its not a application aware backup tool, its doing snapshots via LVM (or power off and power on, depends on the selected mode). Snapshots are not always the final solution, so you should think of backing up your data with a backup tool - there are a lot around.

That's what I do and talk about here. One must augment/replace the snapshot method with reliable additions like database dumps or agent/script based solutions to achieve a consistent backup. Snapshotting in itself won't make a backup that's always consistent, period. On another note, when a VPS runs a shared hosting environment with the most popular software, it's inevitable to use for example non-transactioned database engines like MyISAM. But that's out of scope.

So I do not understand this discussion - what do you expect from vzdump?

The discussion is not about vzdump but my observations on OP's method. I brought vzdump up as an example - it is a great and easy to use tool but out of scope for this thread. I'm sorry if I confused anyone by mentioning it.

 

[dietmar]

Snapshotting in itself won't make a backup that's always consistent, period.

As said above, it is 100% consistent if the application/database use transactions. Using Database tools which does not have transactions is very risky, and I would not use such tools anyways (why?).

 

[dietmar]

The discussion is not about vzdump but my observations on OP's method. I brought vzdump up as an example - it is a great and easy to use tool but out of scope for this thread. I'm sorry if I confused anyone by mentioning it.

Well, simply using rsync is not enough, that is right.

 

[kobuki]

As said above, it is 100% consistent if the application/database use transactions. Using Database tools which does not have transactions is very risky, and I would not use such tools anyways (why?).

You are absolutetly right, any serious application engeneer should choose a transaction-capable DB. But there isn't always a choice - one example being the shared hosting environment with MySQL. I've recently migrated several live systems to the Proxmox PVE, and I don't really have a choice over what's running in them - that's customer responsibility.

On another note, relying on error recovery in cases of unexpected failures is perfectly normal - and that's why, among other considerations, the recommended setup is transaction-aware, with redo buffer/log/journal, etc. But relying on error recovery mechanisms for a regular backup in a way defeats the purpose of making the backup which should be in itself consistent and reliable. We can talk about differences of interpretations of what constitutes a (consistent and reliable) backup, if you like. And then, at the other end of the scale, some systems can even afford losing some data and be happy with restoring the last nightly. I think we've made our points.