Backup methods for Servers and Users

[Eingedi]

Ubuntu 14.04 LTS is a good choice to store backups. do you have a folder [ directory ] with enough storage for the backups? what is the folder name? answer those and we can help you set up a nfs server. nfs has been around for a long time and just works easy with pve.

Yup, enough space. Folder's name is "pve_backup", it's in the "Home" folder. So path should be /Home/pve_backup.
The names are just for the convenience of working with them right now, once I would like to implement the whole thing I will user different names.

 

[Eingedi]

Alright, a small update.
I've created an NFS Server, used it as a storage and made a backup. Now, I have the backup file stored in this server.
This is great, just one problem - I want the access to the backup file to be ONLY from the host server (the nfs server). Meaning, that the only access and changes to this backup will be made through this host, and not the client (right now if the file is deleted from the client server, aka pve, it is also deleted from the host server, which is bad).
So, just tell me... How do I change it? Is this a game of permissions?
Oh, and one more thing - In order to install the NFS Service, I need to "apt-get update" the system. Problem is my PVE version isn't the newest, is it safe to update it? Will it make any changes or it'll all be the same?

 

[m.ardito]

Meaning, that the only access and changes to this backup will be made through this host, and not the client (right now if the file is deleted from the client server, aka pve, it is also deleted from the host server, which is bad).
So, just tell me... How do I change it? Is this a game of permissions?

the pve backup system needs to read and write in that folder. it has to keep the specified "Minimum" number of old copies there, so it needs read and write: if the new backup succeeds, it deletes the oldest one, and so on.
and thus, it can always write (and delete) automatically there. but after the backup you can make a script to move compelted backups elsewhere. You are on your own, though.

Oh, and one more thing - In order to install the NFS Service, I need to "apt-get update" the system. Problem is my PVE version isn't the newest, is it safe to update it? Will it make any changes or it'll all be the same?

"apt-get update" does not upgrade anything, it just refreshes the list of available versions in configured repositories.
but you installed NFS where on the ubuntu 14.04 or a pve node (if yes, is it the same wher you run backup jobs?)?

Marco

 

[Eingedi]

- So other than the script to transfer it, is there any backup method to just create a backup on the external storage? I just don't want any kind of access to the backup file from the pve server after it completes the backup.
- I installed "nfs-common" on pve, and "nfs-kernel" on ubuntu, both seem to require update before installing.
- By the way, is rsync any different from the backup I am using now? Both seems to backup the whole data, only difference is that I can't manage to make rsync work with ssh...
- What about Upgrading PVE? Is it safe or will result in lost/transferred data?

 

[m.ardito]

- So other than the script to transfer it, is there any backup method to just create a backup on the external storage? I just don't want any kind of access to the backup file from the pve server after it completes the backup.

you could use something like a usb drive, and disconnect it after the backup completes.

- I installed "nfs-common" on pve, and "nfs-kernel" on ubuntu, both seem to require update before installing.

doing "Update" lets apt-get pick latest versions, after. But if you "update" ubuntu, why be worried about your "pve version" ?
pve should be able to connect to a NFS server without additional packages! nfs-common should be installed by default, afaik...

- By the way, is rsync any different from the backup I am using now? Both seems to backup the whole data, only difference is that I can't manage to make rsync work with ssh...

pve backup can happen in various ways (stop, suspend, snapshot, etc states depending on your choice) and for both VMs and CTs, and using vma should handle well all kind of VM disk storage (lvm, nfs, ceph, etc) pve allows for each VM, eg.
with rsnc you can transfer files/foders, while many VMs usually run on RAW disks (LVM or other storage)... it could work if you have qcow2 disks but not well, imho. I suggest you to stick with pve backup for full VM backups at least.
If you prefer, and have space enough on PVE, you could backup on local storage, and then move those file elsewhere with rsync. No NFS needed.

- What about Upgrading PVE? Is it safe or will result in lost/transferred data?

sorry didn't get it. Your new NFS server is a different machine from PVE hosts, right?



Marco

 

[Eingedi]

There is "apt-get update" and "apt-get upgrade" right? You said "update" doesn't change anything. But what does update do? Thing is my PVE version is kind of old, and in order to use new features it needs to be updated. So what will happen if I update/upgrade?
I decided I'll probably create an NFS Server and use the PVE backup service for weekly backup, and daily rsync for incremental stuff.

 

[m.ardito]

There is "apt-get update" and "apt-get upgrade" right? You said "update" doesn't change anything. But what does update do?
Thing is my PVE version is kind of old, and in order to use new features it needs to be updated. So what will happen if I update/upgrade?.

well, you said

Oh, and one more thing - In order to install the NFS Service, I need to "apt-get update" the system.

and I answered about that.

afaik:
"apt-get update" updates the local DB of updated for the apt-get system
"apt-get upgrade" upgrades all installed packages that have new versions, and also their dependencies if needed

if you wish (eg) to upgrade just one package, you can "apt-get update" and then "apt-get install packagename".
to see if there are new versions for a package you can install a utility called "apt-show-versions" (install this using the command "apt-get install apt-show-versions")
and use it like "apt-show-versions packagename"

it will just report if there are updates for that package (to be sure, you have to "apt-get update" first, however)
to install any found upfgrade, you do as always with "apt-get install packagename" though.

anyway, NFS is no new feature, I use it from version 1.5, and is installed by default.
So you should not have any trouble upgrading. imho.

I decided I'll probably create an NFS Server and use the PVE backup service for weekly backup, and daily rsync for incremental stuff.

Mmm. As I said before, VM use RAW format or (qcow2 or else) files, for disks storage.

* with RAW you can't rsync anything from pve. afaik. the RAW format is on A LVM volume which looks like a partition, not a filesystem, from pve side.
Inside the VM, you can find the same LVM, where your VM OS is installed and see the filesystem. It's different.
* with file formats (qcow2 or else) each one representing a big whole hard disk, ie, a big file of several GB: you can use rsync but how can you do this incrementally from PVE?

This is why pve only has FULL backups, not incremental, afaik. It does always the full disk backup (except empty space).

Of course you can use rsync for all kind of full/incremental stuff INSIDE the vm, but not from PVE side, ie: outside the VM.
What you probably need is a combination of
* regular full pve backups (say for disaster recovery), which is a specialized for VM backup tool.
* regular OS full/diff/incremental backups inside the VM, with usual backup tools typical of physical machines too (eg: bacula, bareos, backuppc, whatever).
I use both backuppc and hp data protector, would like to have the time to try bareos.

so you could have both full backups and granular backups and use whichever fits best when in the need.

Marco

 

[Eingedi]

Of course you can use rsync for all kind of full/incremental stuff INSIDE the vm, but not from PVE side, ie: outside the VM.
What you probably need is a combination of
* regular full pve backups (say for disaster recovery), which is a specialized for VM backup tool.
* regular OS full/diff/incremental backups inside the VM, with usual backup tools typical of physical machines too (eg: bacula, bareos, backuppc, whatever).
I use both backuppc and hp data protector, would like to have the time to try bareos.

so you could have both full backups and granular backups and use whichever fits best when in the need.

But wasn't the purpose of backing up is storing it on a different server? If I store the backups on the same server it's like I didn't do anything.
Alright, so I can use the NFS server for the full backups, and then use a script to copy the files from the shared folder the the backup server, so there will be no access to it from the pve server. But I can't do this backup daily right? It'll take too much space. So let's say I use this kind of backup weekly. Will it work if I use the PVE backup weekly, and rsync daily? Is the approach correct?
What can I use for the daily backup? Install something like "backuppc" on the pve server itself? And make the backup path to a different server? It even works that way?

Also you said that rsync is different than the full backup the pve offers (Obviously), that means the restoration from rsync wouldn't work?

I know I'm asking a lot of questions, it's because I really want to understand what I'm doing and looking at other perspectives while doing it.
I really appreciate your help!

Barak.

 

[m.ardito]

But wasn't the purpose of backing up is storing it on a different server? If I store the backups on the same server it's like I didn't do anything.

sorry can you point out where I wrote "on the same server"? it's not about the destination host/folder, choose the place you want: it's about the method...
Perhaps is this: I said "regular OS full/diff/incremental backups inside the VM" ?
but I meant backup tools run inside the vm, so they can see the full filesystem and also do diff/backups, but obviously the backup archives could be stored wherever you want...

Trying to recap, what I meant is:
- you can use pve backup (storing archives wherever you want, local, nfs or elsewhere) for full "disaster recovery" with the frequency you need (I can't suggest which, it's up to you, and VM usage. I do them daily, on nfs (nas), keeping a few older copies)
- you can in addition also use other methods (rsync will work for linux VMs, for windows maybe not) to backup whatever you need, even in incremental mode (I do this with backuppc and hp data protector, and then copy backup files also to tape) to have granular restores for, say, a document in a folder which was deleted from a VM user.

for daily backup use whatever works well, but if your VMs have SQL servers be sure to have backup tool suited to that case, because SQL servers need data consistency (eg: for MYSQL you can use a tools like "automysqlbackup")

Yes, you can run a backuppc VM on the pve nodes, but as you said, better store its backup archives on another server (maybe a nas or else), because you have to saparate data and backup.

restoring from rsync will work, of course.

Marco

 

[Eingedi]

OH, now it all makes so much more sense!

I pretty much have it all wrapped now, just... 1-2 more questions.
Let's say I decide not to use the PVE backup service and use only rsync with ssh, which path do I give it to backup? I mean, is there a folder that keeps the system config? I am talking about a single/multiple file/s that when they are backed-up and a file is deleted, I just restore that "config" file and it's all back to normal.

How can you restore a single document? The PVE backup file is all compressed in itself (the .tar.lzo file), in order to restore something you need to restore the whole thing, am I wrong?

 

[udo]

...
Thing is my PVE version is kind of old, and in order to use new features it needs to be updated.

Hi,
sounds, that you don't have the right repositories?!

See here: http://pve.proxmox.com/wiki/Package_repositories
And, depends on your running version, the upgrade need perhaps a little bit attention (first section): http://pve.proxmox.com/wiki/Quick_installation#Introduction

Udo

 

[Eingedi]

I can use the NFS Storage and stuff like that. But I saw that the newer versions have more Storage options and things like "firewall" and such... Don't think it makes any difference since I am using what I have and it seems to work fine.

 

[udo]

I can use the NFS Storage and stuff like that. But I saw that the newer versions have more Storage options and things like "firewall" and such... Don't think it makes any difference since I am using what I have and it seems to work fine.

Hi,
due to security reasons (and fixed issues), you should update your system!

Which version do you are running?

pveversion -v

Udo

 

[Eingedi]

my pve version is 3.1, running kernel is 2.6.